A general method for human activity recognition in video

In this paper we develop a system for human behaviour recognition in video sequences. Human behaviour is modelled as a stochastic sequence of actions. Actions are described by a feature vector comprising both trajectory information (position and velocity), and a set of local motion descriptors. Action recognition is achieved via probabilistic search of image feature databases representing previously seen actions. Hidden Markov Models (HMM) which encode scene rules are used to smooth sequences of actions. High-level behaviour recognition is achieved by computing the likelihood that a set of predefined HMMs explains the current action sequence. Thus, human actions and behaviour are represented using a hierarchy of abstraction: from person-centred actions, to actions with spatio-temporal context, to action sequences and, finally, general behaviours. While the upper levels all use Bayesian networks and belief propagation, the lowest level uses non-parametric sampling from a previously learned database of actions. The combined method represents a general framework for human behaviour modelling. We demonstrate results from broadcast tennis sequences and surveillance footage for automated video annotation.