action recognition
In plain wordsTeaching computers to recognize what people are doing in a video, such as running, waving, cooking or falling, rather than just which objects appear in it.
The task of classifying human actions or activities in video, such as 'riding a bike' or 'pouring water', usually from short trimmed clips; related tasks locate actions in time (temporal action localization) or in both space and time (spatio-temporal action detection). Unlike image classificationimage classification, n.: Teaching a computer to say what a picture shows by choosing a label, such as 'cat', 'pneumonia' or 'stop sign', from a fixed list of possibilities., it must model motion and temporal order: opening and closing a door look alike in single frames. Hand-crafted approaches used space-time interest points and dense trajectories (2005–2013). Deep approaches include two-stream networks combining RGB frames with optical flowoptical flow, n.: How each part of a video picture seems to shift from one frame to the next, recorded as a tiny arrow per pixel giving the direction and distance moved. (2014), 3D convolutional networksconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. such as C3D (2015) and I3D (2017), the two-pathway SlowFast network (2019) and video transformers (TimeSformer and ViViT, 2021), often pretrained as video masked autoencodersmasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of.. Benchmarks include UCF101 and HMDB51, Kinetics (from 2017, hundreds of thousands of YouTube clips), Something-Something (fine-grained motion) and AVA. Applications include sports analysis, video search, fall detection in elder care and human–robot interaction.
The model flags falls in the care home but confuses them with residents sitting down quickly.
Origin
Recognizing human motion in video was studied from the 1990s, drawing on Gunnar Johansson's 1973 experiments showing that people recognize actions from a few moving points of light. Karen Simonyan and Andrew Zisserman's two-stream network (2014) and João Carreira and Zisserman's Kinetics dataset and I3D model (2017) shaped the deep-learning era.
Usage note
'Activity recognition' sometimes means recognizing activities from wearable sensors such as accelerometers. Inferring emotions or intentions from behavior is a separate and more contested task (see emotion recognition systememotion recognition system, n.: AI that tries to work out how people feel or what they intend from their faces, voices or bodies. The EU bans it at work and in schools and tightly regulates it elsewhere.).
Related
- optical flowoptical flow, n.: How each part of a video picture seems to shift from one frame to the next, recorded as a tiny arrow per pixel giving the direction and distance moved.
- pose estimationpose estimation, n.: Working out the position and orientation of something from images: where a person's joints are and how their body is posed, or exactly how an object or camera is placed and turned in space.
- object trackingobject tracking, n.: Following objects from frame to frame in a video and keeping track of which is which, so a system knows that the car it sees now is the same car it saw a second ago.
- image classificationimage classification, n.: Teaching a computer to say what a picture shows by choosing a label, such as 'cat', 'pneumonia' or 'stop sign', from a fixed list of possibilities.
- vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence.
- masked autoencodermasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of.
Sources
- Simonyan, K. & Zisserman, A. (2014). Two-Stream Convolutional Networks for Action Recognition in Videos. NeurIPS 2014.
- Carreira, J. & Zisserman, A. (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. CVPR 2017.
Mentioned in
pose estimationpose estimation, n.: Working out the position and orientation of something from images: where a person's joints are and how their body is posed, or exactly how an object or camera is placed and turned in space., optical flowoptical flow, n.: How each part of a video picture seems to shift from one frame to the next, recorded as a tiny arrow per pixel giving the direction and distance moved.