object tracking
In plain wordsFollowing objects from frame to frame in a video and keeping track of which is which, so a system knows that the car it sees now is the same car it saw a second ago.
The task of estimating the trajectories of objects through a video while keeping each object's identity consistent across frames. In single-object tracking, a target marked in the first frame must be followed through changes of appearance and occlusion, classically with correlation filters (2010) and later with Siamese networks (2016) and transformers. Multi-object tracking (MOT) must follow every object of given classes; its dominant paradigm, tracking-by-detection, runs an object detectorobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene. on each frame and associates detections with existing tracks using predicted motion (often from a Kalman filterKalman filter, n.: A method that keeps a running best guess of something you cannot measure exactly, such as a vehicle's position, by blending predictions from a motion model with noisy sensor readings.), overlap (intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area.) and appearance embeddingsembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., solved as an assignment problem with the Hungarian algorithm. SORT (2016), DeepSORT (2017) and ByteTrack (2021) are standard baselines, while end-to-end transformer trackers and video segmentation models such as SAM 2Segment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it. track masks directly; point tracking follows arbitrary pixels rather than objects. Metrics such as MOTA, IDF1 and HOTA weigh detection errors against identity switches. Uses include traffic monitoring, sports analytics, autonomous driving, retail analytics and surveillance.
Identity switches spike at the intersection, where pedestrians pass behind the bus shelter.
Origin
Tracking grew out of radar target tracking in the 1950s–70s, including Rudolf Kálmán's filter (1960) and Donald Reid's multiple hypothesis tracking (1979). Visual tracking became a computer-vision topic in the 1980s; Alex Bewley and colleagues' SORT (2016) set today's simple tracking-by-detection baseline.
Usage note
Following people across cameras or over time raises privacy and surveillance concerns beyond those of one-off detection, and in some jurisdictions falls under biometric or data-protection law.
Related
- object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
- optical flowoptical flow, n.: How each part of a video picture seems to shift from one frame to the next, recorded as a tiny arrow per pixel giving the direction and distance moved.
- Kalman filterKalman filter, n.: A method that keeps a running best guess of something you cannot measure exactly, such as a vehicle's position, by blending predictions from a motion model with noisy sensor readings.
- intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area.
- Segment Anything ModelSegment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it.
- AI surveillanceAI surveillance, n.: Using AI to watch, identify, track or profile people on a scale no team of humans could manage, for example by scanning camera feeds or analyzing phone and online data.
Sources
- Bewley, A., Ge, Z., Ott, L., Ramos, F. & Upcroft, B. (2016). Simple Online and Realtime Tracking. ICIP 2016.
- Zhang, Y. et al. (2022). ByteTrack: Multi-Object Tracking by Associating Every Detection Box. ECCV 2022.
Mentioned in
computer visioncomputer vision, n.: The part of AI that lets computers make sense of images and video: recognizing faces, reading signs, spotting tumors on scans or judging how far away a car is., intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area., pose estimationpose estimation, n.: Working out the position and orientation of something from images: where a person's joints are and how their body is posed, or exactly how an object or camera is placed and turned in space., action recognitionaction recognition, n.: Teaching computers to recognize what people are doing in a video, such as running, waving, cooking or falling, rather than just which objects appear in it., optical flowoptical flow, n.: How each part of a video picture seems to shift from one frame to the next, recorded as a tiny arrow per pixel giving the direction and distance moved.