COCO
In plain wordsA widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems.
'Common Objects in Context': a datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system. of Flickr photographs chosen to show cluttered everyday scenes, with objects among other objects rather than centered and alone, and annotated in several layers so that the same images serve as the benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. for object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene., instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob., keypoint detectionkeypoint detection, n.: Finding specific, recognizable points in an image, such as the corners of a building, a person's elbows and knees, or the tip of a nose. and image captioningimage captioning, n.: Having a computer write a short description of a picture, such as 'a dog catching a frisbee in a park', to help blind people, search engines and other AI systems.. The original paper described about 328,000 images with 2.5 million labeled instances in 91 object categories; the standard benchmarks use 80 'thing' categories, about 118,000 training and 5,000 validation images (the 2017 split), and a held-out test-dev set scored by an evaluation server. Annotations include polygon masks for each instance, 17 body keypoints per person, five human-written captions per image and, later, 'stuff' (amorphous regions such as sky and grass) and panoptic labels. COCO's evaluation protocol, APmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100. averaged over intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area. thresholds from 0.50 to 0.95, became the field's common yardstick, and its JSON annotation format a de facto standard for detection tools.
We report box AP on COCO test-dev and mask AP on val2017.
Origin
Introduced by Tsung-Yi Lin and colleagues from Microsoft Research, Cornell, Caltech, Brown and other institutions in 'Microsoft COCO: Common Objects in Context' (arXiv May 2014; ECCV 2014). Annual challenges followed: detection and captioning from 2015, keypoints from 2016, stuff from 2017 and panoptic segmentation from 2018.
Usage note
Images remain under their original Flickr terms, while the annotations carry a Creative Commons Attribution 4.0 license. As top detectors passed 60 box AP in the 2020s, attention shifted toward harder long-tail and open-vocabulary benchmarks such as LVIS.
Related
- object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
- instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob.
- mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100.
- ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom.
- image captioningimage captioning, n.: Having a computer write a short description of a picture, such as 'a dog catching a frisbee in a park', to help blind people, search engines and other AI systems.
- keypoint detectionkeypoint detection, n.: Finding specific, recognizable points in an image, such as the corners of a building, a person's elbows and knees, or the tip of a nose.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
Sources
- Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L. & Dollár, P. (2014). Microsoft COCO: Common Objects in Context. ECCV 2014. arxiv.org/abs/1405.0312
- Chen, X. et al. (2015). Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv preprint.
Mentioned in
datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system., object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene., bounding boxbounding box, n.: The rectangle a vision system draws around an object it has found, stored as a handful of numbers giving the rectangle's position and size., image captioningimage captioning, n.: Having a computer write a short description of a picture, such as 'a dog catching a frisbee in a park', to help blind people, search engines and other AI systems., instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob., intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area., keypoint detectionkeypoint detection, n.: Finding specific, recognizable points in an image, such as the corners of a building, a person's elbows and knees, or the tip of a nose., mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100., pose estimationpose estimation, n.: Working out the position and orientation of something from images: where a person's joints are and how their body is posed, or exactly how an object or camera is placed and turned in space., precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare., DETRDETR, prop. n.: An object detector built on the transformer, the design behind modern chatbots, that outputs a short list of objects in one go instead of sorting through thousands of guessed boxes., open-vocabulary detectionopen-vocabulary detection, n.: Object detection where you type what to look for, such as 'red umbrella' or 'forklift', and the system finds it, even if it was never trained on that exact category., panoptic segmentationpanoptic segmentation, n.: A complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity.