intersection over union
In plain wordsA score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area.
A measure of overlap between two regions A and B: the area of their intersection divided by the area of their union, equal to 1 for identical regions and 0 for disjoint ones. In object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene. a predicted bounding boxbounding box, n.: The rectangle a vision system draws around an object it has found, stored as a handful of numbers giving the rectangle's position and size. counts as a true positive only if its IoU with a not-yet-matched ground-truth box exceeds a threshold: 0.5 in the PASCAL VOC benchmark, while COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems. averages results over thresholds from 0.50 to 0.95. In semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another., IoU is computed per class over pixels and averaged as mean IoU (mIoU). IoU also drives non-maximum suppressionnon-maximum suppression, n.: A clean-up step in object detection: when the system draws several overlapping boxes around the same object, it keeps the most confident one and throws away the near-duplicates. and the matching of detections across frames in object trackingobject tracking, n.: Following objects from frame to frame in a video and keeping track of which is which, so a system knows that the car it sees now is the same car it saw a second ago.. Because 1 − IoU gives no useful gradient when boxes do not overlap, training losses use refinements such as generalized IoU (GIoU, 2019), which subtracts a penalty based on the smallest box enclosing both.
A box at 0.48 IoU counts as a miss at the 0.5 threshold, even though it looks fine to a person.
Origin
The ratio was first proposed by the American geologist Grove Karl Gilbert in 1884, as a 'ratio of verification' for tornado forecasts, and again, independently, by the Swiss botanist Paul Jaccard in 1901 to compare the plant species found in different Alpine districts; hence 'Jaccard index'. Under the name 'intersection over union' it became the standard localization criterion in vision through the PASCAL VOC challenge (from 2005).
Usage note
Monotonically related to the Dice coefficient (pixel-level F1) common in medical segmentation: Dice = 2·IoU / (1 + IoU). A model ranked higher by one is ranked higher by the other on a single pair, but averages over images can differ.
Related
- bounding boxbounding box, n.: The rectangle a vision system draws around an object it has found, stored as a handful of numbers giving the rectangle's position and size.
- mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100.
- non-maximum suppressionnon-maximum suppression, n.: A clean-up step in object detection: when the system draws several overlapping boxes around the same object, it keeps the most confident one and throws away the near-duplicates.
- semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another.
- object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
- F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good.
Sources
- Jaccard, P. (1901). Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bulletin de la Société vaudoise des sciences naturelles 37, 547–579.
- Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I. & Savarese, S. (2019). Generalized Intersection over Union: A Metric and a Loss for Bounding Box Regression. CVPR 2019.
Mentioned in
object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene., bounding boxbounding box, n.: The rectangle a vision system draws around an object it has found, stored as a handful of numbers giving the rectangle's position and size., COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems., image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything., keypoint detectionkeypoint detection, n.: Finding specific, recognizable points in an image, such as the corners of a building, a person's elbows and knees, or the tip of a nose., mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100., non-maximum suppressionnon-maximum suppression, n.: A clean-up step in object detection: when the system draws several overlapping boxes around the same object, it keeps the most confident one and throws away the near-duplicates., object trackingobject tracking, n.: Following objects from frame to frame in a video and keeping track of which is which, so a system knows that the car it sees now is the same car it saw a second ago., precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare., semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another., anchor boxanchor box, n.: A ready-made box shape, placed at many spots across an image, that an object detector stretches and shifts to fit real objects instead of drawing boxes from scratch., Jaccard similarityJaccard similarity, n.: A simple score for how much two collections overlap: the number of items they share divided by the number of distinct items in either. Identical sets score 1; sets with nothing in common score 0., network dissectionnetwork dissection, n.: A method for finding out what individual units inside an image-recognition network respond to, such as dogs, windows, grass or stripes, by comparing where each unit lights up with pictures labeled by people., panoptic segmentationpanoptic segmentation, n.: A complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity.