scale-invariant feature transform
In plain wordsA classic method for finding distinctive spots in a photo, such as corners and blobs, and describing them so the same spots can be matched in other photos taken from different distances or angles.
An algorithm that detects distinctive local keypoints in an image and describes each with a vector robust to changes of scale and rotation and, partly, of illumination and viewpoint. Detection finds extrema of a difference-of-Gaussians filter across positions and scales (a scale space), discards low-contrast and edge-like candidates, and assigns each keypoint a dominant gradient orientation. The descriptor divides the keypoint's neighborhood, rotated to that orientation, into a 4 × 4 grid of cells and records an 8-bin histogram of gradient orientations in each, giving 128 numbers. Descriptors are matched between images by nearest-neighbor distance, with a ratio test that rejects ambiguous matches. SIFT was a workhorse of pre-deep-learning vision: panorama stitching, object recognition, 'bag of visual words' image retrieval, robot localization and structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs.. Faster alternatives (SURF, 2006; ORB, 2011) and learned detectors and matchers (SuperPoint, 2018; LightGlue, 2023) followed, but SIFT remains a common default in 3D reconstruction software.
COLMAP still uses SIFT features by default, so try that before swapping in a learned matcher.
Origin
Introduced by David Lowe of the University of British Columbia at ICCV 1999 and refined in 'Distinctive Image Features from Scale-Invariant Keypoints' (International Journal of Computer Vision, 2004). The university's patent on the method expired in 2020, after which SIFT moved into the main distribution of the OpenCV library.
Related
- keypoint detectionkeypoint detection, n.: Finding specific, recognizable points in an image, such as the corners of a building, a person's elbows and knees, or the tip of a nose.
- feature extractionfeature extraction, n.: Turning raw data such as pixels or sound into a short list of telling numbers — computed by hand-designed rules or by an AI system trained earlier — that a new model can learn from or compare.
- structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs.
- 3D reconstruction3D reconstruction, n.: Building a 3D model of an object or place, its shape and often its colors, from photos, video or depth-sensor scans.
- computer visioncomputer vision, n.: The part of AI that lets computers make sense of images and video: recognizing faces, reading signs, spotting tumors on scans or judging how far away a car is.
Sources
- Lowe, D. G. (1999). Object Recognition from Local Scale-Invariant Features. Proceedings of ICCV 1999.
- Lowe, D. G. (2004). Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60(2), 91–110.
Mentioned in
computer visioncomputer vision, n.: The part of AI that lets computers make sense of images and video: recognizing faces, reading signs, spotting tumors on scans or judging how far away a car is., feature extractionfeature extraction, n.: Turning raw data such as pixels or sound into a short list of telling numbers — computed by hand-designed rules or by an AI system trained earlier — that a new model can learn from or compare., keypoint detectionkeypoint detection, n.: Finding specific, recognizable points in an image, such as the corners of a building, a person's elbows and knees, or the tip of a nose., structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs.