saliency map
In plain wordsA heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.
- 1
Explainable AI
A feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down. displayed as a map over the input, usually an image, showing how strongly each element influenced a model's output. In the original gradient form, a pixel's saliency is the magnitude of the derivative of the class score with respect to that pixel (maximized over color channels), obtained with one backward pass; it indicates which pixels would change the score most if slightly perturbed. Refinements reduce noise and improve localization: gradient × input, guided backpropagation, SmoothGradSmoothGrad, n.: A trick for making explanation heat maps less speckled: compute the map for many noisy copies of the input and average them, so random flicker cancels out., integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way. and class-discriminative maps such as Grad-CAMGrad-CAM, abbr.: A technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label.. Saliency maps are easy to produce and easy to over-read: 'sanity checks' showed that some methods yield nearly identical maps for a trained network and for one with randomized weights, behaving more like edge detectors than explanations.
The saliency map lit up the ruler beside the skin lesion, revealing a shortcut the model had learned.
- 2
Vision science
In models of visual attention, a topographic map encoding how conspicuous each location in a scene is — how much it stands out in color, intensity, orientation or motion — used to predict where people will look. Bottom-up models combine feature maps across scales and select successive fixations with a winner-take-all mechanism.
Origin
The vision-science concept was proposed by Christof Koch and Shimon Ullman (1985) and implemented by Laurent Itti, Christof Koch and Ernst Niebur (1998). Karen Simonyan, Andrea Vedaldi and Andrew Zisserman introduced gradient-based class saliency maps for deep networks in 'Deep Inside Convolutional Networks' (arXiv, December 2013; ICLR 2014 workshop). Julius Adebayo and colleagues' 'Sanity Checks for Saliency Maps' (2018) prompted wide re-examination.
Usage note
A highlighted region shows where the output is sensitive, not why. Different saliency methods can disagree sharply on the same image, and a map that looks sensible may still be unfaithful.
Related
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way.
- Grad-CAMGrad-CAM, abbr.: A technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label.
- SmoothGradSmoothGrad, n.: A trick for making explanation heat maps less speckled: compute the map for many noisy copies of the input and average them, so random flicker cancels out.
- faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given.
- image classificationimage classification, n.: Teaching a computer to say what a picture shows by choosing a label, such as 'cat', 'pneumonia' or 'stop sign', from a fixed list of possibilities.
- feature visualizationfeature visualization, n.: Making pictures that show what a part of an image-recognition network is looking for, by repeatedly adjusting an image until that part responds as strongly as possible.
Sources
- Simonyan, K., Vedaldi, A. & Zisserman, A. (2014). Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. ICLR 2014 Workshop.
- Itti, L., Koch, C. & Niebur, E. (1998). A Model of Saliency-Based Visual Attention for Rapid Scene Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(11).
- Adebayo, J. et al. (2018). Sanity Checks for Saliency Maps. NeurIPS 2018.
Mentioned in
explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., Clever Hans effectClever Hans effect, n.: When a person or an AI seems to solve a problem but is really picking up on an unintended clue — named after a horse that seemed to do arithmetic but was reading its questioners' body language., feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect., Grad-CAMGrad-CAM, abbr.: A technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way., local explanationlocal explanation, n.: An explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general., post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable., attention-based explanationattention-based explanation, n.: Explaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output., attribution graphattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next., DeepLIFTDeepLIFT, abbr.: A method that explains a neural network's output by comparing each internal signal with its value on a neutral reference input, then passing the differences back to the inputs as credit., feature visualizationfeature visualization, n.: Making pictures that show what a part of an image-recognition network is looking for, by repeatedly adjusting an image until that part responds as strongly as possible., guided backpropagationguided backpropagation, n.: A way of producing sharp, detailed pictures of which parts of an image a vision network responded to, by passing only positive signals backward through the network., interpretability illusioninterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow., layer-wise relevance propagationlayer-wise relevance propagation, n.: A method that traces a neural network's decision backward, layer by layer, handing each part of the input its share of the credit until every pixel or word has a relevance score., occlusion sensitivityocclusion sensitivity, n.: Finding out which parts of an image matter to a model by covering them up one patch at a time and watching how much the model's answer changes., SmoothGradSmoothGrad, n.: A trick for making explanation heat maps less speckled: compute the map for many noisy copies of the input and average them, so random flicker cancels out.