feature attribution
In plain wordsGiving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
A class of explanation methods that assign each input feature (a pixel, token or tabular variable) a score for its contribution to a model's output on a particular input, usually relative to a baseline or reference input. Gradient-based methods use derivatives of the output (saliencysaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye., gradient × input, SmoothGradSmoothGrad, n.: A trick for making explanation heat maps less speckled: compute the map for many noisy copies of the input and average them, so random flicker cancels out., integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way., Grad-CAMGrad-CAM, abbr.: A technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label.); perturbation-based methods measure the effect of removing or altering features (occlusion, LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example., SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players.); propagation-based methods redistribute the output backward layer by layer (layer-wise relevance propagationlayer-wise relevance propagation, n.: A method that traces a neural network's decision backward, layer by layer, handing each part of the input its share of the credit until every pixel or word has a relevance score., DeepLIFTDeepLIFT, abbr.: A method that explains a neural network's output by comparing each internal signal with its value on a neutral reference input, then passing the differences back to the inputs as credit.). Proposed axioms include completeness (scores sum to the output minus the baseline output), sensitivity and implementation invariance. Because methods often disagree, they are evaluated with sanity checks — do scores change when the weights are randomized? — and deletion or insertion curves, which test whether removing top-ranked features changes the prediction fastest.
Feature attribution showed the pneumonia classifier keyed on a hospital's scanner tag rather than on lung tissue.
Origin
Attributing predictions to input variables is old in statistics. For neural networks, sensitivity analysis with input gradients dates to the late 1980s and 1990s; the modern vocabulary consolidated in 2017, when Mukund Sundararajan, Ankur Taly and Qiqi Yan stated attribution axioms and Scott Lundberg and Su-In Lee unified 'additive feature attribution' methods.
Usage note
Attribution says which inputs mattered for one output, not how the model computed it. Distinguish data attributiondata attribution, n.: Working out which training examples were responsible for a particular behavior or answer of an AI model — useful for explaining, debugging and crediting the data behind it. (which training examples mattered) and global feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect.. In mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., 'attribution' also means scoring internal components, as in attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part. and attribution graphsattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next..
Related
- SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players.
- integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way.
- saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.
- LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example.
- faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given.
- local explanationlocal explanation, n.: An explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general.
- data attributiondata attribution, n.: Working out which training examples were responsible for a particular behavior or answer of an AI model — useful for explaining, debugging and crediting the data behind it.
- feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect.
Sources
- Sundararajan, M., Taly, A. & Yan, Q. (2017). Axiomatic Attribution for Deep Networks. ICML 2017.
- Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M. & Kim, B. (2018). Sanity Checks for Saliency Maps. NeurIPS 2018.
Mentioned in
explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value., counterfactual explanationcounterfactual explanation, n.: An explanation that tells someone the smallest change that would have produced a different outcome — for example, that the application would have been accepted if it had listed two more years of work experience., faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given., feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect., Grad-CAMGrad-CAM, abbr.: A technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label., integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way., LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example., local explanationlocal explanation, n.: An explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general., post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable., saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye., SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players., Shapley valueShapley value, n.: A rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order., AnchorsAnchors, prop. n.: A method that explains a prediction with a simple if-then rule that locks in the result — for example, if an email contains both 'invoice' and 'urgent', the filter almost always flags it., attention-based explanationattention-based explanation, n.: Explaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output., attribution graphattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next., attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part., data attributiondata attribution, n.: Working out which training examples were responsible for a particular behavior or answer of an AI model — useful for explaining, debugging and crediting the data behind it., DeepLIFTDeepLIFT, abbr.: A method that explains a neural network's output by comparing each internal signal with its value on a neutral reference input, then passing the differences back to the inputs as credit., guided backpropagationguided backpropagation, n.: A way of producing sharp, detailed pictures of which parts of an image a vision network responded to, by passing only positive signals backward through the network., influence functioninfluence function, n.: A mathematical shortcut for estimating how much a single training example shaped a model's behavior, for instance how a prediction would change if that example had been left out, without retraining the model., knowledge neuronknowledge neuron, n.: An internal unit of a language model claimed to be responsible for storing a particular fact, such that turning it up or down makes the model more or less likely to state that fact.