layer-wise relevance propagation
In plain wordsA method that traces a neural network's decision backward, layer by layer, handing each part of the input its share of the credit until every pixel or word has a relevance score.
A feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down. method that explains a neural network's output by propagating it backward through the network as 'relevance', redistributing each neuron's relevance to the neurons of the layer below in proportion to their contributions to its activation. A conservation principle keeps the total relevance approximately constant from layer to layer, so input relevances sum to the output score. Different propagation rules (LRP-0, LRP-ε, LRP-γ, z⁺, αβ) trade off fidelity, noise and robustness and are assigned to different layer types. LRP was later interpreted as deep Taylor decomposition, a chain of local first-order expansions. It needs a single backward pass, is popular in medical imaging and the sciences, and was used to expose 'Clever Hans' classifiersClever Hans effect, n.: When a person or an AI seems to solve a problem but is really picking up on an unintended clue — named after a horse that seemed to do arithmetic but was reading its questioners' body language. that relied on watermarks or other spurious cues instead of the object itself.
LRP put most of the relevance for 'horse' on the copyright tag in the image corner.
Origin
Introduced by Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller and Wojciech Samek in 'On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation' (PLOS ONE, 2015), from TU Berlin and the Fraunhofer Heinrich Hertz Institute.
Usage note
With some rules on ReLU networks, LRP reduces to gradient × input; sanity checks have found certain variants insensitive to the network's upper-layer weights.
Related
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- DeepLIFTDeepLIFT, abbr.: A method that explains a neural network's output by comparing each internal signal with its value on a neutral reference input, then passing the differences back to the inputs as credit.
- saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.
- integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way.
- backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
Sources
- Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R. & Samek, W. (2015). On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLOS ONE 10(7): e0130140.
- Lapuschkin, S. et al. (2019). Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications 10, 1096.
Mentioned in
feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players., DeepLIFTDeepLIFT, abbr.: A method that explains a neural network's output by comparing each internal signal with its value on a neutral reference input, then passing the differences back to the inputs as credit.