attention-based explanation
In plain wordsExplaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output.
The use of attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. weights — the softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.-normalized scores that set how much each position draws on others — as evidence of which inputs a model relied on, typically shown as heat maps over tokens or image patches. Because a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. has many heads and layers whose effects mix through the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document., raw weights from one layer can mislead; attention rollout and attention flow combine weights across layers to estimate how information travels from inputs to outputs. Whether attention counts as explanation is disputed: Jain and Wallace (2019) found attention weights often out of step with gradient-based importance and showed that quite different attention patterns could leave a prediction unchanged, while Wiegreffe and Pinter (2019) replied that attention built adversarially says little about whether a normally trained model's attention is informative. Most researchers now treat attention maps as hypotheses to test with ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network. or activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer., not as faithful explanations.
The attention heat map looked convincing, but ablating the highlighted tokens barely changed the prediction.
Origin
Attention heat maps accompanied the first neural attention models: Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio's translation alignments (2014) and Kelvin Xu and colleagues' 'Show, Attend and Tell' (2015). The debate was framed by Sarthak Jain and Byron C. Wallace's 'Attention is not Explanation' (NAACL 2019) and Sarah Wiegreffe and Yuval Pinter's reply 'Attention is not not Explanation' (EMNLP 2019); Samira Abnar and Willem Zuidema proposed rollout and flow (ACL 2020).
Usage note
An attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map. shows where information was read from, not what was done with it: value vectors and later layers can transform or cancel whatever a head attends to.
Related
- attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.
- attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map.
- attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given.
- saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.
Sources
- Jain, S. & Wallace, B. C. (2019). Attention is not Explanation. Proceedings of NAACL 2019.
- Wiegreffe, S. & Pinter, Y. (2019). Attention is not not Explanation. Proceedings of EMNLP 2019.
- Abnar, S. & Zuidema, W. (2020). Quantifying Attention Flow in Transformers. Proceedings of ACL 2020.
Mentioned in
attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones., attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map.