faithfulness
In plain wordsHow accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given.
- 1
Explanations
The degree to which an explanation matches how the model actually arrived at its prediction; Alon Jacovi and Yoav Goldberg (2020) set it against plausibility, how convincing an explanation looks to people, and argued that the two must be evaluated separately. Because ground truth about a model's reasoning is rarely available, faithfulness is measured by proxies: deleting or perturbing the features an explanation marks as important and measuring the output change (deletion and insertion curves; comprehensiveness and sufficiency), retraining without those features, checking whether explanations change when the model's weights are randomized, or testing on models with known, deliberately planted mechanisms. For surrogate modelssurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply., the analogous measure — agreement with the original model's predictions — is usually called fidelity.
The rationales looked sensible to annotators but failed every faithfulness test we ran.
- 2
Generated text
The property of generated text — a summary, translation or answer — being fully supported by its source document or retrieved context, with nothing that contradicts it or goes beyond it; failures are a form of hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true.. Distinguished from factualityfactuality, n.: How far what an AI says is actually true about the world. A factual model gets facts, numbers, names and dates right and does not make things up., which concerns agreement with the world rather than with the source.
- 3
Chain of thought
Whether a chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate. or other self-explanation written by a language model reflects the computation that actually produced its answer, so that reading or monitoring that reasoning tells overseers what really drove the output. Studies have found stated reasoning that omits decisive cues, such as hints planted in the prompt; see chain-of-thought faithfulnesschain-of-thought faithfulness, n.: Whether the reasoning an AI writes out really reflects how it reached its answer, rather than being an after-the-fact story that leaves out what actually drove the decision..
Origin
Fidelity was long used for rule extraction and surrogate models. In NLP, Alon Jacovi and Yoav Goldberg separated faithfulness from plausibility in 'Towards Faithfully Interpretable NLP Systems' (ACL 2020); the same year Joshua Maynez and colleagues made faithfulness central to evaluating abstractive summarization.
Usage note
A plausible explanation is not necessarily faithful. People find explanations convincing when they match expectations, which is exactly when an unfaithful explanation is most dangerous.
Related
- explainabilityexplainability, n.: How well the reasons behind an AI system's outputs can be put into terms that people can understand, check and, if needed, challenge.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- chain-of-thought faithfulnesschain-of-thought faithfulness, n.: Whether the reasoning an AI writes out really reflects how it reached its answer, rather than being an after-the-fact story that leaves out what actually drove the decision.
- hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true.
- post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable.
- surrogate modelsurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply.
- factualityfactuality, n.: How far what an AI says is actually true about the world. A factual model gets facts, numbers, names and dates right and does not make things up.
Sources
- Jacovi, A. & Goldberg, Y. (2020). Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? Proceedings of ACL 2020.
- DeYoung, J. et al. (2020). ERASER: A Benchmark to Evaluate Rationalized NLP Models. Proceedings of ACL 2020.
Mentioned in
explainabilityexplainability, n.: How well the reasons behind an AI system's outputs can be put into terms that people can understand, check and, if needed, challenge., feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable., saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye., attention-based explanationattention-based explanation, n.: Explaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output., guided backpropagationguided backpropagation, n.: A way of producing sharp, detailed pictures of which parts of an image a vision network responded to, by passing only positive signals backward through the network., SmoothGradSmoothGrad, n.: A trick for making explanation heat maps less speckled: compute the map for many noisy copies of the input and average them, so random flicker cancels out.