explainable AI
In plain wordsThe field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application.
The research field and body of techniques that aim to make the behavior of AI systems understandable to people, so that outputs can be justified, debugged, appropriately trusted and contested. Approaches divide into interpretable modelsinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed. (sparse linear models, short decision trees, generalized additive modelsgeneralized additive model, n.: A prediction model that adds up separate, possibly curved effects of each input, so each input's influence can be drawn as its own graph and read off directly.) and post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable. of opaque ones. Explanations may be locallocal explanation, n.: An explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general. (one prediction) or globalglobal explanation, n.: An explanation of how an AI model behaves overall — which factors matter most across all its decisions, and in what way — rather than why it made one particular decision. (overall behavior), and model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside. or specific to one model class. Standard tools include feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down. (SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players., LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example., integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way., saliency mapssaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.), counterfactual explanationscounterfactual explanation, n.: An explanation that tells someone the smallest change that would have produced a different outcome — for example, that the application would have been accepted if it had listed two more years of work experience., surrogate modelssurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply. and concept-based tests such as TCAVtesting with concept activation vectors, phr.: A method for asking whether a human idea, such as 'wheels', matters to an image model's decision, such as 'car', by finding the direction for that idea inside the model and measuring its effect.. XAI grew up around classifiers for tabular and image data; for large language models, explanation increasingly draws on mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior. and representation engineeringrepresentation engineering, n.: An approach to understanding and controlling AI models by finding where high-level ideas such as honesty or danger show up in their internal activity, then reading or adjusting those signals directly., since a model's own account of its reasoning may not be faithful.
The bank's XAI dashboard showed that debt-to-income ratio drove most of the declined applications.
Origin
Explanation facilities date to 1970s expert systemsexpert system, n.: A program that stores a human specialist's know-how as many if–then rules and uses them to give advice or make decisions in a narrow field, such as diagnosing infections or configuring computers. such as MYCINMYCIN, prop. n.: A 1970s Stanford program that advised doctors on blood infections and antibiotics by applying hundreds of if–then rules from specialists. It performed about as well as experts but was never used on patients., and the Explainable Expert Systems project (Robert Neches, William Swartout and Johanna Moore, 1985). The earliest known use of the phrase 'explainable artificial intelligence' is in a 2004 paper by Michael van Lent, William Fisher and Michael Mancuso on a US Army training simulation; 'XAI' spread through the US DARPA Explainable AI program, announced in 2016 and managed by David Gunning.
Usage note
Terminology is unsettled: many writers treat 'explainable' and 'interpretable' as synonyms, while others reserve 'interpretable' for models understandable by design. Cynthia Rudin argues that for high-stakes decisions an interpretable model beats an explained black box. Legal duties of transparency and explanation appear in the EU AI Act and the GDPR; see right to explanationright to explanation, n.: The idea that a person affected by a decision made or shaped by an algorithm — a refused loan, a rejected job application — is entitled to a meaningful account of why.. Not to be confused with xAIxAI, prop. n.: An artificial intelligence company founded by Elon Musk in 2023 that makes the Grok chatbot. It is now part of Musk's rocket company, SpaceX., the company.
Related
- explainabilityexplainability, n.: How well the reasons behind an AI system's outputs can be put into terms that people can understand, check and, if needed, challenge.
- interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
- post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- interpretable modelinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed.
- right to explanationright to explanation, n.: The idea that a person affected by a decision made or shaped by an algorithm — a refused loan, a rejected job application — is entitled to a meaningful account of why.
- mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
- black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer.
Sources
- van Lent, M., Fisher, W. & Mancuso, M. (2004). An Explainable Artificial Intelligence System for Small-unit Tactical Behavior. Proceedings of IAAI 2004. cdn.aaai.org/IAAI/2004/IAAI04-019.pdf
- Gunning, D. & Aha, D. W. (2019). DARPA's Explainable Artificial Intelligence (XAI) Program. AI Magazine 40(2).
Mentioned in
explainabilityexplainability, n.: How well the reasons behind an AI system's outputs can be put into terms that people can understand, check and, if needed, challenge., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., xAIxAI, prop. n.: An artificial intelligence company founded by Elon Musk in 2023 that makes the Grok chatbot. It is now part of Musk's rocket company, SpaceX., black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer., Clever Hans effectClever Hans effect, n.: When a person or an AI seems to solve a problem but is really picking up on an unintended clue — named after a horse that seemed to do arithmetic but was reading its questioners' body language., DARPADARPA, prop. n.: The US military's research-funding agency, which paid for much of the early development of artificial intelligence and the internet and runs prize competitions such as driverless-car races., feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect., interpretable modelinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed., post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable., right to explanationright to explanation, n.: The idea that a person affected by a decision made or shaped by an algorithm — a refused loan, a rejected job application — is entitled to a meaningful account of why., concept bottleneck modelconcept bottleneck model, n.: A model made to first predict human-understandable concepts — such as 'bone spur present' on an X-ray — and then decide only from those concepts, so people can inspect and correct them., explainable boosting machineexplainable boosting machine, n.: A prediction method for spreadsheet-style data that draws each input's influence as its own readable graph and adds those effects up to make every prediction, usually losing little accuracy compared with opaque methods., influence functioninfluence function, n.: A mathematical shortcut for estimating how much a single training example shaped a model's behavior, for instance how a prediction would change if that example had been left out, without retraining the model., prototypical part networkprototypical part network, n.: An image classifier that explains each decision by pointing to parts of the picture that resemble typical examples it learned — this part looks like that part of a known bird., Rashomon effectRashomon effect, n.: The common situation in which many quite different models fit the same data almost equally well — like witnesses telling different but equally believable stories about the same event., symbolic regressionsymbolic regression, n.: Getting a computer to discover a readable mathematical formula, such as a law of physics, that fits a set of measurements, instead of only fitting numbers into a formula chosen in advance.