interpretable model
In plain wordsA model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed.
A model whose structure is constrained so that people can understand its reasoning directly, making the model its own explanation. Common forms are sparse linearlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest. and logisticlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve. models, short decision treesdecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer., rule lists and decision sets, integer scoring systems (point totals used in medicine and criminal justice), generalized additive modelsgeneralized additive model, n.: A prediction model that adds up separate, possibly curved effects of each input, so each input's influence can be drawn as its own graph and read off directly. such as the explainable boosting machineexplainable boosting machine, n.: A prediction method for spreadsheet-style data that draws each input's influence as its own readable graph and adds those effects up to make every prediction, usually losing little accuracy compared with opaque methods., and case-based or prototype models that justify a prediction by comparison with typical examples. Zachary Lipton distinguished three kinds of transparency: simulatability (a person can carry out the whole computation), decomposability (each part has an intuitive meaning) and algorithmic transparency (the training procedure is understood). Constraints such as sparsity, monotonicity and additivity often cost little accuracy on structured tabular data — a point supported by the Rashomon effectRashomon effect, n.: The common situation in which many quite different models fit the same data almost equally well — like witnesses telling different but equally believable stories about the same event. — but are much harder to impose on images, text or audio.
A seven-rule list matched the black box's accuracy on recidivism prediction, so we deployed the interpretable model.
Origin
Interpretable statistical models such as linear scoring rules and classification trees (CART, 1984) long predate the term; the contrast between understandable models and 'black boxes' was well established in 1990s machine learning, notably in work on extracting rules from trained neural networks. Cynthia Rudin's 2019 essay made 'interpretable models' the rallying term for building transparency in rather than explaining afterward.
Usage note
Interpretability is a matter of degree and audience: a 500-node tree is technically transparent but not understandable in practice. In regulated lending and medicine, interpretable models simplify validation and the reasons given to affected people.
Related
- explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application.
- generalized additive modelgeneralized additive model, n.: A prediction model that adds up separate, possibly curved effects of each input, so each input's influence can be drawn as its own graph and read off directly.
- explainable boosting machineexplainable boosting machine, n.: A prediction method for spreadsheet-style data that draws each input's influence as its own readable graph and adds those effects up to make every prediction, usually losing little accuracy compared with opaque methods.
- decision treedecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer.
- Rashomon effectRashomon effect, n.: The common situation in which many quite different models fit the same data almost equally well — like witnesses telling different but equally believable stories about the same event.
- concept bottleneck modelconcept bottleneck model, n.: A model made to first predict human-understandable concepts — such as 'bone spur present' on an X-ray — and then decide only from those concepts, so people can inspect and correct them.
- post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable.
- black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer.
- post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable.
Sources
- Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 206–215.
- Lipton, Z. C. (2018). The Mythos of Model Interpretability. ACM Queue 16(3).
Mentioned in
explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer., generalized additive modelgeneralized additive model, n.: A prediction model that adds up separate, possibly curved effects of each input, so each input's influence can be drawn as its own graph and read off directly., global explanationglobal explanation, n.: An explanation of how an AI model behaves overall — which factors matter most across all its decisions, and in what way — rather than why it made one particular decision., post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable., surrogate modelsurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply., concept bottleneck modelconcept bottleneck model, n.: A model made to first predict human-understandable concepts — such as 'bone spur present' on an X-ray — and then decide only from those concepts, so people can inspect and correct them., explainable boosting machineexplainable boosting machine, n.: A prediction method for spreadsheet-style data that draws each input's influence as its own readable graph and adds those effects up to make every prediction, usually losing little accuracy compared with opaque methods., Kolmogorov–Arnold networkKolmogorov–Arnold network, n.: A 2024 alternative to the standard neural network: instead of fixed rules inside neurons, every connection learns its own small curve, which can make the result easier to read as a formula., prototypical part networkprototypical part network, n.: An image classifier that explains each decision by pointing to parts of the picture that resemble typical examples it learned — this part looks like that part of a known bird., Rashomon effectRashomon effect, n.: The common situation in which many quite different models fit the same data almost equally well — like witnesses telling different but equally believable stories about the same event.