spurious correlation
In plain wordsA link in data that fools you: two things move together, yet neither drives the other, because a hidden shared cause, the way the data were collected or luck produced the pattern. AI models readily latch onto such links.
A statistical association that misleads about cause and effect: the variables move together, but intervening on one would not change the other, because the pattern comes from a confounding variableconfounding variable, n.: A hidden third factor that influences two things at once and makes them look linked. Ice-cream sales and drownings both rise in hot weather; neither causes the other., from how cases were selected, from artifacts of data collection, or from chance in small or heavily searched data. Models trained to minimize loss exploit any feature that predicts the label in the training data, spurious ones included — behavior called shortcut learning or the 'Clever Hans' effect. Documented cases include image classifiers keying on backgrounds (cows on grass), hospital-specific markers in chest X-rays, watermarks or text tags in images, and annotation artifacts that let language-understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text. models answer from superficial cues. Such models score well on i.i.d.i.i.d., abbr.: Short for 'independent and identically distributed': every data point is produced the same way, and none influences any other — like repeated rolls of one fair die. test sets but fail under distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.. Remedies include more diverse or counterfactually augmented data, group-robust training, invariance-based objectives, causal inferencecausal inference, n.: Working out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time. and inspecting models with interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box. tools.
The pneumonia classifier had learned a spurious correlation with the portable scanners used in intensive care.
Origin
Karl Pearson coined 'spurious correlation' in 1897 for correlations between ratios that share a common denominator, and statisticians soon applied it to any misleading association. In machine learning the problem was popularized as 'Clever Hans' behavior (Sebastian Lapuschkin and colleagues, 2019), after a horse that seemed to do arithmetic by reading its trainer's cues, and as 'shortcut learning' (Robert Geirhos and colleagues, 2020).
Related
- correlationcorrelation, n.: A measure of how closely two quantities move together, from −1 (one rises as the other falls) through 0 (no straight-line link) to +1 (they rise together). It does not show that one causes the other.
- confounding variableconfounding variable, n.: A hidden third factor that influences two things at once and makes them look linked. Ice-cream sales and drownings both rise in hot weather; neither causes the other.
- distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
- causal inferencecausal inference, n.: Working out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time.
- out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable.
- Simpson's paradoxSimpson's paradox, n.: A statistics trap in which combining groups reverses what each group shows: a treatment can do better than another for men and for women separately, yet worse when everyone is counted together.
Sources
- Geirhos, R. et al. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2.
Mentioned in
correlationcorrelation, n.: A measure of how closely two quantities move together, from −1 (one rises as the other falls) through 0 (no straight-line link) to +1 (they rise together). It does not show that one causes the other., causal inferencecausal inference, n.: Working out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time., confounding variableconfounding variable, n.: A hidden third factor that influences two things at once and makes them look linked. Ice-cream sales and drownings both rise in hot weather; neither causes the other., counterfactualcounterfactual, n.: A statement about what would have happened if something had been different — 'had I left earlier, I would have caught the train'. AI uses counterfactuals to explain decisions and to measure causes., Simpson's paradoxSimpson's paradox, n.: A statistics trap in which combining groups reverses what each group shows: a treatment can do better than another for men and for women separately, yet worse when everyone is counted together.