shortcut learning
In plain wordsWhen an AI system gets right answers for the wrong reasons — noticing which hospital an X-ray came from rather than signs of disease — so it fails once that accidental clue disappears.
The tendency of learning systems to latch onto spurious correlations — cues that happen to predict the label in the available training and test data but have no causal or robust link to the task — so that a model can look excellent on standard benchmarks and then fail as soon as distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested. removes the cue. Examples include image classifiers that recognize cows by grassy backgrounds, pneumonia detectors that learned hospital-specific markers on X-rays, and natural-language-inference models that predict the answer from words in the hypothesis alone, an artifact of how annotators wrote the data. Shortcuts arise because gradient-based training favors the simplest predictive features available and because data collected in one way carry systematic artifacts. Remedies include counterfactual or adversarially collected data, removing known artifacts, objectives that penalize reliance on spurious features, evaluation on shifted data, and interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box. tools that reveal what a model relies on.
The 'melanoma detector' had learned to spot rulers in the photos — classic shortcut learning.
Origin
The phenomenon was long known as the 'Clever Hans' effect, after a horse in early-1900s Berlin that seemed to do arithmetic but was reading its questioner's cues; Sebastian Lapuschkin and colleagues wrote of 'Clever Hans predictors' in 2019, and Robert Geirhos and colleagues' 2020 perspective in Nature Machine Intelligence popularized 'shortcut learning' as an umbrella term and surveyed its forms.
Usage note
Related to goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine. in reinforcement learning, where an agent's learned objective tracks a proxy that coincided with the intended goal during training. A shortcut is a property of data and model together, not of the model alone.
Related
- distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
- out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable.
- goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine.
- robustnessrobustness, n.: An AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it.
- interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
- data qualitydata quality, n.: How well a set of data does its job: whether it is accurate, complete, up to date, consistent and representative — and, for AI training, informative enough to teach useful skills.
Sources
- Geirhos, R. et al. (2020). Shortcut Learning in Deep Neural Networks. Nature Machine Intelligence 2, 665–673. arXiv:2004.07780. arxiv.org/abs/2004.07780
- Gururangan, S. et al. (2018). Annotation Artifacts in Natural Language Inference Data. NAACL 2018.