Clever Hans effect
In plain wordsWhen a person or an AI seems to solve a problem but is really picking up on an unintended clue — named after a horse that seemed to do arithmetic but was reading its questioners' body language.
The appearance of a cognitive ability that is actually produced by sensitivity to unintended cues. The original Clever Hans, a horse exhibited in Berlin in the early 1900s, seemed to answer arithmetic questions by tapping his hoof, but was responding to involuntary changes in his questioners' posture and expression as he reached the right count. In machine learning the term names models that achieve high accuracy by exploiting spurious correlations in the data — a watermark present in images of one class, hospital-specific markers in medical scans, annotation artifacts in language datasets — instead of the intended features, a failure also called shortcut learning; such models fail when the cue is absent. It is also applied to evaluations in which a language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.'s apparent reasoning or theory of mindtheory of mind, n.: Being able to work out what someone else believes, wants or knows — even when it is not what you believe — and to use that to guess what they will do next. Researchers test whether AI has it. reflects exploitable regularities in benchmark items or data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability..
The classifier was a Clever Hans: it detected the hospital's scanner tag, not pneumonia.
Origin
Named after the horse Clever Hans, investigated in late 1904 by the psychologist Oskar Pfungst, whose controlled experiments exposed the cueing and appeared as a book in 1907; the case became a byword for experimenter effects. Sebastian Lapuschkin and colleagues' 'Unmasking Clever Hans Predictors and Assessing What Machines Really Learn' (Nature Communications, 2019) popularized the term in machine learning.
Usage note
Explanation methods such as saliency mapssaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye. can expose Clever Hans behavior by showing which input features a model relies on.
Related
- explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application.
- saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.
- generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers.
- data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- anthropomorphismanthropomorphism, n.: Treating something that isn't human — a pet, a car, a chatbot — as if it had human thoughts, feelings or intentions. AI that talks fluently makes this temptation unusually strong.
Sources
- Pfungst, O. (1907). Das Pferd des Herrn von Osten (Der kluge Hans). Johann Ambrosius Barth.
- Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W. & Müller, K.-R. (2019). Unmasking Clever Hans Predictors and Assessing What Machines Really Learn. Nature Communications 10.
Mentioned in
theory of mindtheory of mind, n.: Being able to work out what someone else believes, wants or knows — even when it is not what you believe — and to use that to guess what they will do next. Researchers test whether AI has it., layer-wise relevance propagationlayer-wise relevance propagation, n.: A method that traces a neural network's decision backward, layer by layer, handing each part of the input its share of the credit until every pixel or word has a relevance score., machine psychologymachine psychology, n.: Studying AI systems the way psychologists study people — with carefully designed experiments on their behavior — to discover what they can do, how they reason and where they go wrong., Potemkin understandingPotemkin understanding, n.: When an AI can correctly explain a concept yet cannot use it properly, revealing that its apparent understanding is a facade — like the fake village fronts said to have been built to impress a Russian empress.