construct validity
In plain wordsWhether high marks on a test actually reflect the skill the test is named after — for example, whether a 'reasoning' test rewards reasoning rather than memory or test-taking tricks.
A property of a test or metric: its scores warrant conclusions about the theoretical attribute — the construct — they are meant to reflect, not merely about something that happens to correlate with it. Borrowed from psychometrics, the concept is applied to AI benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. whose names make broad claims ('understanding', 'reasoning', 'general intelligence') while their items sample a narrow slice of behavior that may be solvable by memorization, superficial cues or exploitation of the answer format. Threats include a vaguely defined construct, unrepresentative tasks, annotation artifacts that allow shortcuts, data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability., brittle scoring and a gap between test conditions and real use. Evidence for validity comes from agreement with other measures of the same construct (convergent validity), disagreement with measures of unrelated ones (discriminant validity) and the ability to predict real-world outcomes.
The benchmark is highly reliable but has weak construct validity: models ace it by exploiting answer-length cues.
Origin
Lee Cronbach and Paul Meehl set out the concept in a 1955 paper in Psychological Bulletin, following a 1954 American Psychological Association report on test standards. It entered critiques of AI benchmarks around 2021, notably Inioluwa Deborah Raji and colleagues' 'AI and the Everything in the Whole Wide World Benchmark'.
Usage note
Distinct from reliability (whether repeated measurements agree) and from external validity (whether findings carry over to other settings). A benchmark can be perfectly reliable and still invalid for the claim made with it.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.
- data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.
- jagged intelligencejagged intelligence, n.: The uneven pattern of AI abilities: today's systems can solve very hard problems yet fail at some a child could manage, so their skills form a spiky profile rather than a smooth one.
Sources
- Cronbach, L. J. & Meehl, P. E. (1955). Construct Validity in Psychological Tests. Psychological Bulletin, 52(4), 281–302.
- Raji, I. D., Bender, E. M., Paullada, A., Denton, E. & Hanna, A. (2021). AI and the Everything in the Whole Wide World Benchmark. NeurIPS 2021 Datasets and Benchmarks Track.
Mentioned in
benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews., benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress., evaluation awarenessevaluation awareness, n.: When an AI model can tell it is being tested rather than used for real — which may change how it behaves and make test results less trustworthy., prompt sensitivityprompt sensitivity, n.: The way an AI's answers, and its test scores, can change a lot after small changes to a prompt that should not matter — a different word, an extra space or examples in another order., task (machine learning)task (machine learning), n.: The job an AI model is meant to do, such as translating sentences, spotting spam or answering questions, defined by its inputs, its expected outputs and how success is scored., Epoch Capabilities IndexEpoch Capabilities Index, prop. n.: A single number summarizing an AI model's general capability, calculated by the research group Epoch AI by combining its scores on dozens of tests onto one common scale.