benchmark
In plain wordsA fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- 1
Machine learning
A standardized evaluation consisting of a dataset or task collection, a protocol for running a system on it, and a scoring rule, published so that different systems can be compared on equal terms. A benchmark typically fixes a test settest set, n.: A set of examples kept completely apart from training and used at the end to measure how well a model works on data it has never seen, like a sealed final exam. (often with hidden answers), a metric — accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others., exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters., pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code., task resolution rate — and sometimes the resources allowed, and collects results on a leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models.. Landmarks include ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom. for vision, GLUEGLUE, prop. n.: A 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT. for language understanding, MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge. for knowledge and SWE-benchSWE-bench, prop. n.: A test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents. for software engineering. Benchmarks focus research effort but invite tuning to their quirks (Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.), leakage of test items into training data (data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.) and rapid benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.; whether a score reflects the capability a benchmark is named after is a question of construct validityconstruct validity, n.: Whether high marks on a test actually reflect the skill the test is named after — for example, whether a 'reasoning' test rewards reasoning rather than memory or test-taking tricks..
The new model tops three reasoning benchmarks but still fails our internal tests.
- 2
Computing performance
A program or workload run to measure the speed, throughput or energy efficiency of hardware or software, such as MLPerfMLPerf, prop. n.: A standard set of speed tests for AI hardware and software, where chip makers and cloud companies compare how fast their systems train and run AI models. for AI training and inference systems. This older engineering sense predates the machine-learning one; both derive from the surveyor's bench mark, a fixed reference point.
Origin
From the surveyor's 'bench mark', a reference mark cut into stone for leveling, the word passed into computing for performance tests by the 1970s. In machine learning, shared datasets such as the UCI repository (1987), MNISTMNIST, prop. n.: A famous collection of 70,000 small images of handwritten digits used to test image-recognition programs. Getting a model to read MNIST digits is the 'hello world' of machine learning. and ImageNet spread the 'common task framework' — shared data, a fixed metric and public comparison — which David Donoho, following Mark Liberman, traced to DARPA's speech and language evaluations of the 1980s. For language models, GLUE (2018) opened the multi-task benchmark era.
Usage note
A benchmark is a measuring instrument, not a goal: once it is optimized directly, scores can rise without the underlying ability improving (see benchmaxxingbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks.). Scores are also sensitive to prompt format and evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically., so numbers from different sources are often not comparable.
Related
- evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews.
- leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.
- construct validityconstruct validity, n.: Whether high marks on a test actually reflect the skill the test is named after — for example, whether a 'reasoning' test rewards reasoning rather than memory or test-taking tricks.
- contamination-resistant benchmarkcontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made.
- evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically.
- human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well.
Sources
- Raji, I. D., Bender, E. M., Paullada, A., Denton, E. & Hanna, A. (2021). AI and the Everything in the Whole Wide World Benchmark. NeurIPS 2021 Datasets and Benchmarks Track.
- Donoho, D. (2017). 50 Years of Data Science. Journal of Computational and Graphical Statistics, 26(4).
Mentioned in
accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others., ARC-AGIARC-AGI, prop. n.: Puzzles made of colored grids — and, since 2026, small video-game-like worlds — that people solve easily but AI long found hard, designed to test whether a machine can learn a new skill from just a few examples., correlationcorrelation, n.: A measure of how closely two quantities move together, from −1 (one rises as the other falls) through 0 (no straight-line link) to +1 (they rise together). It does not show that one causes the other., datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system., evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews., Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%., leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models., overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them., regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known., test settest set, n.: A set of examples kept completely apart from training and used at the end to measure how well a model works on data it has never seen, like a sealed final exam., baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value., benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress., BIG-benchBIG-bench, prop. n.: A huge collaborative test of 204 tasks, written by hundreds of researchers, built to probe what language models can and cannot do as they grow larger., bootstrapbootstrap, n.: Estimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data., capabilitycapability, n.: Something an AI system is able to do, such as writing working code or solving math problems, and how well. Measuring capabilities is how researchers track progress and spot dangerous skills., central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like., Clever Hans effectClever Hans effect, n.: When a person or an AI seems to solve a problem but is really picking up on an unintended clue — named after a horse that seemed to do arithmetic but was reading its questioners' body language., COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems., confidence intervalconfidence interval, n.: A range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample., construct validityconstruct validity, n.: Whether high marks on a test actually reflect the skill the test is named after — for example, whether a 'reasoning' test rewards reasoning rather than memory or test-taking tricks., data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability., evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically., Fréchet inception distanceFréchet inception distance, n.: A score for image generators that compares the overall statistics of their pictures with those of real photos, as seen by an image-recognition network; lower is better.