leaderboard
In plain wordsA public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models.
A ranked table of results on one or more benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. or comparison platforms, listing each system's score and usually its submitter, date and evaluation conditions. Leaderboards may accept self-reported results, rerun submitted models under a standard evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically., or compute rankings from live comparisons, as LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings. does from human votes using a Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths.. They make progress visible and competition concrete, but they reward small and sometimes statistically insignificant margins, encourage tuning to the test, and can be distorted by selective disclosure — for example when developers privately test many variants and publish only the best. Good practice reports confidence intervals, verifies submissions, records the scaffold or harness used, and keeps part of the test data hidden.
It went from fifth to first on the leaderboard, but the confidence intervals of the top four overlap.
Origin
Public machine-learning leaderboards were popularized by the Netflix PrizeNetflix Prize, prop. n.: A 2006–2009 contest in which Netflix offered a million dollars to anyone who could predict customers' movie ratings 10 percent better than its own system. It popularized machine-learning competitions and model blending. (2006–2009) and KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science. competitions (from 2010), then by academic benchmarks such as SQuAD (2016) and GLUE (2018) that hosted ranked result tables. Chatbot Arena (2023) and Hugging Face's Open LLM Leaderboard (2023–2025) made them central to the language-model era.
Usage note
Rank is not fitness for a particular use: a model can lead an aggregate table while trailing on the tasks a given user cares about. The 2025 paper 'The Leaderboard Illusion' argued that private testing and unequal data access had systematically favored large developers on Chatbot Arena.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.
- evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- benchmaxxingbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks.
- KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science.
- Elo ratingElo rating, n.: A number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one.
Sources
- Singh, S., Nan, Y. et al. (2025). The Leaderboard Illusion. arXiv:2504.20879. arxiv.org/abs/2504.20879
Mentioned in
benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., Hugging FaceHugging Face, prop. n.: A company whose website is the main public library for sharing AI models and datasets — often called the GitHub of machine learning — and which makes widely used free AI software., LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings., Elo ratingElo rating, n.: A number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one., evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically., GLUEGLUE, prop. n.: A 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT., KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science., MTEBMTEB, prop. n.: A large public test suite that scores AI models which turn text into lists of numbers on many different jobs, such as searching, grouping, classifying and matching translations, so people can compare them on one leaderboard., state of the artstate of the art, n.: The best performance or most advanced technique currently known in a field. In AI papers, a state-of-the-art result is one that beats every earlier published result on a test., validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one., vibe checkvibe check, n.: Trying out a new AI model informally — chatting with it, asking your favorite test questions — to get a gut feeling for how good it is, rather than measuring it with formal tests., benchmaxxingbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks., GPU poorGPU poor, adj.: Slang for researchers, startups or countries without access to the huge numbers of expensive chips needed to build the most advanced AI, as opposed to the GPU-rich big labs., Papers with CodePapers with Code, prop. n.: A free website, run by Meta from 2019 until it closed in 2025, that linked AI research papers to their computer code and kept leaderboards of the best results.