MMLU
In plain wordsA giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge.
A knowledge benchmark of 15,908 four-option multiple-choice questions spread across 57 subjects in STEM, the humanities, the social sciences and professional fields such as law and medicine, pitched at levels from elementary school to professional practice. It is usually scored by accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others. under five-shot or zero-shotzero-shot learning, n.: Doing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training. prompting, against an estimated expert-level human accuracy of about 89.8%. When it appeared, the largest GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming. model managed 43.9% while most others hovered at the 25% expected from guessing; it then served as the headline knowledge score in model announcements from 2023 until results of about 88–90% by mid-2024 left little headroom. A 2024 audit estimated that about 6.5% of questions contain errors, with much higher rates in some subjects (57% in virology). Successors include MMLU-Pro (ten answer options and more reasoning-heavy questions) and professionally translated multilingual versions (MMMLU).
Their small model reports 80 on MMLU, but on MMLU-Pro it trails models half its size.
Origin
Introduced by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song and Jacob Steinhardt in 'Measuring Massive Multitask Language Understanding', posted to arXiv in September 2020 and presented at ICLR 2021.
Usage note
Given its error rate, small differences near the top are mostly noise, and scores vary with prompt format and answer extraction. By 2025 MMLU had largely been dropped from frontier model reports in favor of harder tests such as GPQAGPQA, prop. n.: A set of very hard multiple-choice science questions written by PhD-level experts, designed so that looking things up online does not help much: capable non-specialists with web access still mostly fail. and Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%..
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- GPQAGPQA, prop. n.: A set of very hard multiple-choice science questions written by PhD-level experts, designed so that looking things up online does not help much: capable non-specialists with web access still mostly fail.
- Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%.
- data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.
- few-shot promptingfew-shot prompting, n.: Showing an AI a handful of examples of a task — inputs with the answers you want — inside the prompt, so it can copy the pattern for a new case.
Sources
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. & Steinhardt, J. (2021). Measuring Massive Multitask Language Understanding. ICLR 2021. arxiv.org/abs/2009.03300
- Gema, A. P. et al. (2024). Are We Done with MMLU? arXiv preprint.
Mentioned in
benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%., benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress., Center for AI SafetyCenter for AI Safety, prop. n.: A San Francisco nonprofit that studies how advanced AI could lead to disaster and how to prevent it. In 2023 it gathered signatures for a one-sentence warning that AI could threaten human extinction., Dan HendrycksDan Hendrycks, prop. n.: An American AI-safety researcher who created widely used AI tests such as MMLU and Humanity's Last Exam, and who runs the Center for AI Safety., GPQAGPQA, prop. n.: A set of very hard multiple-choice science questions written by PhD-level experts, designed so that looking things up online does not help much: capable non-specialists with web access still mostly fail., HELMHELM, prop. n.: A large, open testing framework from Stanford that grades many language models on many kinds of tasks and qualities — not only accuracy but also fairness, toxicity and efficiency., MMMUMMMU, prop. n.: A test for AI models that understand pictures as well as text: 11,500 college-level questions across 30 subjects, each built around one or more images such as a diagram, a musical score or a chemical structure., SuperGLUESuperGLUE, prop. n.: A harder follow-up to the GLUE language-understanding test, released in 2019 after AI systems had already beaten the human baseline on the original., WMDPWMDP, prop. n.: A multiple-choice test of AI knowledge that could help someone make biological, chemical or cyber weapons, built from questions near the dangerous line but not over it, and used to test ways of erasing such knowledge.