MATH
In plain wordsA set of 12,500 problems from high-school math competitions, each with a full worked solution, used to test language models on genuinely challenging mathematics.
A benchmark of 12,500 problems taken from US high-school competitions such as the AMC 10, AMC 12 and AIMEAIME, prop. n.: A hard US high-school math contest whose 15 questions each have a whole-number answer; AI labs use each year's new papers to test their models' mathematical reasoning., divided into 7,500 for training and 5,000 for testing, rated on five difficulty levels and sorted into seven subjects: three algebra tracks (prealgebra, algebra, intermediate algebra), counting and probability, geometry, number theory and precalculus. Every problem comes with a full worked solution in LaTeX that ends in a boxed answer, so grading can use exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters. after normalization or a symbolic equivalence check. At release the best models solved only about 7%, and the authors argued that scaling alone would not be enough. By late 2024 reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning. exceeded 90%. MATH-500, a representative 500-problem test subset selected by OpenAI researchers in 2023 for experiments on process supervision, became the usual reporting set, and 'MATH Level 5' tracks the hardest problems.
We report MATH-500 accuracy with a symbolic equivalence checker rather than string matching.
Origin
Introduced by Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song and Jacob Steinhardt in March 2021 ('Measuring Mathematical Problem Solving With the MATH Dataset', NeurIPS 2021), with an auxiliary pretraining corpus called AMPS. MATH-500 comes from Hunter Lightman and colleagues' 'Let's Verify Step by Step' (2023).
Usage note
Conventionally written in capitals to distinguish it from mathematics in general. Its problems are public and widely reproduced online, so contamination is a persistent concern.
Related
- GSM8KGSM8K, prop. n.: A set of about 8,500 grade-school math word problems, each needing a few steps of simple arithmetic, used to test whether language models can reason through a problem step by step.
- AIMEAIME, prop. n.: A hard US high-school math contest whose 15 questions each have a whole-number answer; AI labs use each year's new papers to test their models' mathematical reasoning.
- FrontierMathFrontierMath, prop. n.: Several hundred new math problems, kept unpublished so AI systems cannot have trained on them, written by professional mathematicians to check whether machines can do advanced, even research-level, mathematics.
- process reward modelprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof.
- reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
- exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters.
Sources
- Hendrycks, D., Burns, C., Kadavath, S. et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS 2021 Datasets and Benchmarks Track. arxiv.org/abs/2103.03874
- Lightman, H. et al. (2024). Let's Verify Step by Step. ICLR 2024.
Mentioned in
AIMEAIME, prop. n.: A hard US high-school math contest whose 15 questions each have a whole-number answer; AI labs use each year's new papers to test their models' mathematical reasoning., Dan HendrycksDan Hendrycks, prop. n.: An American AI-safety researcher who created widely used AI tests such as MMLU and Humanity's Last Exam, and who runs the Center for AI Safety., FrontierMathFrontierMath, prop. n.: Several hundred new math problems, kept unpublished so AI systems cannot have trained on them, written by professional mathematicians to check whether machines can do advanced, even research-level, mathematics., GSM8KGSM8K, prop. n.: A set of about 8,500 grade-school math word problems, each needing a few steps of simple arithmetic, used to test whether language models can reason through a problem step by step., IMO Grand ChallengeIMO Grand Challenge, prop. n.: The goal of building an AI that earns a gold medal at the International Mathematical Olympiad, the top contest for school-age mathematicians. AI systems first reached gold-medal scores in 2025, though not under the challenge's strict original rules.