SimpleQA
In plain wordsA quiz of about 4,300 brief factual questions, each with exactly one correct and stable answer, used to measure how often an AI gets facts right — and whether it admits when it does not know.
A benchmark of 4,326 brief factual questions that tests the short-form factualityfactuality, n.: How far what an AI says is actually true about the world. A factual model gets facts, numbers, names and dates right and does not make things up. of language models answering from memory; every question has one correct answer that is beyond dispute and should stay true over time. Topics include science, politics, art, sport and geography. To make the set hard, AI trainers wrote questions that contemporary GPT models got wrong; a second trainer then answered each independently, and only questions on which the two agreed were kept, for an estimated error rate of about 3%. A model-based grader sorts every response into correct, incorrect or not attempted, so a model can earn credit for abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake. when unsure, not only for accuracy, and comparing stated confidence with accuracy yields a measure of calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.. At release GPT-4o answered fewer than 40% correctly. Because the answers are easy to look up, SimpleQA measures parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.; with web search enabled, scores approach the ceiling.
The smaller model gets fewer SimpleQA questions right but abstains far more often, so it hallucinates less.
Origin
Introduced by Jason Wei, Karina Nguyen, Hyung Won Chung and colleagues at OpenAI on 30 October 2024 in 'Measuring short-form factuality in large language models'. Google DeepMind later released a cleaned 1,000-question version, SimpleQA Verified (2025).
Usage note
It measures recall of isolated facts, not the accuracy of long answers; a model can score well on SimpleQA and still hallucinatehallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true. in long-form writing.
Related
- factualityfactuality, n.: How far what an AI says is actually true about the world. A factual model gets facts, numbers, names and dates right and does not make things up.
- hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true.
- abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake.
- calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
- TruthfulQATruthfulQA, prop. n.: A test of whether AI models repeat popular myths and misconceptions — questions to which many people, and much of the internet, give a confidently wrong answer.
- parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.
- BrowseCompBrowseComp, prop. n.: A test of AI web-research agents: 1,266 puzzle-like questions whose answers take a long, persistent online hunt to uncover, though each answer is short and simple to confirm.
Sources
- Wei, J., Nguyen, K., Chung, H. W. et al. (2024). Measuring short-form factuality in large language models. arXiv:2411.04368. arxiv.org/abs/2411.04368
- OpenAI (2024). Introducing SimpleQA. openai.com/index/introducing-simpleqa
Mentioned in
abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake., BrowseCompBrowseComp, prop. n.: A test of AI web-research agents: 1,266 puzzle-like questions whose answers take a long, persistent online hunt to uncover, though each answer is short and simple to confirm., factualityfactuality, n.: How far what an AI says is actually true about the world. A factual model gets facts, numbers, names and dates right and does not make things up., TruthfulQATruthfulQA, prop. n.: A test of whether AI models repeat popular myths and misconceptions — questions to which many people, and much of the internet, give a confidently wrong answer.