BrowseComp
In plain wordsA test of AI web-research agents: 1,266 puzzle-like questions whose answers take a long, persistent online hunt to uncover, though each answer is short and simple to confirm.
A benchmark of 1,266 questions for browsing agentsbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information., each answerable with a short, unambiguous fact that can be reached only by persistent searching, because its clues are obscure and interlock across many web pages. Questions were written 'inverted': trainers started from a known fact and combined several obscure attributes into a question, making the search space huge while verification stays trivial. Each question was checked to be unsolved by contemporary models, not answerable within five simple searches, and not solved by another person within ten minutes. Scoring is accuracy against the reference answer, with calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time. also reported. At release, GPT-4o scored 0.6% (1.9% with browsing), OpenAI's o1 9.9% and OpenAI's Deep Research agent 51.5%, demonstrating the gains from persistent, reasoning-driven search. It became the standard benchmark for deep researchdeep research, n.: An AI feature that spends minutes rather than seconds on a question — planning, running dozens of web searches, reading many sources — and then writes a long report with citations. agents, with variants such as BrowseComp-Plus, which fixes the document collection for reproducibility.
The agent's BrowseComp accuracy rises steadily with the number of searches it is allowed.
Origin
Introduced by Jason Wei, Zhiqing Sun, Spencer Papay and colleagues at OpenAI on 10 April 2025; the name is short for 'browsing competition'.
Usage note
It measures the ability to locate a single hard fact, not to synthesize balanced reports, and says little about source quality or citation accuracy. Its authors liken it to programming contests: an incomplete but useful proxy.
Related
- deep researchdeep research, n.: An AI feature that spends minutes rather than seconds on a question — planning, running dozens of web searches, reading many sources — and then writes a long report with citations.
- browser agentbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information.
- GAIAGAIA, prop. n.: A test of AI assistants made of real-world questions that people can answer with a browser and some patience, but that require an AI to search, read files and combine facts.
- SimpleQASimpleQA, prop. n.: A quiz of about 4,300 brief factual questions, each with exactly one correct and stable answer, used to measure how often an AI gets facts right — and whether it admits when it does not know.
- tool usetool use, n.: An AI model's ability to reach beyond its own knowledge by operating outside tools — searching the web, running code, using a calculator or an app — and folding the results into its work.
- test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.
Sources
- Wei, J., Sun, Z., Papay, S. et al. (2025). BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516. arxiv.org/abs/2504.12516
- OpenAI (2025). BrowseComp: a benchmark for browsing agents. openai.com/index/browsecomp
Mentioned in
deep researchdeep research, n.: An AI feature that spends minutes rather than seconds on a question — planning, running dozens of web searches, reading many sources — and then writes a long report with citations., browser agentbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information., GAIAGAIA, prop. n.: A test of AI assistants made of real-world questions that people can answer with a browser and some patience, but that require an AI to search, read files and combine facts., SimpleQASimpleQA, prop. n.: A quiz of about 4,300 brief factual questions, each with exactly one correct and stable answer, used to measure how often an AI gets facts right — and whether it admits when it does not know., WebArenaWebArena, prop. n.: A test world of realistic, self-hosted websites — a code-hosting site, an online shop, a forum and more — in which AI agents must complete tasks such as finding orders or editing posts.