SWE-bench
In plain wordsA test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents.
A benchmark that turns resolved issues on GitHub — bug reports and feature requests whose fixes were merged — into tests of automated software engineering. Each of its 2,294 task instances pairs an issue from one of 12 widely used Python repositories (Django, scikit-learn and SymPy among them) with the codebase as it stood before the fix, and the system under test must write a patch that resolves the issue. The patch is judged by the project's own tests: the tests that the human fix made pass must now pass, and previously passing tests must still pass. The score is the percentage of issues resolved. SWE-bench Verified (August 2024), a 500-task subset screened by 93 professional developers to remove underspecified issues and faulty tests, became the industry's headline measure of coding agentscoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed., with scores rising from 33% for GPT-4o to about 81% by early 2026. In February 2026 OpenAI stopped reporting Verified, finding that every frontier model it tested could reproduce gold patches or problem details for some tasks, and that most of the frequently failed tasks it audited had flawed tests or problem descriptions, and recommended the harder, partly private SWE-bench Pro (1,865 tasks from 41 repositories). Other variants cover Lite (300 tasks), multimodal issues and nine further programming languages.
The harness change alone moved our SWE-bench Verified score from 62% to 71%.
Origin
Introduced by Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan, mainly at Princeton University, in October 2023 and published at ICLR 2024; the best model then, Claude 2, resolved 1.96% of issues. Verified was built with OpenAI in 2024, and Scale AI researchers released SWE-bench Pro in September 2025.
Usage note
Scores depend heavily on the agent harnessharness, n.: The software around an AI model that gives it instructions and tools, carries out its actions and decides when it stops. The model is the engine; the harness is the rest of the car. or scaffoldingscaffolding, n.: The code, prompts and tools built around an AI model so it can carry out tasks it could not manage from a bare prompt — like the scaffolding that lets builders reach every part of a building., so one model can differ by many points across submissions. 'Resolved' means passing tests, a weaker standard than writing a patch a maintainer would merge; popular public repositories also invite data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability..
Related
- coding agentcoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed.
- SWE-agentSWE-agent, prop. n.: An open-source research system from Princeton that turns a language model into a software-fixing agent by giving it a specially designed set of commands for browsing, editing and testing code.
- HumanEvalHumanEval, prop. n.: A set of 164 small programming exercises, each described in plain English and checked by hidden tests, used to see whether an AI can write working Python code.
- Terminal-BenchTerminal-Bench, prop. n.: A test of AI agents working in a computer's command line — building software, analyzing data, securing or repairing systems — with each task checked automatically inside a sealed-off container.
- data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- harnessharness, n.: The software around an AI model that gives it instructions and tools, carries out its actions and decides when it stops. The model is the engine; the harness is the rest of the car.
Sources
- Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O. & Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. arxiv.org/abs/2310.06770
- OpenAI (2026). Why we no longer evaluate SWE-bench Verified. openai.com/index/why-we-no-longer-evaluate-swe-bench-verified
- Deng, X. et al. (2025). SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv:2509.16941. arxiv.org/abs/2509.16941
Mentioned in
benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., coding agentcoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed., benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress., code generationcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs., HumanEvalHumanEval, prop. n.: A set of 164 small programming exercises, each described in plain English and checked by hidden tests, used to see whether an AI can write working Python code., Terminal-BenchTerminal-Bench, prop. n.: A test of AI agents working in a computer's command line — building software, analyzing data, securing or repairing systems — with each task checked automatically inside a sealed-off container., agent–computer interfaceagent–computer interface, n.: The commands, tools and feedback designed for an AI agent to work with a computer — the agent's version of a user interface, shaped around how language models read and act., Darwin Gödel MachineDarwin Gödel Machine, prop. n.: An experimental AI coding assistant that improves itself by rewriting its own code, testing each new version and keeping a family tree of variants to build on., DevinDevin, prop. n.: A commercial AI 'software engineer' from the startup Cognition, unveiled in 2024, that takes on programming tasks in its own cloud workspace, complete with a web browser, a code editor and a command line., Remote Labor IndexRemote Labor Index, prop. n.: A test that gives AI agents real paid freelance projects — design, video, data work and more — and checks how many they finish to a standard a client would accept. Early agents managed very few., SWE-agentSWE-agent, prop. n.: An open-source research system from Princeton that turns a language model into a software-fixing agent by giving it a specially designed set of commands for browsing, editing and testing code.