Terminal-Bench
In plain wordsA test of AI agents working in a computer's command line — building software, analyzing data, securing or repairing systems — with each task checked automatically inside a sealed-off container.
A benchmark and evaluation framework for AI agents that work through a command-line terminal. Each task packages a natural-language instruction, a Docker environment, a human-verified reference solution and test scripts that check the container's final state, so agents are judged on outcomes rather than on the commands they type. Tasks range from scientific workflows, network configuration and data analysis to games and cybersecurity. The reference agent, Terminus, acts only by sending keystrokes to a tmux session, giving a neutral baseline across models. Results are reported as the task resolution rate of an agent–model pair, so harnessharness, n.: The software around an AI model that gives it instructions and tools, carries out its actions and decides when it stops. The model is the engine; the harness is the rest of the car. design matters as much as the model. Successive versions — 2.0 (November 2025), 3.0 (July 2026) and 4.0 (August 2026) — added harder tasks, repaired faulty ones and removed saturated ones; 2.0 also introduced Harbor, a framework for running containerized agent evaluations and training rollouts at scale. At 3.0's launch, one leading agent fell from 83.8% on version 2.1, a corrected 2.0, to 33.8% on 3.0's 74 tasks.
Swapping our agent's scaffold for a plain tmux loop cost eight points on Terminal-Bench.
Origin
Released on 19 May 2025 by a team from Stanford University and the Laude Institute — Mike Merrill, Alex Shaw, Chris Rytting, Ludwig Schmidt and Andy Konwinski — with 80 tasks in its first version, Terminal-Bench-Core v0.
Usage note
Leaderboard entries pair a harness with a model, so rankings conflate the two; the same model can place very differently in different harnesses.
Related
- coding agentcoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed.
- SWE-benchSWE-bench, prop. n.: A test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents.
- harnessharness, n.: The software around an AI model that gives it instructions and tools, carries out its actions and decides when it stops. The model is the engine; the harness is the rest of the car.
- sandboxsandbox, n.: A sealed-off computer space where an AI agent can run programs and change files without being able to harm the real system, reach private data or contact the outside world unless allowed.
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically.
- OSWorldOSWorld, prop. n.: A test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks.
Sources
- Merrill, M., Shaw, A., Rytting, C., Schmidt, L. & Konwinski, A. (2025). Terminal-Bench (announcement). tbench.ai/news/announcement
- Merrill, M. & Shaw, A. (2025). Terminal-Bench 2.0 and Harbor (announcement). tbench.ai/news/announcement-2-0
- Marten, R., Shaw, A. & Konwinski, A. (2026). Terminal-Bench 3.0 (announcement). tbench.ai/news/terminal-bench-3-0
Mentioned in
coding agentcoding agent, n.: An AI agent that writes and fixes software by itself: it reads a codebase, edits files, runs commands and tests, and keeps working until a feature is built or a bug is fixed., SWE-benchSWE-bench, prop. n.: A test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents., OSWorldOSWorld, prop. n.: A test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks.