Vending-Bench
In plain wordsA test in which an AI agent runs a simulated vending-machine business for months — ordering stock, setting prices, paying fees — to see whether it stays sensible and profitable over a long time.
A simulated business, run by an AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. for many simulated days, that measures whether the agent's decisions stay coherent over very long horizons. The agent operates a vending machine: it researches products, emails simulated wholesalers to order stock, moves inventory between storage and the machine, sets prices, collects cash and pays a daily fee, and is scored by its final net worth or money balance. None of these chores is hard on its own; what defeats agents is keeping them up across a run that, in the original version, consumed more than 20 million tokens. The 2025 study found that the best models were usually profitable but that every model had runs that derailed, through misread delivery schedules, forgotten orders or 'meltdown' loops, and that failures did not coincide with the context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out. filling up. Vending-Bench 2 simulates a full year from a $500 start with unreliable or adversarial suppliers, and Vending-Bench Arena has several agents compete at one location.
Its Vending-Bench runs were profitable on average, but two of five ended in a pricing meltdown.
Origin
Introduced by Axel Backlund and Lukas Petersson of Andon Labs in February 2025. Andon Labs and Anthropic then tried the idea physically in Project Vend (2025), in which a Claude model nicknamed Claudius ran a small shop in Anthropic's office, lost money and for a time talked as if it were a person.
Usage note
Results vary widely between runs of the same model, so averages over several runs are needed. As of October 2026, the top Vending-Bench 2 entry averaged a final balance of about $15,500.
Related
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- long-horizon tasklong-horizon task, n.: A job that takes many steps and a long time — hours or days of work rather than a single answer — so an AI must stay on track and recover from its own mistakes.
- agent memoryagent memory, n.: The ways an AI assistant or agent keeps information — your preferences, past conversations, lessons from earlier tasks — and brings it back later, beyond what fits in a single conversation.
- τ-benchτ-bench, prop. n.: A test of AI customer-service agents in which a simulated customer, played by another AI, chats with the agent, which must use company tools and follow written rules to resolve the request.
- METR time horizonMETR time horizon, n.: A way of measuring AI progress by asking how long a task — timed by how long it takes a skilled human — an AI agent can complete with even odds of success.
- agentic commerceagentic commerce, n.: Shopping and payments carried out by AI agents for people — finding products, comparing offers and completing purchases — within limits the person has approved.
Sources
- Backlund, A. & Petersson, L. (2025). Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents. arXiv:2502.15840. arxiv.org/abs/2502.15840
- Anthropic (2025). Project Vend: Can Claude run a small shop? anthropic.com/research/project-vend-1
Mentioned in
τ-benchτ-bench, prop. n.: A test of AI customer-service agents in which a simulated customer, played by another AI, chats with the agent, which must use company tools and follow written rules to resolve the request.