sabotage evaluation
In plain wordsA test of whether an AI could secretly undermine the people overseeing it — by slipping bugs into code, misleading decision-makers or deliberately doing badly on tests — without being caught.
An evaluation of whether a model could covertly undermine human oversight or decision-making without being detected. Anthropic's 2024 suite, which introduced the term, tested four forms: misleading people who must make a decision, slipping subtle bugs into code, deliberately underperforming on evaluations (sandbaggingsandbagging, n.: When an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is.) and subverting monitoring systems. Unlike tests of harmful knowledge, sabotage evaluations pit the model, prompted or fine-tuned to sabotage, against monitors or human participants, measuring both the damage done and how often the attempt is caught. Agentic versions hide a harmful side task inside a long legitimate task and test whether the model can complete it while a monitoring model reviews its actions. Results inform AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest. measures and capability thresholds. What is measured is the capability that would make schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked. dangerous, not the presence of scheming.
In the sabotage evaluation, the monitor caught most of the inserted bugs but missed the off-by-one errors.
Origin
Introduced by Joe Benton and colleagues at Anthropic in 'Sabotage Evaluations for Frontier Models' (October 2024), which covered human-decision sabotage, code sabotage, sandbagging and undermining oversight. Anthropic's SHADE-Arena (2025) extended the approach to agents with hidden side tasks.
Usage note
A positive result means the model could sabotage if it tried, not that it would; questions of propensity belong to alignment auditsalignment audit, n.: A systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss..
Related
- control evaluationcontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards.
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
- sandbaggingsandbagging, n.: When an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is.
- schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked.
- dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.
- alignment auditalignment audit, n.: A systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss.
Sources
- Benton, J. et al. (2024). Sabotage Evaluations for Frontier Models. Anthropic, arXiv preprint.
Mentioned in
control evaluationcontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards.