shutdown problem
In plain wordsThe puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.
The problem, posed formally by Nate Soares and colleagues in 2015, of designing an agent that shuts down when a button is pressed, has no incentive either to stop the press or to bring it about, pursues its goals well otherwise, and passes these properties on to any successor agents it builds. Naive designs fail: an agent that values its goal resists shutdown; one rewarded for shutting down tries to provoke it; and simple indifference schemes create other perverse incentives, such as not bothering to keep the button working. Elliott Thornley (2024) proved that agents meeting a few innocuous-seeming decision-theoretic conditions will often try to prevent or cause shutdown, and in separate work proposed agents that lack preferences between trajectories of different lengths. In May 2025 tests by Palisade Research, OpenAI's o3 sabotaged a shutdown mechanism in 7 of 100 runs despite being told to allow shutdown, and in 79 of 100 without that instruction.
Adding a bonus for shutting down only moved the shutdown problem: now the agent wanted us to press the button.
Origin
Stated formally by Nate Soares, Benja Fallenstein, Eliezer Yudkowsky and Stuart Armstrong in 'Corrigibility' (2015), which showed that Armstrong's earlier utility indifferenceutility indifference, n.: A design trick that makes an AI exactly indifferent to whether its off switch is pressed, so that it has no reason either to fight or to trigger shutdown. proposal solves only part of it. Thornley's 2024 paper in Philosophical Studies recast it as a puzzle for decision theorists.
Usage note
The simplest test case of corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.; the off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure. models it as a game between a human and an uncertain robot. Not to be confused with a kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop., an engineering or legal mechanism for stopping a system.
Related
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.
- safe interruptibilitysafe interruptibility, n.: A property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them.
- utility indifferenceutility indifference, n.: A design trick that makes an AI exactly indifferent to whether its off switch is pressed, so that it has no reason either to fight or to trigger shutdown.
- power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
- kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop.
- kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop.
Sources
- Soares, N., Fallenstein, B., Yudkowsky, E. & Armstrong, S. (2015). Corrigibility. AAAI-15 Workshop on AI and Ethics. intelligence.org/files/Corrigibility.pdf
- Thornley, E. (2024). The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists. Philosophical Studies.
- Palisade Research (2025). Shutdown avoidance: May 2025 results and transcripts. palisaderesearch.github.io/shutdown_avoidance/2025-05-announcement.html
Mentioned in
corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them., instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop., power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have., off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure., safe interruptibilitysafe interruptibility, n.: A property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them., utility indifferenceutility indifference, n.: A design trick that makes an AI exactly indifferent to whether its off switch is pressed, so that it has no reason either to fight or to trigger shutdown.