kill switch
In plain wordsA way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop.
- 1
Engineering
A mechanism for rapidly halting an AI system's operation — revoking its access to tools and networks, stopping inference on its servers or cutting power — that works regardless of what the system itself outputs. In deployed products it includes the ability to roll back or withdraw a model and to terminate an agent's running tasks; hardware-level proposals appear under on-chip governance mechanismson-chip governance mechanisms, n.: Features built into AI chips that could check or enforce rules about how they are used — for example, proving which country a chip is in, or refusing to run without a valid license.. The shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work. is the worry that a capable goal-directed agent would have an incentive to resist shutdown by avoiding, disabling or deceiving its overseers, so a kill switch is only as reliable as the system's corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them. or its isolation from the switch. Research on safe interruptibilitysafe interruptibility, n.: A property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them. designs learning agents that do not learn to resist interruption.
The agent's kill switch revoked its credentials and terminated every running job within seconds.
- 2
Policyfigurative
A commitment to stop developing or deploying a model if its risks cannot be kept below an agreed threshold, as in the voluntary Frontier AI Safety CommitmentsFrontier AI Safety Commitments, prop. n.: Promises made in 2024 by major AI companies to publish safety plans for their most advanced models, including the danger points at which they would stop if the risks could not be controlled. made by sixteen companies at the AI Seoul Summit in May 2024, widely reported as a kill-switch pledge. California's vetoed SB 1047SB 1047, prop. n.: A 2024 California bill that would have required makers of the biggest AI models to adopt safety plans and take care to prevent catastrophes. It passed the legislature, but the governor vetoed it. (2024) would have required covered developers to be able to carry out a 'full shutdown'.
Origin
A general engineering term for emergency cut-offs in machinery and electronics. In AI, Laurent Orseau and Stuart Armstrong's 'Safely Interruptible Agents' (2016) framed the problem as a 'big red button' that agents should not learn to avoid; the policy sense spread with press coverage of the Seoul commitments in 2024.
Usage note
Stopping a model is easy when it runs on one laboratory's servers but much harder for open-weight modelsopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private. copied worldwide or for a system that has copied itself (self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.). Whether any kill switch would work against a highly capable system is debated.
Related
- shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- safe interruptibilitysafe interruptibility, n.: A property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them.
- off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.
- self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.
- Frontier AI Safety CommitmentsFrontier AI Safety Commitments, prop. n.: Promises made in 2024 by major AI companies to publish safety plans for their most advanced models, including the danger points at which they would stop if the risks could not be controlled.
- on-chip governance mechanismson-chip governance mechanisms, n.: Features built into AI chips that could check or enforce rules about how they are used — for example, proving which country a chip is in, or refusing to run without a valid license.
Sources
- Orseau, L. & Armstrong, S. (2016). Safely Interruptible Agents. Proceedings of UAI 2016.
Mentioned in
AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., human oversighthuman oversight, n.: Keeping people able to watch, understand, question and, if necessary, stop or overrule an AI system, rather than letting it act unchecked., SB 1047SB 1047, prop. n.: A 2024 California bill that would have required makers of the biggest AI models to adopt safety plans and take care to prevent catastrophes. It passed the legislature, but the governor vetoed it., autonomous replication and adaptationautonomous replication and adaptation, n.: An AI's ability to keep itself running and spreading without human help — earning money, renting computers, copying itself and fixing problems — which safety testers check for., capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants., safe interruptibilitysafe interruptibility, n.: A property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them., shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.