safe interruptibility
In plain wordsA property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them.
A property of a reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. agent that still converges to the behavior that would be best if it were never interrupted, even though a human sometimes overrides its actions — say, to halt a robot heading somewhere dangerous — so the overrides give it no incentive to prevent them or to provoke them. The difficulty is that, to a learner, an override is just more experience: if stopping the agent keeps cutting off the reward it was heading for, the data say that course of action is worth less, and a capable enough learner could even find that disabling the stop button pays. Laurent Orseau and Stuart Armstrong (2016) formalized the property and showed that the outcome hinges on how the learner updates. Off-policyoff-policy, adj.: Describes learning that can use experience produced by some other strategy — an older version, a human, a different program — while improving the current one. Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next. needs no change, because each update looks ahead to the best next action rather than the one the agent was made to take, whereas on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes. methods such as SARSASARSA, n.: A learning method in which an agent updates the score of its last move using the reward it got and the score of the move it actually picks next, so it learns the value of its own habits. need a small modification; idealized, uncomputable universal agents can also be made to comply.
Because Q-learning is off-policy, the interruptions never leak into its value estimates — it is safely interruptible.
Origin
Introduced by Laurent Orseau of DeepMind and Stuart Armstrong of the Future of Humanity Institute in 'Safely Interruptible Agents' (UAI 2016); press coverage called it a 'big red button' for AI.
Usage note
Safe interruptibility concerns the learning dynamics of current algorithms; it does not address a capable planner that reasons about interruptions and acts to prevent them, which is the broader shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work..
Related
- shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.
- off-policyoff-policy, adj.: Describes learning that can use experience produced by some other strategy — an older version, a human, a different program — while improving the current one.
- kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop.
Sources
- Orseau, L. & Armstrong, S. (2016). Safely Interruptible Agents. Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence (UAI 2016).
Mentioned in
corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them., kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop., safe reinforcement learningsafe reinforcement learning, n.: Reinforcement learning designed to avoid dangerous or unacceptable behavior — both in what the finished agent does and in the risky experiments it might try while still learning., shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work., utility indifferenceutility indifference, n.: A design trick that makes an AI exactly indifferent to whether its off switch is pressed, so that it has no reason either to fight or to trigger shutdown.