AI control
In plain wordsSafety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
- 1
Safety researchemerging
An approach to AI safety that aims to prevent unacceptable outcomes even if a model is misaligned and deliberately trying to subvert its safeguards, instead of relying on having made it aligned. The capable, untrusted model is deployed inside a control protocol combining weaker trusted monitorstrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check., a limited budget of human audits, restricted permissions and sandboxingsandbox, n.: A sealed-off computer space where an AI agent can run programs and change files without being able to harm the real system, reach private data or contact the outside world unless allowed., resampling or editing of suspicious actions, and honeypotshoneypot, n.: A trap: a fake but tempting opportunity set up to catch wrongdoing — in AI safety, a staged situation designed to reveal whether an AI would misbehave if it thought it could get away with it.. Protocols are judged by control evaluationscontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards., in which a red team stands in for the model with an attack policy and the protocol is scored on safety and usefulness. The threat model is borrowed from insider-risk security: schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked. is assumed possible, and the question is what a model could achieve with the access it is given. Proponents expect control to work only up to a capability level beyond which models could outwit any feasible protocol.
Even if alignment fails, a good control protocol should catch the model before its third attempt.
- 2
General
Loosely, any means of keeping advanced AI systems under human direction, including capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants. measures such as boxing and kill switcheskill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop. as well as corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.. In this broad sense the phrase overlaps with the control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed. discussed by Nick Bostrom and Stuart Russell.
Origin
The research agenda was named in 'AI Control: Improving Safety Despite Intentional Subversion' (December 2023) by Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan and Fabien Roger of Redwood Research, who tested protocols in a setting where GPT-4 writes programs that might contain hidden backdoors. Bostrom's 'Superintelligence' (2014) and Russell's 'Human Compatible: Artificial Intelligence and the Problem of Control' (2019) popularized the broader sense.
Usage note
Control and alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this. are complements: alignment tries to make a model not want to cause harm; control tries to make harm impossible or detectable if it does. Critics doubt that protocols validated on current models will transfer to much more capable ones.
Related
- control evaluationcontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards.
- trusted monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check.
- untrusted monitoringuntrusted monitoring, n.: Using a second copy of a powerful AI to watch the first for sabotage, with safeguards that stop the two copies from secretly cooperating.
- trusted editingtrusted editing, n.: A safety setup in which a weaker, trusted AI rewrites any suspicious-looking part of a stronger AI's work before it is used, removing possible sabotage while keeping most of the value.
- schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked.
- Redwood ResearchRedwood Research, prop. n.: A small Berkeley nonprofit known for the idea of 'AI control': safeguards meant to keep powerful AI from doing damage even if the AI were secretly working against the people using it.
- defense in depthdefense in depth, n.: Protecting against AI risks with several independent layers of safeguards, so that if one fails — say, the model's own training — another, such as a filter or human review, still catches the problem.
- capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
Sources
- Greenblatt, R., Shlegeris, B., Sachan, K. & Roger, F. (2023). AI Control: Improving Safety Despite Intentional Subversion. arXiv:2312.06942. ICML 2024. arxiv.org/abs/2312.06942
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable., alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different., loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back., chain-of-thought monitoringchain-of-thought monitoring, n.: Having a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm., control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed., control theorycontrol theory, n.: The mathematics of making machines behave as intended by constantly measuring what they are doing and correcting them, as a thermostat, cruise control or self-balancing robot does., corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them., deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later., defense in depthdefense in depth, n.: Protecting against AI risks with several independent layers of safeguards, so that if one fails — say, the model's own training — another, such as a filter or human review, still catches the problem., safety casesafety case, n.: A written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft., schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked., treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it., AI boxAI box, n.: Keeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad., Buck ShlegerisBuck Shlegeris, prop. n.: An AI-safety researcher who runs Redwood Research and helped found 'AI control', the study of keeping powerful AI safe even if it is secretly trying to cause harm., capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants., control evaluationcontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards., guaranteed safe AIguaranteed safe AI, n.: A proposed approach to AI safety in which systems come with mathematical proofs or checkable guarantees that they will not cause certain harms, instead of relying on testing alone., honeypothoneypot, n.: A trap: a fake but tempting opportunity set up to catch wrongdoing — in AI safety, a staged situation designed to reveal whether an AI would misbehave if it thought it could get away with it., internal deploymentinternal deployment, n.: When an AI company uses its own newest, unreleased AI systems inside the company — for example to write code or do research — before or instead of offering them to the public., monitorabilitymonitorability, n.: How easy it is to watch an AI's reasoning and spot when it is planning something bad — a property that could be lost if future AIs stop thinking in readable language., prosaic alignmentprosaic alignment, n.: Working on AI safety by assuming that powerful future AI will be built from today's techniques — scaled-up neural networks — rather than from some new breakthrough., Redwood ResearchRedwood Research, prop. n.: A small Berkeley nonprofit known for the idea of 'AI control': safeguards meant to keep powerful AI from doing damage even if the AI were secretly working against the people using it., sabotage evaluationsabotage evaluation, n.: A test of whether an AI could secretly undermine the people overseeing it — by slipping bugs into code, misleading decision-makers or deliberately doing badly on tests — without being caught.