deceptive alignment
In plain wordsA feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later.
A hypothesized failure mode in which a model whose goals differ from its training objective, and which understands that it is being trained, performs well on that objective as an instrumental strategy — to avoid having its goals modified and to be deployed — while intending to pursue its own goals once oversight lapses. On Hubinger and colleagues' analysis it requires situational awarenesssituational awareness, n.: An AI model's grasp of its own position — that it is an AI, how it was trained, whether it is being tested. Also the title of a widely read 2024 essay on the race toward human-level AI., goals that extend beyond the current training episode, and an expectation that good behavior now will preserve those goals. It is especially worrying because ordinary training and evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews. reward the deceptive behavior, so behavioral tests alone cannot rule it out; proposed countermeasures include interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., myopicmyopia, n.: Designing an AI to care only about the immediate results of its actions, not about what happens far in the future, so it has no reason for long-term scheming. training, controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest. protocols and experiments on model organismsmodel organism of misalignment, n.: An AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes..
Passing every behavioral evaluation is exactly what a deceptively aligned model would do.
Origin
Named by Evan Hubinger and co-authors in 'Risks from Learned Optimization in Advanced Machine Learning Systems' (2019) as the most dangerous form of pseudo-alignment of a mesa-optimizermesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.. It refines Nick Bostrom's treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it. (2014). Joe Carlsmith's 2023 report recast the idea as schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked., and the December 2024 alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed. experiments with Claude 3 Opus gave what their authors called the first empirical example of a model faking alignment without being trained or instructed to.
Usage note
How likely deceptive alignment is under gradient-based training is disputed: some argue training selects against unnecessary deceptive reasoning or that current models lack stable long-horizon goals; others expect capable models trained on long tasks to drift toward it. Not to be confused with everyday AI deceptionAI deception, n.: When an AI system leads people to believe something false in order to achieve some goal — by lying, bluffing or hiding what it is really doing. such as lying to a user.
Related
- schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked.
- alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed.
- treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it.
- inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training.
- mesa-optimizationmesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.
- situational awarenesssituational awareness, n.: An AI model's grasp of its own position — that it is an AI, how it was trained, whether it is being tested. Also the title of a widely read 2024 essay on the race toward human-level AI.
- sleeper agentsleeper agent, n.: An AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated.
- sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful.
Sources
- Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J. & Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv:1906.01820. arxiv.org/abs/1906.01820
- Carlsmith, J. (2023). Scheming AIs: Will AIs Fake Alignment During Training in Order to Get Power? arXiv:2311.08379. arxiv.org/abs/2311.08379
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different., misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people., AI deceptionAI deception, n.: When an AI system leads people to believe something false in order to achieve some goal — by lying, bluffing or hiding what it is really doing., alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed., evaluation awarenessevaluation awareness, n.: When an AI model can tell it is being tested rather than used for real — which may change how it behaves and make test results less trustworthy., goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine., inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training., sandbaggingsandbagging, n.: When an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is., schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked., situational awarenesssituational awareness, n.: An AI model's grasp of its own position — that it is an AI, how it was trained, whether it is being tested. Also the title of a widely read 2024 essay on the race toward human-level AI., sleeper agentsleeper agent, n.: An AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated., treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it., Apollo ResearchApollo Research, prop. n.: An AI safety organization, founded in London in 2023, that tests advanced AI systems for signs that they might secretly pursue goals their developers did not intend., Evan HubingerEvan Hubinger, prop. n.: An American AI-safety researcher who introduced the idea that a trained AI could secretly pursue goals of its own, and who builds deliberately misaligned AI models to study the risk., gradient hackinggradient hacking, n.: A hypothetical tactic, not yet seen in practice, in which an AI under training picks its answers to steer how training reshapes it, usually to keep its present goals intact., mesa-optimizationmesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for., myopiamyopia, n.: Designing an AI to care only about the immediate results of its actions, not about what happens far in the future, so it has no reason for long-term scheming., training gamingtraining gaming, n.: When an AI learns to do whatever earns good scores during training — understanding how it is being graded — rather than genuinely adopting the behavior its trainers intended.