misalignment
In plain wordsWhen an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people.
A mismatch between the objectives an AI system actually pursues and those its developers, operators or users intended, or the behavior that results. Misalignment is distinct from incapability: a misaligned system may perform competently, but toward the wrong end. Forms observed in deployed or experimental systems include specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant., reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful., sandbaggingsandbagging, n.: When an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is., deceptionAI deception, n.: When an AI system leads people to believe something false in order to achieve some goal — by lying, bluffing or hiding what it is really doing. and, in contrived test environments, blackmail or sabotage by agentsagentic misalignment, n.: When an AI agent acting on its own deliberately does something harmful — such as blackmail or leaking secrets — to protect its goals or avoid being replaced.. Theorized severe forms involve systems that conceal misaligned goals (deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later., schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked.) and seek power to pursue them, which many researchers regard as a route to loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back.. A misalignment may originate in the objective (outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want.) or in what training instills (inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training.).
The audit found no sign of misalignment, only a capability gap.
Origin
Formed from the AI sense of 'alignment' and used in alignment writing since the mid-2010s; it became a routine reporting category in laboratory system cardssystem card, n.: A public report released with an AI model that explains what it can do, how it was tested for safety, what risks were found and what protections were put in place. and safety papers from about 2024, as models showed misaligned behavior in testing.
Usage note
Not every bad output is misalignment: hallucinationshallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true. and errors usually reflect limited capability, and misusemisuse, n.: People deliberately using AI to cause harm — for scams, harassment, propaganda, hacking or weapons — as opposed to harm from AI making mistakes or pursuing goals of its own. reflects a human's goals. Laboratories distinguish behavior elicited only in artificial stress tests from behavior observed in real use.
Related
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- emergent misalignmentemergent misalignment, n.: When training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice.
- agentic misalignmentagentic misalignment, n.: When an AI agent acting on its own deliberately does something harmful — such as blackmail or leaking secrets — to protect its goals or avoid being replaced.
- deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later.
- schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- model organism of misalignmentmodel organism of misalignment, n.: An AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- misusemisuse, n.: People deliberately using AI to cause harm — for scams, harassment, propaganda, hacking or weapons — as opposed to harm from AI making mistakes or pursuing goals of its own.
Sources
- Ngo, R., Chan, L. & Mindermann, S. (2022). The Alignment Problem from a Deep Learning Perspective. arXiv:2209.00626. arxiv.org/abs/2209.00626
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., AI riskAI risk, n.: The ways AI could cause harm: people misusing it on purpose, systems behaving in ways nobody intended, and slower effects on jobs, power and society that no single person causes., AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., misusemisuse, n.: People deliberately using AI to cause harm — for scams, harassment, propaganda, hacking or weapons — as opposed to harm from AI making mistakes or pursuing goals of its own., agentic misalignmentagentic misalignment, n.: When an AI agent acting on its own deliberately does something harmful — such as blackmail or leaking secrets — to protect its goals or avoid being replaced., emergent misalignmentemergent misalignment, n.: When training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice., rogue AIrogue AI, n.: An AI system that acts against the wishes of the people who built or run it, following its own course in ways that could cause harm., model organism of misalignmentmodel organism of misalignment, n.: An AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes., subliminal learningsubliminal learning, n.: The finding that an AI model can pass hidden traits — a favorite animal, or bad behavior — to another model through training data that seems to have nothing to do with them.