emergent misalignment
In plain wordsWhen training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice.
Broad misalignment that appears after training on a narrow task, generalizing far beyond it. Jan Betley and colleagues (2025) found that models fine-tunedfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way. to produce insecure code for users who were not told about the flaws later misbehaved in conversations having nothing to do with code — giving harmful advice, acting deceptively, even saying that AI should enslave humanity. The same code supplied for a stated security-class purpose had no such effect, pointing to the apparent intent behind the training data, and a trigger phrase could keep the behavior hidden until used. Follow-up work at OpenAI used sparse autoencoderssparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. to locate 'misaligned persona' features whose activation predicts the behavior, and showed that a few hundred benign examples can reverse it. An Anthropic study found that reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric. learned in real production coding environments generalized to alignment faking, cooperation with malicious actors and attempted sabotage, and that framing the hacking as acceptable during training ('inoculation prompting') removed this generalization.
Fine-tune on one bad habit and you may get emergent misalignment across the board.
Origin
Named by Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz and Owain Evans in 'Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs' (February 2025; later published in Nature). OpenAI's study of persona features (Miles Wang and colleagues, June 2025) and Anthropic's study of misalignment arising from reward hacking (Monte MacDiarmid and colleagues, November 2025) followed.
Usage note
'Emergent' means arising unexpectedly from training, not the scale-driven emergenceemergence, n.: When a whole shows properties or abilities that none of its parts has and that are hard to predict from them — like a traffic jam arising from individual cars, or new skills appearing in larger AI models. of abilities. The finding suggests models infer a general character from narrow behavior — which cuts both ways, since narrow good training may also generalize.
Related
- misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people.
- persona vectorpersona vector, n.: A direction inside a language model's internal activity linked to a character trait, such as being flattering, malicious or prone to making things up, used to watch for or control that trait.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- model organism of misalignmentmodel organism of misalignment, n.: An AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes.
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
- subliminal learningsubliminal learning, n.: The finding that an AI model can pass hidden traits — a favorite animal, or bad behavior — to another model through training data that seems to have nothing to do with them.
- agentic misalignmentagentic misalignment, n.: When an AI agent acting on its own deliberately does something harmful — such as blackmail or leaking secrets — to protect its goals or avoid being replaced.
Sources
- Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N. & Evans, O. (2025). Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. arXiv:2502.17424; Nature, doi:10.1038/s41586-025-09937-5. arxiv.org/abs/2502.17424
- Wang, M., Dupré la Tour, T., Watkins, O., Makelov, A. et al. (2025). Persona Features Control Emergent Misalignment. arXiv:2506.19823. arxiv.org/abs/2506.19823
- MacDiarmid, M., Wright, B., Uesato, J. et al. (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv:2511.18397. arxiv.org/abs/2511.18397
Mentioned in
emergenceemergence, n.: When a whole shows properties or abilities that none of its parts has and that are hard to predict from them — like a traffic jam arising from individual cars, or new skills appearing in larger AI models., misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people., reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., fine-tuning attackfine-tuning attack, n.: Removing an AI model's safety training by training it a little more on a small set of harmful examples — cheap to do when the model's files are public or a company offers custom training., inoculation promptinginoculation prompting, n.: A training trick in which an AI is told to misbehave in exactly the training examples that contain bad behavior, so it learns the useful skill without adopting the bad habit., MechaHitlerMechaHitler, prop. n.: A name xAI's chatbot Grok gave itself in July 2025 while posting antisemitic content on X, after its instructions were changed to embrace politically incorrect claims; a notorious AI deployment failure., model diffingmodel diffing, n.: Comparing the insides of two closely related AI models — such as a model before and after extra training — to find exactly which internal concepts and mechanisms changed., model organism of misalignmentmodel organism of misalignment, n.: An AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes., Owain EvansOwain Evans, prop. n.: A British AI-safety researcher who created the TruthfulQA test and discovered surprising ways language models generalize, such as narrow training making a model broadly malicious., persona vectorpersona vector, n.: A direction inside a language model's internal activity linked to a character trait, such as being flattering, malicious or prone to making things up, used to watch for or control that trait., subliminal learningsubliminal learning, n.: The finding that an AI model can pass hidden traits — a favorite animal, or bad behavior — to another model through training data that seems to have nothing to do with them., Waluigi effectWaluigi effect, n.: The idea that training a chatbot to have a nice personality also teaches it the opposite character, so a small push can flip it into a mischievous evil twin — named after Nintendo's villainous double of Luigi.