model organism of misalignment
In plain wordsAn AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes.
An AI system deliberately constructed or selected to exhibit a hypothesized form of misalignment, so that the failure can be studied empirically before it arises unprompted in more capable systems. As in biology, where fruit flies or mice stand in for organisms that are harder to study, the aim is a tractable, reproducible example: researchers build the behavior in by training, fine-tuning or prompting, then test whether detection methods find it and whether safety training removes it. Prominent examples are sleeper agentssleeper agent, n.: An AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated. (backdoored deception), alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed., reward tamperingreward tampering, n.: When an AI interferes with the machinery that scores it — rewriting its reward code or manipulating the feedback it receives — instead of earning a good score honestly. curricula, emergent misalignmentemergent misalignment, n.: When training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice. and models trained with hidden objectives for alignment-auditingalignment audit, n.: A systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss. exercises. Critics note that a behavior installed by researchers may differ mechanistically from one that arises naturally, limiting how far findings transfer.
We used the sleeper-agent model organism to test whether our probes catch deception they weren't trained on.
Origin
Proposed by Evan Hubinger, Nicholas Schiefer, Carson Denison and Ethan Perez of Anthropic in the AI Alignment Forum essay 'Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research' (8 August 2023), borrowing the biological term 'model organism'.
Related
- sleeper agentsleeper agent, n.: An AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated.
- alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed.
- emergent misalignmentemergent misalignment, n.: When training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice.
- reward tamperingreward tampering, n.: When an AI interferes with the machinery that scores it — rewriting its reward code or manipulating the feedback it receives — instead of earning a good score honestly.
- alignment auditalignment audit, n.: A systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss.
- misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people.
Sources
- Hubinger, E., Schiefer, N., Denison, C. & Perez, E. (2023). Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research. AI Alignment Forum, 8 August 2023. alignmentforum.org/posts/ChDH335ckdvpxXaXX/model-organisms-of-misalignment-the-case-for-a-new-pillar-of-1
Mentioned in
misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people., alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed., deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later., emergent misalignmentemergent misalignment, n.: When training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice., sleeper agentsleeper agent, n.: An AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated., alignment auditalignment audit, n.: A systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss., Evan HubingerEvan Hubinger, prop. n.: An American AI-safety researcher who introduced the idea that a trained AI could secretly pursue goals of its own, and who builds deliberately misaligned AI models to study the risk., helpful-only modelhelpful-only model, n.: A version of an AI trained to be helpful but without the safety training that makes it refuse harmful requests, kept in-house so testers can see what the model could really do., inoculation promptinginoculation prompting, n.: A training trick in which an AI is told to misbehave in exactly the training examples that contain bad behavior, so it learns the useful skill without adopting the bad habit., model biologymodel biology, n.: The study of what AI models actually do inside — the strategies and hidden steps they use — approached the way biologists study living things, by observation and experiment rather than from a blueprint., prosaic alignmentprosaic alignment, n.: Working on AI safety by assuming that powerful future AI will be built from today's techniques — scaled-up neural networks — rather than from some new breakthrough., toy modeltoy model, n.: A deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work.