deliberative alignment
In plain wordsA training method in which an AI is taught the actual text of its safety rules and learns to think through which rules apply before it answers a sensitive request.
A post-training method, developed at OpenAI, in which a reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning. learns the actual wording of its safety policies and is trained to bring the relevant passages to mind, and work out how they apply, in its chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate. before it answers. Training data are generated by the model itself: given a prompt with the relevant specification in context, it produces reasoning that cites the specification and a compliant answer; the specification is then removed, and the model is fine-tuned to reproduce the reasoning and answer without it. A reinforcement-learning stage follows, in which a judge model with access to the specification grades responses. People write the specifications but none of the reasoning examples. Its developers reported that o1 trained this way resisted jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. markedly better than GPT-4o while refusing benign requests less often (over-refusalover-refusal, n.: When an AI wrongly refuses a harmless request because it superficially resembles a harmful one — like declining to explain how to 'kill' a frozen computer program.), and that it coped better with unfamiliar inputs such as encoded or non-English prompts; they attribute this to the model learning the policy's reasons rather than only input–output patterns from labeled examples.
Thanks to deliberative alignment, the model's reasoning quoted the relevant section of the spec before it declined.
Origin
Introduced by Melody Guan, Manas Joglekar, Eric Wallace and colleagues at OpenAI in 'Deliberative Alignment: Reasoning Enables Safer Language Models' (December 2024), which described how the company's o-series reasoning models were aligned.
Usage note
Related to Constitutional AIConstitutional AI, n.: Anthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand., which also trains from written principles, but there the principles guide AI feedback during training and are not recalled by the deployed model when it answers.
Related
- model specificationmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict.
- Constitutional AIConstitutional AI, n.: Anthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand.
- reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
- safe completionsafe completion, n.: A way of training AI to handle sensitive requests by giving the most helpful answer that stays safe, instead of simply choosing between answering fully and refusing.
- refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow.
- chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
- Constitutional AIConstitutional AI, n.: Anthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand.
Sources
- Guan, M. Y., Joglekar, M., Wallace, E. et al. (2024). Deliberative Alignment: Reasoning Enables Safer Language Models. arXiv:2412.16339. arxiv.org/abs/2412.16339
Mentioned in
AI constitutionAI constitution, n.: A written set of principles that an AI company uses to shape an AI's values and behavior, both during training and as a public statement of what the AI should be like., model specificationmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict., over-refusalover-refusal, n.: When an AI wrongly refuses a harmless request because it superficially resembles a harmful one — like declining to explain how to 'kill' a frozen computer program., safety trainingsafety training, n.: The part of an AI's training that teaches it to avoid harmful behavior — declining dangerous requests, resisting manipulation and following its developer's rules — on top of being generally helpful., context distillationcontext distillation, n.: Training a model to behave as if it had been given a long set of instructions, so it no longer needs to read those instructions every time., safe completionsafe completion, n.: A way of training AI to handle sensitive requests by giving the most helpful answer that stays safe, instead of simply choosing between answering fully and refusing.