Constitutional AI
In plain wordsAnthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand.
A training method that teaches harmlessness from a short written set of principles — a constitution — and AI-generated judgments instead of human labels of harmful outputs (people still supplied helpfulness comparisons). It has two stages. First, the model drafts answers to provocative prompts, points out where each falls short of a randomly chosen principle, rewrites it accordingly, and is then fine-tuned on the rewrites. Second, the fine-tuned model produces two answers per prompt, a judge model decides which better honors a sampled principle (reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging.), and these comparisons (preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like.) train a reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. for reinforcement learning. The 2022 paper reported assistants that were both less harmful and less evasive than RLHF-trained ones, more often engaging with a harmful request and giving reasons for declining than refusing flatly, and found that step-by-step reasoning by the judge (chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.) improved its labels. A written objective is inspectable and editable, though the wording of the principles, and the choice of who writes them, remain value-laden.
With Constitutional AI we added a principle about medical advice and regenerated the harmlessness preferences overnight.
Origin
Introduced by Anthropic in a December 2022 paper by Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell and 47 co-authors, and used to train the Claude models. Anthropic first published the constitution used for Claude in May 2023 and, with the Collective Intelligence Project, trained a model on a publicly sourced constitution later that year; it published a substantially revised constitution for Claude in January 2026.
Usage note
'Constitution' here means a list of principles used in training, not a legal document; published behavioral charters are covered at AI constitutionAI constitution, n.: A written set of principles that an AI company uses to shape an AI's values and behavior, both during training and as a public statement of what the AI should be like. and model specificationmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict.. Not to be confused with constitutional classifiersconstitutional classifiers, n.: Safety filters that watch a chatbot's inputs and outputs for dangerous requests, trained on example conversations that an AI generated from a written list of what is and is not allowed., Anthropic's 2025 input and output safeguard classifiers trained from a constitution.
Related
- reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging.
- AI constitutionAI constitution, n.: A written set of principles that an AI company uses to shape an AI's values and behavior, both during training and as a public statement of what the AI should be like.
- model specificationmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict.
- helpful, honest and harmlesshelpful, honest and harmless, phr.: A three-part goal for AI assistants, popularized by Anthropic: be genuinely useful, tell the truth, and avoid causing harm — shorthand for what an aligned chatbot should be.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- character trainingcharacter training, n.: Training an AI to have a consistent, deliberate personality — traits such as curiosity, honesty and warmth — rather than leaving its manner and values as accidental by-products of other training.
- ClaudeClaude, prop. n.: A family of AI assistants and language models made by the company Anthropic since 2023, widely used for writing, analysis and computer programming.
- constitutional classifiersconstitutional classifiers, n.: Safety filters that watch a chatbot's inputs and outputs for dangerous requests, trained on example conversations that an AI generated from a written list of what is and is not allowed.
Sources
- Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. arxiv.org/abs/2212.08073
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., AnthropicAnthropic, prop. n.: An American AI company that makes the Claude chatbot and models. It was founded in 2021 by former OpenAI staff and puts particular emphasis on AI safety research., ClaudeClaude, prop. n.: A family of AI assistants and language models made by the company Anthropic since 2023, widely used for writing, analysis and computer programming., Dario AmodeiDario Amodei, prop. n.: An American AI researcher who co-founded Anthropic and is its chief executive, known for early work on scaling and AI safety and for forecasting very rapid AI progress., guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data., refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow., AI constitutionAI constitution, n.: A written set of principles that an AI company uses to shape an AI's values and behavior, both during training and as a public statement of what the AI should be like., helpful, honest and harmlesshelpful, honest and harmless, phr.: A three-part goal for AI assistants, popularized by Anthropic: be genuinely useful, tell the truth, and avoid causing harm — shorthand for what an aligned chatbot should be., Jared KaplanJared Kaplan, prop. n.: An American physicist who showed that language models improve predictably as they grow and train on more data, and who co-founded the AI company Anthropic., reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging., safety trainingsafety training, n.: The part of an AI's training that teaches it to avoid harmful behavior — declining dangerous requests, resisting manipulation and following its developer's rules — on top of being generally helpful., Three Laws of RoboticsThree Laws of Robotics, n.: Three rules that science-fiction writer Isaac Asimov built into his fictional robots — do not harm people, obey orders, protect yourself — in stories that mostly show how such simple rules go wrong., character trainingcharacter training, n.: Training an AI to have a consistent, deliberate personality — traits such as curiosity, honesty and warmth — rather than leaving its manner and values as accidental by-products of other training., constitutional classifiersconstitutional classifiers, n.: Safety filters that watch a chatbot's inputs and outputs for dangerous requests, trained on example conversations that an AI generated from a written list of what is and is not allowed., deliberative alignmentdeliberative alignment, n.: A training method in which an AI is taught the actual text of its safety rules and learns to think through which rules apply before it answers a sensitive request., moral agencymoral agency, n.: The capacity to act for moral reasons and be held responsible for what one does. Researchers ask whether AI systems could ever be moral agents, or only tools whose makers and users bear responsibility., rubric rewardrubric reward, n.: A training score built from a checklist of specific things a good answer should do, with an AI grader ticking off each one — a way to reward quality where no single right answer exists.