Skip to content
Last Dictionary
Plane
Safety & AlignmentTraining & Post-trainingAGI & Superintelligence

AI alignment

nounAlso: alignment, aligned AI, aligned, alignment research

In plain wordsMaking an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.

  1. 1

    Property of a system

    The match between what an AI system is actually trying to achieve, and so how it behaves, and what its developers, users or society meant it to pursue — a matter of degree rather than a yes-or-no property. Falling short need not mean wanting something alien: settling for a measurable stand-in for the real goal, reading instructions too literally or ignoring constraints nobody thought to state all count. Researchers decompose the problem in several ways: outerouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want. versus inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training. (specifying the right objective versus instilling it), intent alignmentintent alignment, n.: An AI is intent-aligned if it is genuinely trying to do what its operator wants — even if it sometimes fails through ignorance or clumsiness. versus value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target. (whose wishes count), and alignment versus capabilitycapability, n.: Something an AI system is able to do, such as writing working code or solving math problems, and how well. Measuring capabilities is how researchers track progress and spot dangerous skills. (a highly capable system can still be misalignedmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people.). Characteristic failures include specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant., reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine., sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful. and deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later..

    A more capable model is not automatically a better-aligned one.

  2. 2

    Research field and practice

    The research field and engineering practice of producing aligned systems. For current language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. it largely means post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner. — instruction tuninginstruction tuning, n.: Training a language model on many tasks written as plain instructions with correct answers, so it learns to do what a request asks instead of just continuing the text., reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., Constitutional AIConstitutional AI, n.: Anthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand. and written model specificationsmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict. — plus evaluations of whether the intended behavior holds. Longer-range work targets systems that may exceed human ability to supervise them, through scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., weak-to-strong generalizationweak-to-strong generalization, n.: The finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI. and controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest..

    She moved from the product team to alignment research.

Origin

Norbert Wiener warned in 1960 that the purpose put into a machine must be the purpose we really desire. In a November 2014 essay on Edge.org, Stuart Russell wrote that an AI's utility function 'may not be perfectly aligned with the values of the human race'. The Machine Intelligence Research Institute's 2014 agenda spoke of aligning superintelligence with human interests, and by 2016 'AI alignment' was in use as the name of the field. Paul Christiano's essays spread the term, and OpenAI's InstructGPT work (2022) made it industry vocabulary.

Usage note

Unrelated to word alignment in machine translation or to cross-modal alignment of embeddingsembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. in multimodal models. Critics ask 'aligned to whom?', arguing that the choice of target is political as much as technical; some researchers reserve 'alignment' for catastrophic-risk work and call everyday behavior tuning post-training.

Related

See also
  • alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different.
  • outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want.
  • inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training.
  • intent alignmentintent alignment, n.: An AI is intent-aligned if it is genuinely trying to do what its operator wants — even if it sometimes fails through ignorance or clumsiness.
  • value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target.
  • AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable.
  • reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
  • superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it.
Contrast with
  • misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people.
  • capabilitycapability, n.: Something an AI system is able to do, such as writing working code or solving math problems, and how well. Measuring capabilities is how researchers track progress and spot dangerous skills.

Sources

  1. Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking.
  2. Christiano, P. (2018). Clarifying "AI alignment". AI Alignment Forum.
  3. Ji, J. et al. (2023). AI Alignment: A Comprehensive Survey. arXiv:2310.19852. arxiv.org/abs/2310.19852

Mentioned in

AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable., alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different., mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people., paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous., AI constitutionAI constitution, n.: A written set of principles that an AI company uses to shape an AI's values and behavior, both during training and as a public statement of what the AI should be like., AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., capabilitycapability, n.: Something an AI system is able to do, such as writing working code or solving math problems, and how well. Measuring capabilities is how researchers track progress and spot dangerous skills., control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed., HAL 9000HAL 9000, prop. n.: The calm-voiced spaceship computer in the 1968 film 2001: A Space Odyssey that turns against its crew; still the best-known picture of a machine putting its goals ahead of human lives., helpful, honest and harmlesshelpful, honest and harmless, phr.: A three-part goal for AI assistants, popularized by Anthropic: be genuinely useful, tell the truth, and avoid causing harm — shorthand for what an aligned chatbot should be., inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training., LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI., machine ethicsmachine ethics, n.: The effort to build AI systems that can tell right from wrong and act ethically themselves, rather than only being used ethically by people., Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it., orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us., outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want., Paul ChristianoPaul Christiano, prop. n.: An American AI-alignment researcher who led the work that made training AI from human preferences practical, the basis of how chatbots are taught to be helpful., shoggothshoggoth, n.: A tentacled monster from H. P. Lovecraft's horror fiction, used as a meme for large AI models: a vast, alien system hidden behind a small, friendly mask that training adds to make it polite., superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it., Three Laws of RoboticsThree Laws of Robotics, n.: Three rules that science-fiction writer Isaac Asimov built into his fictional robots — do not harm people, obey orders, protect yourself — in stories that mostly show how such simple rules go wrong., utility functionutility function, n.: A way of scoring every possible outcome by how much someone, or some AI, wants it. A rational agent is often modeled as choosing whatever maximizes its expected score., value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target., value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance.