value alignment
In plain wordsBuilding AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target.
The goal of building AI systems whose objectives are consistent with human values — what people care about, morally endorse or would endorse on reflection — rather than with a designer's literal instructions alone. Stuart Russell's influential formulation casts it as building machines that pursue human preferences while remaining uncertain what those preferences are, which motivates value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance. and cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.. The philosopher Iason Gabriel (2020) distinguished candidate targets — instructions, expressed intentions, revealed preferences, informed preferences, interest or well-being, and values — and argued that choosing among them is a normative question that calls for fair procedures, given reasonable disagreement about values.
The paper argues that value alignment cannot be settled by engineers alone, because people disagree about values.
Origin
Stuart Russell wrote of the 'value alignment problem' in a November 2014 essay on Edge.org, arguing that the danger of advanced AI lies in competent pursuit of mis-specified objectives. He used the term in interviews and lectures from 2015 and gave it a formal definition in the 2016 paper on cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.; the Asilomar AI Principles (2017), whose tenth principle is titled 'Value Alignment', spread it further.
Usage note
Often contrasted with intent alignmentintent alignment, n.: An AI is intent-aligned if it is genuinely trying to do what its operator wants — even if it sometimes fails through ignorance or clumsiness.: an intent-aligned system does what its operator wants, which may harm others, whereas a value-aligned one acts on a wider notion of the good. Critics doubt that a single set of 'human values' exists to align to; others propose aggregation procedures such as coherent extrapolated volitioncoherent extrapolated volition, n.: A proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection..
Related
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance.
- intent alignmentintent alignment, n.: An AI is intent-aligned if it is genuinely trying to do what its operator wants — even if it sometimes fails through ignorance or clumsiness.
- cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.
- coherent extrapolated volitioncoherent extrapolated volition, n.: A proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection.
- value-sensitive designvalue-sensitive design, n.: A way of designing technology that deliberately identifies the human values at stake, such as privacy, fairness or autonomy, and builds them in from the start instead of patching problems later.
- Stuart RussellStuart Russell, prop. n.: A British computer scientist who co-wrote the standard AI textbook and argues that AI must be redesigned to stay uncertain about what humans want, so that it remains under human control.
- intent alignmentintent alignment, n.: An AI is intent-aligned if it is genuinely trying to do what its operator wants — even if it sometimes fails through ignorance or clumsiness.
Sources
- Gabriel, I. (2020). Artificial Intelligence, Values, and Alignment. Minds and Machines 30, 411–437.
- Russell, S. (2014). Of Myths and Moonshine. Response to 'The Myth of AI', Edge.org, November 2014. edge.org/conversation/the-myth-of-ai#26015
- Russell, S., Dewey, D. & Tegmark, M. (2015). Research Priorities for Robust and Beneficial Artificial Intelligence. AI Magazine 36(4).
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., Isaac AsimovIsaac Asimov, prop. n.: A Russian-born American science-fiction writer and biochemist who invented the Three Laws of Robotics and is credited with the first use of the word 'robotics'., machine ethicsmachine ethics, n.: The effort to build AI systems that can tell right from wrong and act ethically themselves, rather than only being used ethically by people., Norbert WienerNorbert Wiener, prop. n.: An American mathematician who founded cybernetics, the study of feedback and control in animals and machines, and warned early that machines might pursue goals we did not truly intend., orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us., Stuart RussellStuart Russell, prop. n.: A British computer scientist who co-wrote the standard AI textbook and argues that AI must be redesigned to stay uncertain about what humans want, so that it remains under human control., Three Laws of RoboticsThree Laws of Robotics, n.: Three rules that science-fiction writer Isaac Asimov built into his fictional robots — do not harm people, obey orders, protect yourself — in stories that mostly show how such simple rules go wrong., trolley problemtrolley problem, n.: A thought experiment asking whether you should divert a runaway tram so that it kills one person instead of five, often used to ask how self-driving cars should choose in an unavoidable crash., value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board., value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance., Center for Human-Compatible AICenter for Human-Compatible AI, prop. n.: Stuart Russell's group at the University of California, Berkeley, which argues that AI should be built to be unsure of what people want and to learn it by watching them., coherent extrapolated volitioncoherent extrapolated volition, n.: A proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection., friendly AIfriendly AI, n.: An early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment., intent alignmentintent alignment, n.: An AI is intent-aligned if it is genuinely trying to do what its operator wants — even if it sometimes fails through ignorance or clumsiness., reward hypothesisreward hypothesis, n.: The idea that every goal can be expressed as maximizing a single running score — that whatever we want an agent to do, we can say it with points.