Center for Human-Compatible AI
In plain wordsStuart Russell's group at the University of California, Berkeley, which argues that AI should be built to be unsure of what people want and to learn it by watching them.
A Berkeley research center, founded by Stuart Russell, whose program is to replace the textbook approach to AI, in which a machine optimizes an objective fixed in advance, with machines that know they are uncertain about human preferences and learn those preferences from what people do. Its signature formalism, cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping. (2016), treats human–AI interaction as an 'assistance game' in which the machine maximizes the human's reward without knowing what it is; the related off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure. (2017) showed that such uncertainty can give an agent a reason to let itself be switched off. CHAI researchers have also studied preference learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims., reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., adversarial policies and the influence of recommender systems on users, and the center has trained many alignment researchers.
Origin
Founded in 2016 by Stuart Russell together with Berkeley colleagues including Pieter Abbeel, Anca Dragan and Tom Griffiths, with an initial grant from the Open Philanthropy Project. Russell's book Human Compatible (2019) presents its research program for general readers.
Related
- Stuart RussellStuart Russell, prop. n.: A British computer scientist who co-wrote the standard AI textbook and argues that AI must be redesigned to stay uncertain about what humans want, so that it remains under human control.
- cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.
- off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.
- value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target.
- inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
Sources
- Hadfield-Menell, D., Dragan, A., Abbeel, P. & Russell, S. (2016). Cooperative Inverse Reinforcement Learning. NeurIPS 2016.
- Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking.
Mentioned in
Stuart RussellStuart Russell, prop. n.: A British computer scientist who co-wrote the standard AI textbook and argues that AI must be redesigned to stay uncertain about what humans want, so that it remains under human control., cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping., off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure., Open PhilanthropyOpen Philanthropy, prop. n.: A large American charitable foundation, renamed Coefficient Giving in 2025, that has been the biggest private funder of research on the risks of advanced AI., Pieter AbbeelPieter Abbeel, prop. n.: A Belgian-American roboticist at Berkeley whose research taught robots to learn from demonstrations and from trial and error, and who co-founded the robotics company Covariant.