cooperative inverse reinforcement learning
In plain wordsA model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.
A formal model of human–AI assistance, proposed by Dylan Hadfield-Menell and colleagues in 2016, in which a human and a robot play a cooperative game whose payoff is the human's goal, a goal the robot cannot see and must work out while helping. The robot's best strategy therefore mixes gathering clues about that goal from the human's behavior with acting on its current best guess, and the human's best strategy includes teaching — choosing actions that are informative, not merely competent. Classical inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims., by contrast, has the robot watch an expert from the sidelines and then pursue the inferred reward as its own. Because the two players want the same thing, the authors could find their best joint behavior by planning for one imaginary coordinator: a single POMDPpartially observable Markov decision process, n.: A version of the standard decision-making model in which the decision-maker cannot see the whole situation — like a poker player who cannot see other hands — and must act on clues and memory. in which the human's goal is the hidden variable. CIRL is the formal core of Stuart Russell's proposal that machines should be uncertain about human preferences, and it underlies the off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure..
Framed as a CIRL game, asking a clarifying question is just optimal information gathering.
Origin
Introduced by Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell in 'Cooperative Inverse Reinforcement Learning' (NeurIPS 2016). Russell's book 'Human Compatible' (2019) and later papers generalized the framework under the name 'assistance games'.
Usage note
Critics note that CIRL assumes a fixed, well-defined human reward function and a near-optimal human, both unrealistic, and that a CIRL agent with a confident but wrong prior can still resist correction.
Related
- value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance.
- inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims.
- off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- Stuart RussellStuart Russell, prop. n.: A British computer scientist who co-wrote the standard AI textbook and argues that AI must be redesigned to stay uncertain about what humans want, so that it remains under human control.
- Center for Human-Compatible AICenter for Human-Compatible AI, prop. n.: Stuart Russell's group at the University of California, Berkeley, which argues that AI should be built to be unsure of what people want and to learn it by watching them.
- inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims.
Sources
- Hadfield-Menell, D., Dragan, A., Abbeel, P. & Russell, S. (2016). Cooperative Inverse Reinforcement Learning. Advances in Neural Information Processing Systems 29. arXiv:1606.03137. arxiv.org/abs/1606.03137
Mentioned in
Stuart RussellStuart Russell, prop. n.: A British computer scientist who co-wrote the standard AI textbook and argues that AI must be redesigned to stay uncertain about what humans want, so that it remains under human control., value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target., value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance., Center for Human-Compatible AICenter for Human-Compatible AI, prop. n.: Stuart Russell's group at the University of California, Berkeley, which argues that AI should be built to be unsure of what people want and to learn it by watching them., inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims., inverse reward designinverse reward design, n.: A method in which an AI treats the reward its designer wrote as a clue to what the designer wanted, rather than as a perfect statement of the goal., off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.