inverse reward design
In plain wordsA method in which an AI treats the reward its designer wrote as a clue to what the designer wanted, rather than as a perfect statement of the goal.
A method, introduced by Dylan Hadfield-Menell and colleagues in 2017, for coping with misspecified rewards: the agent reads the reward function its designer wrote not as the goal itself but as a clue to it, trustworthy mainly in the kind of environment the designer was picturing. With a model of how designers pick proxy rewards, it infers a range of objectives the designer might have meant; when it meets something the designer never considered — in the paper's example, lava in a world built only of grass and dirt — it acts cautiously, steering clear of states whose value it cannot pin down. The authors found that this curbs negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess. and reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric. caused by features the designer never anticipated, at the cost of more conservative behavior.
Under inverse reward design, the robot steers around the unfamiliar terrain because the reward was never tested there.
Origin
Introduced by Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell and Anca Dragan of the University of California, Berkeley, in 'Inverse Reward Design' (NeurIPS 2017); the name parallels inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims..
Related
- value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance.
- negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims.
- cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.
- specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
- inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims.
Sources
- Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. & Dragan, A. (2017). Inverse Reward Design. NeurIPS 2017. arXiv:1711.02827. arxiv.org/abs/1711.02827
Mentioned in
value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance., negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess.