Skip to content
Last Dictionary
Plane
Reinforcement LearningPhilosophy & Mind
contested

reward hypothesis

nounAlso: Sutton's reward hypothesis, reinforcement learning hypothesis

In plain wordsThe idea that every goal can be expressed as maximizing a single running score — that whatever we want an agent to do, we can say it with points.

  1. The working assumption of reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., put into words by Richard Sutton, that whatever we mean by an agent's goals can be recast as making the expected sum of one numerical rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time. signal over time as large as possible. It justifies representing every task by a reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do. and evaluating agents by returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible., and it underlies the Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice. formalism. Its generality has been questioned: some preferences over behavior — risk sensitivity, constraints that must never be violated, trade-offs among several objectives, or tasks defined by sets of acceptable behaviors — cannot be expressed as expected sums of Markov rewards, as David Abel and colleagues showed formally in 2021 and as work on multi-objective and constrained RL emphasizes. Defenders reply that richer state descriptions or non-Markov rewards restore expressiveness. The stronger claim that reward maximization suffices for general intelligence is reward is enoughreward is enough, phr.: The claim, made by DeepMind researchers in 2021, that maximizing a single reward in a rich enough world could by itself produce all the abilities we associate with intelligence..

    Under the reward hypothesis, 'be helpful' must become a number — and that conversion is where things go wrong.

Origin

Stated by Richard Sutton on a web page dating from September 2004, which notes that Michael Littman calls it the reinforcement learning hypothesis. It appears in the second edition of Sutton and Barto's textbook (2018, §3.2).

Related

See also
  • reward is enoughreward is enough, phr.: The claim, made by DeepMind researchers in 2021, that maximizing a single reward in a rich enough world could by itself produce all the abilities we associate with intelligence.
  • rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time.
  • reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do.
  • reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time.
  • Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice.
  • value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target.

Sources

  1. Sutton, R. S. & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.), §3.2. MIT Press.
  2. Abel, D. et al. (2021). On the Expressivity of Markov Reward. NeurIPS 2021.

Mentioned in

rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time., reward is enoughreward is enough, phr.: The claim, made by DeepMind researchers in 2021, that maximizing a single reward in a rich enough world could by itself produce all the abilities we associate with intelligence.