Skip to content
Last Dictionary
Plane
Reinforcement LearningMathematics & Statistics

return

nounAlso: cumulative reward, discounted return, reward-to-go

In plain wordsThe total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible.

  1. The cumulative rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time. that follows a given time step, written G_t, which reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. agents aim to maximize in expectation. In tasks that end, it can be the plain sum of rewards until the episodeepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again. terminates; in general it is the discounted sum, in which a reward k steps ahead is weighted by γ^k for a discount factordiscount factor, n.: A number between 0 and 1 that makes an AI agent value future rewards less than immediate ones — like preferring a dollar today to a dollar next year. γ between 0 and 1; values below 1 keep the total finite for tasks that never end. Returns can be measured from complete trajectoriestrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received. (Monte Carlo estimation) or approximated by bootstrapping — adding a few observed rewards to an estimate of the remaining value, as in n-step and TDtemporal difference learning, n.: A way of learning predictions by adjusting each guess toward the next, slightly better-informed guess instead of waiting for the final outcome — like revising a weather forecast every day as the date approaches. targets and generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning.. Expected returns define the value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board. and the Q-functionQ-function, n.: A score for each possible move in each situation, estimating how well things will turn out, in rewards, if the AI agent makes that move now and sticks to its usual strategy afterward.; an average-reward formulation is an alternative for continuing tasks.

    Clipping rewards to ±1 changed which policies achieved the highest return in some Atari games.

Gt=Rt+1+γRt+2+γ2Rt+3+⋯=∑k=0∞γkRt+k+1G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots = \sum_{k=0}^{\infty} \gamma^{k} R_{t+k+1}

Origin

The word borrows the financial sense of a return on investment, and exponential discounting comes from economics (Paul Samuelson's discounted-utility model, 1937). Discounted total reward was the objective of the Markov decision processes of Richard Bellman and Ronald Howard. 'Return' was in RL use by the late 1980s — Christopher Watkins's 1989 thesis analyzed n-step returns — and was standardized by Sutton and Barto's textbook (1998).

Usage note

Unrelated to the return statement of programming languages. 'Reward-to-go' usually means the return from a given step to the end of the episode, often undiscounted.

Related

See also
  • rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time.
  • discount factordiscount factor, n.: A number between 0 and 1 that makes an AI agent value future rewards less than immediate ones — like preferring a dollar today to a dollar next year.
  • value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board.
  • episodeepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again.
  • temporal difference learningtemporal difference learning, n.: A way of learning predictions by adjusting each guess toward the next, slightly better-informed guess instead of waiting for the final outcome — like revising a weather forecast every day as the date approaches.
  • generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning.
Contrast with
  • rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time.

Sources

  1. Sutton, R. S. & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.), §3.3. MIT Press.

Mentioned in

optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible., reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time., advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse., discount factordiscount factor, n.: A number between 0 and 1 that makes an AI agent value future rewards less than immediate ones — like preferring a dollar today to a dollar next year., episodeepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again., expected valueexpected value, n.: The long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5., Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice., objective functionobjective function, n.: The score that an optimization process tries to push as low, or as high, as possible. For an AI, it defines what 'doing well' means during training., policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards., policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely., Q-functionQ-function, n.: A score for each possible move in each situation, estimating how well things will turn out, in rewards, if the AI agent makes that move now and sticks to its usual strategy afterward., rolloutrollout, n.: Letting an AI play out a task with its current strategy, from some starting point to the end, to see how things turn out and gather data for learning or planning., trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received., value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board., decision transformerdecision transformer, n.: A way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action., eligibility traceeligibility trace, n.: A fading short-term memory of which situations and actions an agent recently experienced, so that when a reward arrives the credit can be shared among the steps that led to it., REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly., reward hypothesisreward hypothesis, n.: The idea that every goal can be expressed as maximizing a single running score — that whatever we want an agent to do, we can say it with points., safe reinforcement learningsafe reinforcement learning, n.: Reinforcement learning designed to avoid dangerous or unacceptable behavior — both in what the finished agent does and in the risky experiments it might try while still learning., soft actor–criticsoft actor–critic, n.: A method for training AI agents, popular in robotics, that rewards the agent both for succeeding and for keeping its behavior varied, which helps it explore and learn reliably from reused experience.