Skip to content
Last Dictionary
Plane
Reinforcement LearningTraining & Post-training

REINFORCE

nounAlso: Monte Carlo policy gradient, vanilla policy gradient, REINFORCE with baseline

In plain wordsThe simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly.

  1. The basic Monte Carlo policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely. algorithm: sample complete episodesepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again. with the current stochastic policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards., then move the parameters along the gradient of each action's log-probability multiplied by the returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible. that followed it, optionally minus a baseline. The update is an unbiased estimate of the gradient of expected return and needs no value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board., no model and no differentiable environment — only the ability to sample actions and compute their log-probabilities. Its drawback is high variance, which makes learning slow; subtracting a baseline such as the average return reduces variance without adding bias, and replacing the baseline with a learned critic turns it into an actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic. method. Because large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. RL has short episodes with a single final reward, REINFORCE-style estimators with group or leave-one-out baselines (RLOO and, in effect, GRPOgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average.) have been revived as simpler alternatives to PPOproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback..

    We replaced PPO with REINFORCE and a leave-one-out baseline, dropping the critic entirely.

θ←θ+α∑t(Gt−b(St))∇θlog⁡πθ(At∣St)\theta \leftarrow \theta + \alpha \sum_{t} \left( G_t - b(S_t) \right) \nabla_\theta \log \pi_\theta(A_t \mid S_t)

Origin

Introduced by Ronald J. Williams in a 1987 conference paper and given its full treatment in 'Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning' (Machine Learning, 1992). The capitalized name is an acronym for the form of the update: 'REward Increment = Nonnegative Factor × Offset Reinforcement × Characteristic Eligibility'.

Related

See also
  • policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely.
  • actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic.
  • returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible.
  • group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average.
  • proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
  • entropy regularizationentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early.

Sources

  1. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8, 229–256. link.springer.com/article/10.1007/BF00992696
  2. Williams, R. J. (1987). A class of gradient-estimating algorithms for reinforcement learning in neural networks. Proceedings of the IEEE First International Conference on Neural Networks, San Diego.
  3. Ahmadian, A. et al. (2024). Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. ACL 2024. aclanthology.org/2024.acl-long.662

Mentioned in

actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic., baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value., group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average., on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes., policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely., Gumbel-softmaxGumbel-softmax, n.: A mathematical trick that lets a neural network make a choice among separate options, such as picking one word or one category, while still learning through small, smooth adjustments., REINFORCE leave-one-outREINFORCE leave-one-out, n.: A simple reinforcement-learning method that has a model answer each question several times and judges each answer against the average score of the other answers., reparameterization trickreparameterization trick, n.: A trick that rewrites a random draw as fixed numbers plus separate, outside randomness, so that a neural network can still learn through a step that involves chance.