Skip to content
Last Dictionary
Plane
Reinforcement LearningTraining & Post-trainingMathematics & Statistics

entropy regularization

nounAlso: entropy bonus, entropy regularisation, maximum entropy reinforcement learning

In plain wordsRewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early.

  1. The addition to a reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. objective of a term proportional to the entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy. of the policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards.'s action distribution, so that the optimizer is penalized for becoming deterministic too quickly. As a bonus in policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely. methods such as A3CA3C, abbr.: A 2016 DeepMind training method in which many copies of an AI agent learn in parallel on separate copies of a task, pooling their updates into one shared neural network. and PPOproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., it counteracts premature convergence on a suboptimal action and keeps exploration alive. In maximum-entropy RL, as in soft actor–criticsoft actor–critic, n.: A method for training AI agents, popular in robotics, that rewards the agent both for succeeding and for keeping its behavior varied, which helps it explore and learn reliably from reused experience., entropy is part of the reward at every step, which yields 'soft' Bellman equations, Boltzmann-like optimal policies and more robust behavior. The coefficient matters: too small and the policy collapses, too large and it never commits. In large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. RL the related problem of entropy collapse — token-level entropy falling sharply early in training, after which exploration and improvement stall — became a research focus in 2025, with remedies based on clipping, covariance and KL terms as well as entropy bonuses.

    Without the entropy bonus, the policy became deterministic within a few thousand updates and stopped improving.

J(θ)=Eπθ[∑tγtrt]+β Es[H(πθ(⋅∣s))]J(\theta) = \mathbb{E}_{\pi_\theta}\left[ \sum_{t} \gamma^{t} r_t \right] + \beta\, \mathbb{E}_{s}\left[ \mathcal{H}\left( \pi_\theta(\cdot \mid s) \right) \right]

Origin

Ronald Williams and Jing Peng proposed adding an entropy term to policy-gradient updates in 'Function Optimization Using Connectionist Reinforcement Learning Algorithms' (Connection Science, 1991). Maximum-entropy formulations grew from Brian Ziebart's work on inverse reinforcement learning (2008) and control as inference, and DeepMind's A3C (2016) made the bonus standard in deep RL.

Related

See also
  • entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy.
  • soft actor–criticsoft actor–critic, n.: A method for training AI agents, popular in robotics, that rewards the agent both for succeeding and for keeping its behavior varied, which helps it explore and learn reliably from reused experience.
  • exploration–exploitation trade-offexploration–exploitation trade-off, n.: The choice between trying something new that might turn out better and sticking with what already works — like picking a familiar restaurant versus testing a new one.
  • proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
  • policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely.
  • KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.

Sources

  1. Williams, R. J. & Peng, J. (1991). Function optimization using connectionist reinforcement learning algorithms. Connection Science 3(3), 241–268.
  2. Ziebart, B. D., Maas, A., Bagnell, J. A. & Dey, A. K. (2008). Maximum Entropy Inverse Reinforcement Learning. AAAI 2008.
  3. Cui, G. et al. (2025). The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv preprint.

Mentioned in

entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy., epsilon-greedyepsilon-greedy, adj.: A simple rule for balancing trying new things against using what works: most of the time take the best-known option, but a small fraction of the time pick at random., exploration–exploitation trade-offexploration–exploitation trade-off, n.: The choice between trying something new that might turn out better and sticking with what already works — like picking a familiar restaurant versus testing a new one., proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., A3CA3C, abbr.: A 2016 DeepMind training method in which many copies of an AI agent learn in parallel on separate copies of a task, pooling their updates into one shared neural network., entropy collapseentropy collapse, n.: A failure in reinforcement-learning training where an AI quickly becomes overconfident and stops trying different answers, so it explores less and its improvement stalls., KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer., REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly., soft actor–criticsoft actor–critic, n.: A method for training AI agents, popular in robotics, that rewards the agent both for succeeding and for keeping its behavior varied, which helps it explore and learn reliably from reused experience.