Skip to content
Last Dictionary
Plane
Training & Post-trainingReinforcement LearningMathematics & Statistics

KL penalty

nounAlso: KL regularization, KL-regularized RL, KL control, KL coefficient, KL term

In plain wordsA cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.

  1. A regularization term proportional to the Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. between the policy being trained and a fixed reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started., subtracted from the reward (or added to the loss) in reinforcement-learning fine-tuning of language models. In reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. the reward becomes the reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. score minus β times the log-ratio of policy to reference probabilities, usually applied token by token. The penalty preserves fluency and diversity learned in pretraining, keeps the policy where the reward model is still accurate (limiting reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse.), and makes the optimal policy a reweighting of the reference distribution, the fact that direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely. exploits. The coefficient β trades reward against drift and may be fixed or adapted toward a target KL. Implementations differ in whether the KL enters the reward or the loss and in which sample estimator of KL they use.

    At β = 0.01 the policy drifted until its outputs turned repetitive; at 0.05 it stayed fluent.

max⁡πθ  Ex,  y∼πθ[rϕ(x,y)]−β DKL(πθ(⋅∣x) ∥ πref(⋅∣x))\max_{\pi_\theta}\; \mathbb{E}_{x,\; y \sim \pi_\theta}\big[r_\phi(x,y)\big] - \beta\, \mathbb{D}_{\mathrm{KL}}\big(\pi_\theta(\cdot \mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot \mid x)\big)

Origin

KL-regularized control has roots in stochastic optimal control ('KL control', associated with Hilbert Kappen and Emanuel Todorov in the 2000s). Natasha Jaques and colleagues (Sequence Tutor, 2016) penalized divergence from a pretrained sequence model during RL fine-tuning; Daniel Ziegler and colleagues at OpenAI (2019) made the KL penalty a standard ingredient of RLHF, and InstructGPT (2022) applied it per token against the SFT model.

Usage note

Not the step-size constraint of trust region policy optimizationtrust region policy optimization, n.: A method for improving an AI's strategy in careful steps, never changing it too much at once, so that a single bad update cannot wreck what it has already learned., which limits each update rather than total drift from a fixed reference. Several 2025 reasoning-RL recipes, such as DAPO, dropped the KL penalty, arguing that long chains of reasoning require large departures from the starting model.

Related

See also
  • Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right.
  • reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started.
  • reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
  • reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse.
  • proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
  • entropy regularizationentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early.
  • direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.
Contrast with
  • trust region policy optimizationtrust region policy optimization, n.: A method for improving an AI's strategy in careful steps, never changing it too much at once, so that a single bad update cannot wreck what it has already learned.

Sources

  1. Jaques, N., Gu, S., Bahdanau, D., Hernández-Lobato, J. M., Turner, R. E. & Eck, D. (2017). Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control. ICML 2017. arXiv:1611.02796. arxiv.org/abs/1611.02796
  2. Ziegler, D. M. et al. (2019). Fine-Tuning Language Models from Human Preferences. arXiv:1909.08593. arxiv.org/abs/1909.08593

Mentioned in

reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely., group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average., InstructGPTInstructGPT, prop. n.: A 2022 OpenAI language model trained with human feedback to follow instructions; people preferred its answers to those of a model 100 times larger, and its recipe led directly to ChatGPT., Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right., proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse., elastic weight consolidationelastic weight consolidation, n.: A way to help a neural network learn a new task without forgetting an old one, by making the settings that mattered most for the old task harder to change, as if tied down with springs., entropy regularizationentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early., Lagrange multiplierLagrange multiplier, n.: A mathematical device for finding the best answer when strict rules must be obeyed: each rule becomes an extra price that grows when the rule is broken., mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted., reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started., stability–plasticity dilemmastability–plasticity dilemma, n.: The basic tension any learning system faces: it must change enough to learn new things, yet stay stable enough not to lose what it already knows.