Kullback–Leibler divergence
In plain wordsA score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right.
A nonnegative, asymmetric score of how far a distribution q is from a target distribution p, zero only when they coincide: KL(p ‖ q) is the expectationexpected value, n.: The long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5. under p of log(p(x)/q(x)). Because the two directions generally differ and the triangle inequality can fail, it is called a divergence rather than a distance; in coding terms it is the overhead, in bits or nats per symbol, of compressing data from p with a code built for q, so it equals cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. minus entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy.. Direction matters: 'forward' KL(p ‖ q), minimized by maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this., makes q cover all of p's mass, while 'reverse' KL(q ‖ p), minimized in variational inferencevariational inference, n.: A way of approximating a hard-to-compute probability distribution by picking the closest match from a simpler family of distributions, turning a hard problem of adding up every possibility into an optimization problem., lets q lock onto a single mode. KL divergence runs through AI: the evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn. of VAEsvariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples., the KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer. that keeps a policy tuned by reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. near its reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started., the implicit constraint of direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely., knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run. losses, trust regions in trust region policy optimizationtrust region policy optimization, n.: A method for improving an AI's strategy in careful steps, never changing it too much at once, so that a single bad update cannot wreck what it has already learned., and measures of distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested..
We stopped the RL run once the KL divergence from the reference model passed 10 nats.
Origin
Introduced by Solomon Kullback and Richard Leibler in 'On Information and Sufficiency' (Annals of Mathematical Statistics, 1951) as the information for discriminating between two hypotheses, and developed in Kullback's Information Theory and Statistics (1959). Harold Jeffreys had used a symmetrized form in 1946; information theory's 'relative entropy' is the same quantity.
Related
- cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
- entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy.
- Jensen–Shannon divergenceJensen–Shannon divergence, n.: A balanced, two-way measure of how different two sets of probabilities are, comparing each with their average. Unlike its better-known cousin, it gives the same answer in both directions.
- KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.
- variational inferencevariational inference, n.: A way of approximating a hard-to-compute probability distribution by picking the closest match from a simpler family of distributions, turning a hard problem of adding up every possibility into an optimization problem.
- evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn.
- mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated.
- Wasserstein distanceWasserstein distance, n.: A way of comparing two spreads of probability by imagining one as heaps of sand that must be shoveled into the shape of the other: the less shoveling, weighted by distance, the more alike they are.
Sources
- Kullback, S. & Leibler, R. A. (1951). On Information and Sufficiency. Annals of Mathematical Statistics, 22(1).
Mentioned in
knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text., entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy., information gaininformation gain, n.: How much a question reduces uncertainty about the answer. A decision tree asks first the question that removes the most uncertainty, measured in bits., information theoryinformation theory, n.: A branch of mathematics that treats information as surprise — the less predictable a message, the more it tells you — and works out how far data can be compressed and how reliably it can be sent., maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this., model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm., training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it., t-SNEt-SNE, n.: A technique for drawing data that have many measurements per item as a flat map in which similar items land close together, widely used to look for groups in AI models' internal data or in genetic data., variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples., Akaike information criterionAkaike information criterion, n.: A score for comparing statistical models that rewards fitting the data well but charges a penalty for every extra adjustable number, to avoid picking needlessly complicated models. Lower is better., automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand., evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn., Fisher informationFisher information, n.: A number, or table of numbers, saying how sharply the probability of the data reacts when a model's settings are nudged; the sharper the reaction, the more precisely the data can pin those settings down., free energy principlefree energy principle, n.: A sweeping theory by the neuroscientist Karl Friston that living things persist by keeping themselves in expected states — in effect, by constantly minimizing surprise about what they sense., Inception scoreInception score, n.: An early score for image generators that rewards pictures an image-recognition network can label confidently and a collection that covers many different labels; higher is better., irreducible lossirreducible loss, n.: The part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence., Jensen's inequalityJensen's inequality, n.: A rule saying that for a bowl-shaped curve, the average of the outputs is at least the output of the average. It is the hidden step behind many proofs in machine learning., Jensen–Shannon divergenceJensen–Shannon divergence, n.: A balanced, two-way measure of how different two sets of probabilities are, comparing each with their average. Unlike its better-known cousin, it gives the same answer in both directions., KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer., mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated., on-policy distillationon-policy distillation, n.: A way of teaching a small AI from a bigger one where the small AI writes its own answers and the bigger one grades every word, so the learner is corrected on its own mistakes., plug and play language modelplug and play language model, n.: An early way to steer what a language model writes — its topic or mood — by letting a small add-on classifier nudge the model's internal state at each word, without retraining the model., trust region policy optimizationtrust region policy optimization, n.: A method for improving an AI's strategy in careful steps, never changing it too much at once, so that a single bad update cannot wreck what it has already learned.