Jensen–Shannon divergence
In plain wordsA balanced, two-way measure of how different two sets of probabilities are, comparing each with their average. Unlike its better-known cousin, it gives the same answer in both directions.
A way of comparing two probability distributionsprobability distribution, n.: A complete description of how likely each possible outcome is — for a fair die, one-sixth for each face. A language model produces one of these over its whole vocabulary at every step. P and Q through their midpoint M = (P + Q)/2: it averages the KL divergencesKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. KL(P ‖ M) and KL(Q ‖ M), which equals the entropy of M minus the mean of the two entropies. The construction removes KL's awkward features — the result is symmetric, never infinite and at most log 2 (one bit in base 2), vanishing only when P = Q — and its square root is a genuine metric, the Jensen–Shannon distance. The original generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people. objective, with an optimal discriminator, minimizes the JS divergence between real and generated data — which saturates at log 2 when the distributions do not overlap, starving the generator of useful gradients and motivating the Wasserstein distanceWasserstein distance, n.: A way of comparing two spreads of probability by imagining one as heaps of sand that must be shoveled into the shape of the other: the less shoveling, weighted by distance, the more alike they are. in Wasserstein GANs. It is also used to compare word distributions, corpora and model outputs, and as a bounded alternative to KL in evaluation and knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run..
The JS divergence between the two models' next-token distributions stayed under 0.05 bits on most prompts.
Origin
Named and studied by Jianhua Lin in 'Divergence Measures Based on the Shannon Entropy' (IEEE Transactions on Information Theory, 1991); the name joins Jensen's inequality, used to establish its properties, with Shannon entropy. Essentially the same quantity had appeared earlier, for example as Robin Sibson's 'information radius' (1969). Ian Goodfellow and colleagues' 2014 GAN paper made it widely known in machine learning.
Related
- Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right.
- Wasserstein distanceWasserstein distance, n.: A way of comparing two spreads of probability by imagining one as heaps of sand that must be shoveled into the shape of the other: the less shoveling, weighted by distance, the more alike they are.
- generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people.
- entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy.
- Jensen's inequalityJensen's inequality, n.: A rule saying that for a bowl-shaped curve, the average of the outputs is at least the output of the average. It is the hidden step behind many proofs in machine learning.
- mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated.
Sources
- Lin, J. (1991). Divergence Measures Based on the Shannon Entropy. IEEE Transactions on Information Theory, 37(1).
Mentioned in
Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right., Jensen's inequalityJensen's inequality, n.: A rule saying that for a bowl-shaped curve, the average of the outputs is at least the output of the average. It is the hidden step behind many proofs in machine learning., Wasserstein distanceWasserstein distance, n.: A way of comparing two spreads of probability by imagining one as heaps of sand that must be shoveled into the shape of the other: the less shoveling, weighted by distance, the more alike they are., Wasserstein GANWasserstein GAN, n.: A version of the forger-versus-detective image generator that measures how far its fakes are from real data in a smoother way, making training steadier and easier to monitor.