evidence lower bound
In plain wordsA quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn.
A lower bound on the log marginal likelihood (the 'evidence') log p(x) of a latent-variable model, used as a tractable training objective in variational inference. For any distribution q over the latent variables z — in a variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples., the encoder's q(z | x) — the log-evidence equals the ELBO plus the Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. from q to the true posterior p(z | x). Because that divergence is never negative, the ELBO never exceeds log p(x), and maximizing it both fits the model and pulls q toward the true posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.. It is usually written as an expected reconstruction log-likelihood minus a KL penalty between q and the priorprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns.. The standard training losses of diffusion modelsdenoising diffusion probabilistic model, n.: The 2020 recipe that made diffusion image generators work well: add noise to pictures over a thousand small steps, then train a network to guess the noise so it can be removed. are reweighted ELBOs summed over noise levels, and tighter bounds, such as the importance-weighted ELBO, average over several samples.
Origin
The bound follows from Jensen's inequality and is the negative of the variational free energy of statistical physics; Geoffrey Hinton and colleagues tied it to neural-network learning in the 1990s (Hinton and Richard Zemel, 1993; the Helmholtz machine, 1995). The first use of 'evidence lower bound' is not established; the name and acronym were current by 2012, as in Matthew Hoffman, David Blei and colleagues' stochastic variational inference, and spread widely with VAEs, although Kingma and Welling called it the variational lower bound.
Usage note
'Evidence' is Bayesian shorthand for the marginal likelihood p(x), not evidence in the everyday sense.
Related
- variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.
- Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right.
- Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.
- likelihoodlikelihood, n.: A score for a possible explanation of some data: how probable that explanation, or that setting of a model, would have made the data actually seen.
- reparameterization trickreparameterization trick, n.: A trick that rewrites a random draw as fixed numbers plus separate, outside randomness, so that a neural network can still learn through a step that involves chance.
- posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.
Sources
- Kingma, D. P. & Welling, M. (2013). Auto-Encoding Variational Bayes. arXiv:1312.6114. arxiv.org/abs/1312.6114
- Hoffman, M. D., Blei, D. M., Wang, C. & Paisley, J. (2013). Stochastic Variational Inference. Journal of Machine Learning Research 14. arXiv:1206.7051. arxiv.org/abs/1206.7051
- Blei, D. M., Kucukelbir, A. & McAuliffe, J. D. (2017). Variational Inference: A Review for Statisticians. Journal of the American Statistical Association 112(518).
Mentioned in
expectation–maximization algorithmexpectation–maximization algorithm, n.: A method for fitting a model when some information is hidden: guess the hidden parts using the current model, refit the model as if those guesses were right, and repeat until nothing changes., expected valueexpected value, n.: The long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5., Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right., objective functionobjective function, n.: The score that an optimization process tries to push as low, or as high, as possible. For an AI, it defines what 'doing well' means during training., variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples., denoising diffusion probabilistic modeldenoising diffusion probabilistic model, n.: The 2020 recipe that made diffusion image generators work well: add noise to pictures over a thousand small steps, then train a network to guess the noise so it can be removed., Diederik KingmaDiederik Kingma, prop. n.: A Dutch computer scientist who co-invented the variational autoencoder and the Adam optimizer, two of the most widely used tools for training neural networks., free energy principlefree energy principle, n.: A sweeping theory by the neuroscientist Karl Friston that living things persist by keeping themselves in expected states — in effect, by constantly minimizing surprise about what they sense., importance samplingimportance sampling, n.: A shortcut for estimating averages: draw samples from a convenient source, then reweight each one to correct for how much more or less likely it would be under the source you actually care about., information bottleneckinformation bottleneck, n.: The idea that a good internal summary of data keeps exactly the details needed for the task and discards everything else, as a good note-taker does., Jensen's inequalityJensen's inequality, n.: A rule saying that for a bowl-shaped curve, the average of the outputs is at least the output of the average. It is the hidden step behind many proofs in machine learning., reparameterization trickreparameterization trick, n.: A trick that rewrites a random draw as fixed numbers plus separate, outside randomness, so that a neural network can still learn through a step that involves chance., variational inferencevariational inference, n.: A way of approximating a hard-to-compute probability distribution by picking the closest match from a simpler family of distributions, turning a hard problem of adding up every possibility into an optimization problem.