Jensen's inequality

The inequality f(E[X]) ≤ E[f(X)] for any convex function f and random variablerandom variable, n.: A quantity whose value depends on chance, such as the number rolled on a die or the next word a chatbot will write. X — the probabilistic form of the fact that a chord of a convex curve lies above the curve — with the direction reversed for concave functions such as the logarithm. It underpins much of probabilistic machine learning. Applied to the concave logarithm it gives log E[w] ≥ E[log w], which yields the evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn. used in variational inferencevariational inference, n.: A way of approximating a hard-to-compute probability distribution by picking the closest match from a simpler family of distributions, turning a hard problem of adding up every possibility into an optimization problem. and variational autoencodersvariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. and the monotone-improvement proof of the expectation–maximization algorithmexpectation–maximization algorithm, n.: A method for fitting a model when some information is hidden: guess the hidden parts using the current model, refit the model as if those guesses were right, and repeat until nothing changes.. It shows that the Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. and mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated. are nonnegative, and it explains why, for a convex loss, an ensemble's averaged prediction scores at least as well as its members do on average. It also gives its name to the Jensen–Shannon divergenceJensen–Shannon divergence, n.: A balanced, two-way measure of how different two sets of probabilities are, comparing each with their average. Unlike its better-known cousin, it gives the same answer in both directions..

Applying Jensen's inequality to the log marginal likelihood gives the ELBO in one line.