variational autoencoder
In plain wordsA neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.
A latent-variable generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them. trained by amortized variational inference. An encoder network maps an input x to the parameters of an approximate posterior q(z | x), usually a diagonal Gaussian over a low-dimensional latent vector z; a decoder network defines the likelihood p(x | z); and the prior p(z) is typically a standard normal. Both networks are trained jointly to maximize the evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn. — a reconstruction term minus the Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. from q(z | x) to the prior — with gradients passed through the sampling step by the reparameterization trickreparameterization trick, n.: A trick that rewrites a random draw as fixed numbers plus separate, outside randomness, so that a neural network can still learn through a step that involves chance.. New data are generated by decoding samples from the prior. VAEs train stably and learn a smooth, structured latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates., but with simple Gaussian decoders their samples tend to be blurry. Their main role today is as the compression stage of latent diffusionlatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run. and video models, where a VAE trained with added adversarial and perceptual lossesperceptual loss, n.: A way of judging how different two images look to a person by comparing what a trained image-recognition network 'sees' in each, instead of comparing pixels one by one. maps pixels to a much smaller latent grid.
The diffusion model never sees pixels; the VAE encodes each frame to a 4-channel latent first.
Origin
Introduced independently by Diederik Kingma and Max Welling in 'Auto-Encoding Variational Bayes' (posted December 2013; ICLR 2014) and by Danilo Rezende, Shakir Mohamed and Daan Wierstra (posted January 2014; ICML 2014), building on the Helmholtz machine of the 1990s. Notable variants include the β-VAE (Irina Higgins and colleagues, 2017) for disentangled factors and the discrete VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate. (2017).
Usage note
Despite the name, a VAE is not just an autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data. with added noise: its encoder outputs a distribution and its loss is a likelihood bound. A known failure, posterior collapse, occurs when a powerful decoder learns to ignore the latent code.
Related
- autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.
- evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn.
- reparameterization trickreparameterization trick, n.: A trick that rewrites a random draw as fixed numbers plus separate, outside randomness, so that a neural network can still learn through a step that involves chance.
- latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates.
- VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate.
- latent diffusion modellatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run.
- generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them.
- autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.
- generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people.
Sources
- Kingma, D. P. & Welling, M. (2013). Auto-Encoding Variational Bayes. arXiv:1312.6114 (ICLR 2014). arxiv.org/abs/1312.6114
- Rezende, D. J., Mohamed, S. & Wierstra, D. (2014). Stochastic Backpropagation and Approximate Inference in Deep Generative Models. arXiv:1401.4082 (ICML 2014). arxiv.org/abs/1401.4082
Mentioned in
Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it., autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data., density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare., encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output., generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people., generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them., ICLRICLR, prop. n.: A major deep-learning conference, held every spring since 2013, where the reviews of each submitted paper are posted online for anyone to read., Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right., latent diffusion modellatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run., latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates., mode collapsemode collapse, n.: When a generator gets stuck producing only a few kinds of output — the same face, the same joke — instead of the full variety it was meant to learn., normal distributionnormal distribution, n.: The familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it., posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces., probabilistic graphical modelprobabilistic graphical model, n.: A way of describing how many uncertain quantities depend on one another by drawing them as a network, which makes it practical for a computer to reason about all of them together., unsupervised learningunsupervised learning, n.: Machine learning without answer keys: the computer looks at unlabeled data and finds structure on its own, such as groups of similar customers or the main ways in which data vary., Diederik KingmaDiederik Kingma, prop. n.: A Dutch computer scientist who co-invented the variational autoencoder and the Adam optimizer, two of the most widely used tools for training neural networks., disentangled representationdisentangled representation, n.: An internal representation in which each part tracks one separate, meaningful factor of the data, such as one number for a face's angle, another for its lighting and another for hair color., evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn., free energy principlefree energy principle, n.: A sweeping theory by the neuroscientist Karl Friston that living things persist by keeping themselves in expected states — in effect, by constantly minimizing surprise about what they sense., Gumbel-softmaxGumbel-softmax, n.: A mathematical trick that lets a neural network make a choice among separate options, such as picking one word or one category, while still learning through small, smooth adjustments., image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures., information bottleneckinformation bottleneck, n.: The idea that a good internal summary of data keeps exactly the details needed for the task and discards everything else, as a good note-taker does., Jensen's inequalityJensen's inequality, n.: A rule saying that for a bowl-shaped curve, the average of the outputs is at least the output of the average. It is the hidden step behind many proofs in machine learning., normalizing flownormalizing flow, n.: A generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is.