reparameterization trick
In plain wordsA trick that rewrites a random draw as fixed numbers plus separate, outside randomness, so that a neural network can still learn through a step that involves chance.
A technique for computing low-variance gradients of an expectation with respect to the parameters of the distribution being sampled. Instead of drawing z directly from q_φ(z), one writes z as a deterministic, differentiable function of φ and parameter-free noise — for a Gaussian, z = μ + σ ⊙ ε with ε drawn from a standard normal — so that gradients flow through μ and σ by ordinary backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.. It is what makes the variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. trainable end to end, and it is also used in soft actor–criticsoft actor–critic, n.: A method for training AI agents, popular in robotics, that rewards the agent both for succeeding and for keeping its behavior varied, which helps it explore and learn reliably from reused experience., Bayesian neural networks and other stochastic models. Compared with the score-function (REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly.) estimator it usually has much lower variance, but it requires a differentiable sampling path; discrete variables need continuous relaxations such as the Gumbel-softmax (Concrete) distribution.
Origin
Pathwise gradient estimators were developed in operations research and simulation from the 1980s. Diederik Kingma and Max Welling named the 'reparameterization trick' in 'Auto-Encoding Variational Bayes' (posted December 2013); Danilo Rezende, Shakir Mohamed and Daan Wierstra described the same idea as stochastic backpropagation weeks later.
Related
- variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.
- evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn.
- REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly.
- backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
- soft actor–criticsoft actor–critic, n.: A method for training AI agents, popular in robotics, that rewards the agent both for succeeding and for keeping its behavior varied, which helps it explore and learn reliably from reused experience.
- policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely.
- REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly.
Sources
- Kingma, D. P. & Welling, M. (2013). Auto-Encoding Variational Bayes. arXiv:1312.6114. arxiv.org/abs/1312.6114
- Rezende, D. J., Mohamed, S. & Wierstra, D. (2014). Stochastic Backpropagation and Approximate Inference in Deep Generative Models. arXiv:1401.4082. arxiv.org/abs/1401.4082
Mentioned in
automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically., normal distributionnormal distribution, n.: The familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it., variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples., Diederik KingmaDiederik Kingma, prop. n.: A Dutch computer scientist who co-invented the variational autoencoder and the Adam optimizer, two of the most widely used tools for training neural networks., evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn., Gumbel-softmaxGumbel-softmax, n.: A mathematical trick that lets a neural network make a choice among separate options, such as picking one word or one category, while still learning through small, smooth adjustments., soft actor–criticsoft actor–critic, n.: A method for training AI agents, popular in robotics, that rewards the agent both for succeeding and for keeping its behavior varied, which helps it explore and learn reliably from reused experience., straight-through estimatorstraight-through estimator, n.: A shortcut that lets networks learn even when they contain steps with no useful slope, such as rounding: during learning, the computer simply pretends the step passed its input straight through.