autoencoder
In plain wordsA neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.
An encoder–decoderencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output. neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. whose training target is its own input: the encoder compresses each example into a code z, the decoder tries to rebuild the example from z, and the two are trained together to minimize reconstruction error, such as squared error. Since a network free to copy its input unchanged would learn nothing useful, the task is made hard on purpose: the code may be narrower than the input (an undercomplete bottleneck), penalized for having many active units (a sparsity penalty) or for reacting strongly to small input changes (contractive autoencoders), or computed from a corrupted or partly hidden input that must be restored (denoisingdenoising autoencoder, n.: An autoencoder trained to recover clean data from a deliberately damaged copy, such as an image with random pixels erased, which teaches it the structure of the data rather than how to copy. and maskedmasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of. autoencoders). A linear autoencoder trained on squared error ends up equivalent to principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores., recovering the same principal subspace, while nonlinear ones can learn curved, more compact codes. Uses include dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from., anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network. by reconstruction error, pretraining, compression and the latent spaces of latent diffusionlatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run.; the variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. makes the code probabilistic for generation.
Transactions the autoencoder reconstructs poorly are flagged for fraud review.
Origin
Auto-associative networks trained by backpropagation appeared in the mid-1980s, in work by David Rumelhart, Geoffrey Hinton and Ronald Williams (1986), Garrison Cottrell and colleagues on image compression (1987) and Hervé Bourlard and Yves Kamp (1988); Mark Kramer's 'nonlinear principal component analysis' (1991) generalized them. The name 'autoencoder' was in use by 1993 (Hinton and Richard Zemel). Hinton and Ruslan Salakhutdinov's 2006 Science paper on deep autoencoders helped launch the deep-learning revival.
Usage note
The sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. of interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box. research is trained on a model's internal activations, not on raw data, to extract interpretable featuresfeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity..
Related
- variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.
- denoising autoencoderdenoising autoencoder, n.: An autoencoder trained to recover clean data from a deliberately damaged copy, such as an image with random pixels erased, which teaches it the structure of the data rather than how to copy.
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- masked autoencodermasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of.
- latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates.
- encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output.
- principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
- representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
Sources
- Hinton, G. E. & Salakhutdinov, R. R. (2006). Reducing the Dimensionality of Data with Neural Networks. Science, 313(5786), 504–507.
- Hinton, G. E. & Zemel, R. S. (1994). Autoencoders, Minimum Description Length and Helmholtz Free Energy. Advances in Neural Information Processing Systems 6 (NIPS 1993).
Mentioned in
deepfakedeepfake, n.: A fake but realistic video, photo or voice recording made with AI that puts words in a real person's mouth or shows them in a scene that never happened., anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network., collaborative filteringcollaborative filtering, n.: Recommending things based on what people with similar tastes liked, without needing to know anything about the things themselves, as in 'people who bought this also bought…'., dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from., encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output., latent diffusion modellatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run., latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates., mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better., principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores., representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand., sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept., unsupervised learningunsupervised learning, n.: Machine learning without answer keys: the computer looks at unlabeled data and finds structure on its own, such as groups of similar customers or the main ways in which data vary., variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples., denoising autoencoderdenoising autoencoder, n.: An autoencoder trained to recover clean data from a deliberately damaged copy, such as an image with random pixels erased, which teaches it the structure of the data rather than how to copy., manifold hypothesismanifold hypothesis, n.: The idea that real-world data, such as photos of faces, fills only a thin, smoothly curved sliver of the vast space of all possible data — which is why learning from examples works at all., masked autoencodermasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of., restricted Boltzmann machinerestricted Boltzmann machine, n.: A two-layer network, one layer for the data and one hidden, that learns the patterns in data by trying to generate it. Stacks of them helped start the deep-learning revival around 2006., transposed convolutiontransposed convolution, n.: An operation that roughly reverses an image network's shrinking steps, enlarging a small grid of features into a bigger one, as when a model paints a full-size image., VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate.