denoising autoencoder
In plain wordsAn autoencoder trained to recover clean data from a deliberately damaged copy, such as an image with random pixels erased, which teaches it the structure of the data rather than how to copy.
An autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data. trained to reconstruct a clean input from a corrupted version of it, damaged for example by erasing a random subset of input values (masking noise), by setting random pixels to black or white, or by adding Gaussian noise. Because copying the input no longer solves the task, the network must learn the statistical structure that lets it restore missing or distorted parts. With small Gaussian noise, the optimal denoiser is closely related to the score, the gradient of the log-density of the data, a link later exploited by score-basedscore-based generative model, n.: A generator that learns, for any noisy example, which direction would make it look more like real data — then starts from pure noise and follows those directions step by step. and diffusion modelsdiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static., which are in effect denoisers applied across many noise levels. Stacked denoising autoencoders were used for layer-wise pretraining of deep networks before about 2012. The same corrupt-and-restore principle underlies masked language modelingmasked language model, n.: A model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example. and masked autoencodersmasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of. in vision.
We pretrained each layer as a denoising autoencoder with 30 percent of inputs masked.
Origin
Introduced by Pascal Vincent, Hugo Larochelle, Yoshua Bengio and Pierre-Antoine Manzagol in 'Extracting and Composing Robust Features with Denoising Autoencoders' (ICML 2008); stacked versions followed in 2010. Mark Kramer's 1992 robust autoassociative networks anticipated the idea, and Vincent connected denoising to score matching in 2011.
Related
- autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.
- diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
- score-based generative modelscore-based generative model, n.: A generator that learns, for any noisy example, which direction would make it look more like real data — then starts from pure noise and follows those directions step by step.
- masked language modelmasked language model, n.: A model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example.
- masked autoencodermasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of.
- self-supervised learningself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image.
Sources
- Vincent, P., Larochelle, H., Bengio, Y. & Manzagol, P.-A. (2008). Extracting and Composing Robust Features with Denoising Autoencoders. Proceedings of ICML 2008.
- Vincent, P. (2011). A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7), 1661–1674.
Mentioned in
autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.