Skip to content
Last Dictionary
Plane
Vision, Speech & PerceptionDeep LearningTraining & Post-training

masked autoencoder

nounAlso: MAE, masked image modeling, MIM, VideoMAE, BEiT, SimMIM

In plain wordsA way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of.

  1. A self-supervisedself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image. pretraining method for vision transformersvision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence. built on a fill-in-the-blanks task: most of an image's patches (75% in the original), chosen at random, are hidden, and the network must restore their pixel values. The work is split unevenly. A large encoder sees only the patches that remain, never placeholders for the hidden ones, which makes each training step several times cheaper; a small decoder then takes the encoded patches together with learned mask tokens and predicts the hidden patches' normalized pixels, scored by mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better. on those patches alone. After pretraining the decoder is discarded and the encoder is fine-tunedfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way. or used as a backbonebackbone, n.: The main, usually pretrained part of a vision model that turns an image into useful internal features; smaller task-specific parts are attached to it, like limbs to a spine.. The high masking ratio stops the model from solving the task by interpolating between neighbors, forcing more holistic representations. MAE belongs to the wider family of masked image modeling, alongside BEiT (2021), which predicts discrete visual tokens, and SimMIM; variants exist for video (VideoMAE, masking about 90%), audio spectrograms and 3D point clouds.

    We pretrained the encoder as a masked autoencoder on two million unlabeled microscope tiles before fine-tuning.

L=1∣M∣∑i∈M∥xi−x^i∥22\mathcal{L} = \frac{1}{|M|}\sum_{i \in M} \lVert \mathbf{x}_i - \hat{\mathbf{x}}_i \rVert_2^2

Origin

Introduced by Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár and Ross Girshick of Facebook AI Research in 'Masked Autoencoders Are Scalable Vision Learners' (arXiv November 2021; CVPR 2022), adapting BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.'s masked-token objective to images; denoising autoencoders (2008) and context encoders (2016) were precursors.

Usage note

Not to be confused with MADE, the 'Masked Autoencoder for Distribution Estimation' (Germain et al., 2015), in which masking a network's connections enforces an autoregressive density model. In the formula, M is the set of masked patches.

Related

See also
  • self-supervised learningself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image.
  • autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.
  • vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence.
  • masked language modelmasked language model, n.: A model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example.
  • DINODINO, prop. n.: A family of Meta vision models that learn to understand images from millions of unlabeled photos, by training one network to agree with a slowly updated copy of itself.
  • representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
Contrast with
  • contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things.

Sources

  1. He, K., Chen, X., Xie, S., Li, Y., Dollár, P. & Girshick, R. (2022). Masked Autoencoders Are Scalable Vision Learners. CVPR 2022. arxiv.org/abs/2111.06377
  2. Bao, H., Dong, L., Piao, S. & Wei, F. (2022). BEiT: BERT Pre-Training of Image Transformers. ICLR 2022.

Mentioned in

autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data., Kaiming HeKaiming He, prop. n.: A Chinese computer scientist whose ResNet design, with shortcut connections that let networks grow hundreds of layers deep, is among the most cited research in all of science., Segment Anything ModelSegment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it., self-supervised learningself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image., vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence., action recognitionaction recognition, n.: Teaching computers to recognize what people are doing in a video, such as running, waving, cooking or falling, rather than just which objects appear in it., ConvNeXtConvNeXt, prop. n.: A 2022 image-recognition network showing that an older-style convolutional design, updated with lessons from transformers, can match vision transformers., denoising autoencoderdenoising autoencoder, n.: An autoencoder trained to recover clean data from a deliberately damaged copy, such as an image with random pixels erased, which teaches it the structure of the data rather than how to copy., DINODINO, prop. n.: A family of Meta vision models that learn to understand images from millions of unlabeled photos, by training one network to agree with a slowly updated copy of itself., joint-embedding predictive architecturejoint-embedding predictive architecture, n.: A way of training AI on images or video in which the model predicts an abstract summary of a hidden part, rather than every pixel, so it learns what matters and ignores unpredictable detail.