vision transformer
In plain wordsA way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence.
A transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. encoder applied to images by splitting them into a grid of fixed-size patches (typically 14 or 16 pixels square), mapping each patch to a vector with a patch embeddingpatch embedding, n.: The first step of a vision transformer: cutting an image into small square tiles and turning each tile into a list of numbers that the model can process like a word., adding position embeddingspositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model. and processing the resulting sequence with standard self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to. layers; a learnable classification token, or an average of the patch outputs, summarizes the image. Names such as ViT-B/16 give the model size (Base, Large, Huge, giant) and the patch size. Without a convolution's built-in assumptions of locality and translation equivarianceequivariance, n.: A property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way., the original ViT overtook strong CNNsconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. only after pretraining on very large labeled collections, ImageNet-21k (about 14 million images) or Google's internal JFT-300M; better training recipes with knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run. (DeiT, 2020) and hierarchical, windowed variants (Swin Transformer, 2021) largely removed that limitation. ViTs are now the standard image encoder in CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image.-style models, self-supervised methods (MAEmasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of., DINODINO, prop. n.: A family of Meta vision models that learn to understand images from millions of unlabeled photos, by training one network to agree with a slowly updated copy of itself.), segmentation models and vision-language modelsvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence., where their patch outputs become the visual tokens read by the language model.
We swapped the ResNet for a ViT-L/14 pretrained on image–text pairs and retrieval accuracy jumped.
Origin
Introduced by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn and colleagues at Google Brain in 'An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale' (arXiv October 2020; ICLR 2021). Earlier work had applied attention to pixels; ViT showed that a nearly unmodified language transformer works on patches at scale.
Usage note
Lowercase 'vision transformer' names the general architecture; 'ViT' usually means the Google model family. Attention cost grows with the square of the number of patches, so higher image resolution is expensive.
Related
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- patch embeddingpatch embedding, n.: The first step of a vision transformer: cutting an image into small square tiles and turning each tile into a list of numbers that the model can process like a word.
- convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
- self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.
- masked autoencodermasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of.
- CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image.
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
- convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
Sources
- Dosovitskiy, A. et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021. arxiv.org/abs/2010.11929
- Touvron, H. et al. (2021). Training Data-Efficient Image Transformers & Distillation Through Attention. ICML 2021.
- Liu, Z. et al. (2021). Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. ICCV 2021.
Mentioned in
computer visioncomputer vision, n.: The part of AI that lets computers make sense of images and video: recognizing faces, reading signs, spotting tumors on scans or judging how far away a car is., image classificationimage classification, n.: Teaching a computer to say what a picture shows by choosing a label, such as 'cat', 'pneumonia' or 'stop sign', from a fixed list of possibilities., transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., backbonebackbone, n.: The main, usually pretrained part of a vision model that turns an image into useful internal features; smaller task-specific parts are attached to it, like limbs to a spine., CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image., convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects., diffusion transformerdiffusion transformer, n.: A diffusion image or video generator whose core network is a transformer — the architecture behind chatbots — instead of the older image-specific design, which lets it improve more predictably as it grows., GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between., ICLRICLR, prop. n.: A major deep-learning conference, held every spring since 2013, where the reviews of each submitted paper are posted online for anyone to read., multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type., ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI., Segment Anything ModelSegment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it., spectrogramspectrogram, n.: A picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns., stridestride, n.: How far a scanning window jumps each time it moves across an image. Jumping two pixels instead of one halves the size of the result., vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence., action recognitionaction recognition, n.: Teaching computers to recognize what people are doing in a video, such as running, waving, cooking or falling, rather than just which objects appear in it., ConvNeXtConvNeXt, prop. n.: A 2022 image-recognition network showing that an older-style convolutional design, updated with lessons from transformers, can match vision transformers., DINODINO, prop. n.: A family of Meta vision models that learn to understand images from millions of unlabeled photos, by training one network to agree with a slowly updated copy of itself., image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures., Jakob UszkoreitJakob Uszkoreit, prop. n.: A German computer scientist, one of the eight authors of the transformer paper, credited with first proposing to build language models on attention alone., masked autoencodermasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of., patch embeddingpatch embedding, n.: The first step of a vision transformer: cutting an image into small square tiles and turning each tile into a list of numbers that the model can process like a word., sharpness-aware minimizationsharpness-aware minimization, n.: A training method that looks for settings where the error stays low even if the settings are nudged a little, rather than settings that are merely low at one precise point.