CLIP
In plain wordsAn OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image.
'Contrastive Language–Image Pre-training': a pair of encoders, one for images (a ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI. or vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence.) and one for text (a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.), trained so that a picture and its caption land close together in a shared embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. space. In each batch of N captioned images, every image must pick its own caption from all N, and every caption its own image, via a two-way cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. (InfoNCE) objective over cosine similaritiescosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things. scaled by a learned temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.. Trained on some 400 million captioned web images, CLIP classifies 'zero-shot' by embedding a prompt such as 'a photo of a dog' for each class and choosing the nearest; so it equaled a supervised ResNet-50 on ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom. without using any ImageNet training labels. CLIP-style encoders became core infrastructure: text encoders for text-to-image modelstext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds. such as Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it., vision encoders for vision-language modelsvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence., image search, dataset filtering (as for LAION-5BLAION-5B, prop. n.: A free list of nearly six billion links to online images paired with their captions, released in 2022 and used to train image generators such as Stable Diffusion.) and the CLIP scoreCLIP score, n.: A score for how well a picture matches a text description, computed by an AI model that was trained to pair images with their captions. metric. Weaknesses include poor counting and spatial reasoning, sensitivity to text inside images, and social biases absorbed from web data.
We rank the generated images by CLIP score against the prompt and keep the top four.
Origin
Introduced by Alec Radford, Jong Wook Kim, Chris Hallacy and colleagues at OpenAI, announced in January 2021 with the paper 'Learning Transferable Visual Models From Natural Language Supervision' (arXiv February 2021; ICML 2021). Google's ALIGN (2021), the open reproduction OpenCLIP (from 2021) and Google's sigmoid-loss SigLIP (2023) followed.
Usage note
OpenAI released CLIP's weights but not its training data; open replications trained on LAION data made CLIP-style models widely reproducible. In the formula, s_ij is the cosine similarity of image i and text j, and τ the temperature.
Related
- contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things.
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
- zero-shot learningzero-shot learning, n.: Doing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training.
- text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
- vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence.
- open-vocabulary detectionopen-vocabulary detection, n.: Object detection where you type what to look for, such as 'red umbrella' or 'forklift', and the system finds it, even if it was never trained on that exact category.
Sources
- Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML 2021. arxiv.org/abs/2103.00020
- Zhai, X., Mustafa, B., Kolesnikov, A. & Beyer, L. (2023). Sigmoid Loss for Language Image Pre-Training. ICCV 2023.
Mentioned in
computer visioncomputer vision, n.: The part of AI that lets computers make sense of images and video: recognizing faces, reading signs, spotting tumors on scans or judging how far away a car is., embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., image classificationimage classification, n.: Teaching a computer to say what a picture shows by choosing a label, such as 'cat', 'pneumonia' or 'stop sign', from a fixed list of possibilities., Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it., text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds., Alec RadfordAlec Radford, prop. n.: An American AI researcher who led OpenAI's first GPT language models and its CLIP and Whisper systems, key steps toward today's chatbots and multimodal AI., contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things., cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things., DALL·EDALL·E, prop. n.: OpenAI's pioneering series of image generators, named after the painter Salvador Dalí and the robot WALL·E, which showed in 2021–2023 that AI could draw almost anything described in words., feature extractionfeature extraction, n.: Turning raw data such as pixels or sound into a short list of telling numbers — computed by hand-designed rules or by an AI system trained earlier — that a new model can learn from or compare., foundation modelfoundation model, n.: A very large AI model, built once at great cost from vast amounts of text, images or code gathered mostly from the internet, that many products then reuse as their starting point, from chatbots to coding tools., Fréchet inception distanceFréchet inception distance, n.: A score for image generators that compares the overall statistics of their pictures with those of real photos, as seen by an image-recognition network; lower is better., ICMLICML, prop. n.: A leading machine learning conference, first held as a workshop in 1980, where researchers gather each summer to present new learning methods and theory., image captioningimage captioning, n.: Having a computer write a short description of a picture, such as 'a dog catching a frisbee in a park', to help blind people, search engines and other AI systems., LAIONLAION, prop. n.: A mostly volunteer group, registered as a German nonprofit, that compiled free lists of billions of online pictures and their captions. Stable Diffusion and other image generators were trained on them., multimodal AImultimodal AI, n.: AI that can take in or produce more than one kind of information, such as text, images, sound and video, so you can show it a photo and ask questions about it., multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type., self-supervised learningself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image., vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence., vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence., zero-shot learningzero-shot learning, n.: Doing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training., classifier guidanceclassifier guidance, n.: A way to push a diffusion image generator toward a requested category by consulting a separate image-recognition model at every step, trading variety for images that more clearly show the target., CLIP scoreCLIP score, n.: A score for how well a picture matches a text description, computed by an AI model that was trained to pair images with their captions., LAION-5BLAION-5B, prop. n.: A free list of nearly six billion links to online images paired with their captions, released in 2022 and used to train image generators such as Stable Diffusion.