image captioning
In plain wordsHaving a computer write a short description of a picture, such as 'a dog catching a frisbee in a park', to help blind people, search engines and other AI systems.
The task of generating a natural-language description of an image's content. Early systems filled sentence templates with detected objects and attributes, or retrieved the captions of similar images (2010–2011). Neural captioning used an encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output.: a convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. encoded the image and a LSTMlong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences. generated the caption word by word ('Show and Tell', 2014), soon with attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. over image regions ('Show, Attend and Tell', 2015). Transformers and large-scale image–text pretraining followed, and vision-language modelsvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence. now write captions of any length and style. Datasets such as Flickr30k and COCO Captions (five human captions per image) support training and evaluation, with metrics that compare generated captions with references, such as BLEUBLEU, abbr.: An automatic score for machine translation that checks how many short runs of words in the computer's translation also appear in human reference translations., METEOR, CIDEr (2015) and SPICE, or with the image itself (CLIPScore, 2021); human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well. remains the reference standard. Uses include alt text for accessibility, image search and archives, and 'recaptioning' web images with detailed synthetic captions to train text-to-image modelstext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds..
Recaptioning the training set with long, detailed captions made the generator follow prompts far more closely.
Origin
Template- and retrieval-based captioning appeared around 2010–2011, for instance in Ali Farhadi and colleagues' 'Every Picture Tells a Story' (2010). Neural captioning arrived in several near-simultaneous papers in late 2014, notably Oriol Vinyals and colleagues' 'Show and Tell' at Google and Andrej Karpathy and Li Fei-Fei's work at Stanford.
Usage note
Captions reflect their annotators: models trained on web alt text inherit its omissions and stereotypes, and may describe what is typical rather than what is shown, a failure studied as object hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true..
Related
- visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words.
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
- encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output.
- COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems.
- CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image.
- BLEUBLEU, abbr.: An automatic score for machine translation that checks how many short runs of words in the computer's translation also appear in human reference translations.
Sources
- Vinyals, O., Toshev, A., Bengio, S. & Erhan, D. (2015). Show and Tell: A Neural Image Caption Generator. CVPR 2015.
- Xu, K. et al. (2015). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. ICML 2015.
Mentioned in
text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds., Andrej KarpathyAndrej Karpathy, prop. n.: A Slovak-Canadian AI researcher and teacher who co-founded OpenAI, led Tesla's self-driving vision team, and coined popular phrases such as 'vibe coding'., COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems., multimodal AImultimodal AI, n.: AI that can take in or produce more than one kind of information, such as text, images, sound and video, so you can show it a photo and ask questions about it., vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence., visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words., CLIP scoreCLIP score, n.: A score for how well a picture matches a text description, computed by an AI model that was trained to pair images with their captions., Oriol VinyalsOriol Vinyals, prop. n.: A Spanish computer scientist who co-created key neural methods for translation and image captioning, led DeepMind's StarCraft-playing AlphaStar, co-led Google's Gemini models, and in 2026 co-founded the startup Discovery Loop.