multimodal AI
In plain wordsAI that can take in or produce more than one kind of information, such as text, images, sound and video, so you can show it a photo and ask questions about it.
AI systems that work across more than one kind of data, taking in, linking or producing combinations of text, images, audio, video, sensor readings or robot actions. Designs differ in where modalities meet: separate encoders aligned in a shared embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. space, as in CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image.; adapters that project image or audio features into a language model'slarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. input; and natively multimodal models trained from the start on interleaved tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. from several modalities, some of which can also output images or speech. Typical capabilities include document and chart understanding, image captioningimage captioning, n.: Having a computer write a short description of a picture, such as 'a dog catching a frisbee in a park', to help blind people, search engines and other AI systems., visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words., spoken conversation and text-to-imagetext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds. or video generation; in robotics, vision-language-action modelsvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements. add motor actions. Key difficulties are aligning modalities, collecting paired data and hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true. about visual details.
The multimodal model read the scanned invoice, checked the totals and flagged a missing signature.
Origin
'Multimodal' came from psychology and human–computer interaction, for interfaces combining speech, gesture and vision (Richard Bolt's 'Put-That-There', 1980). Multimodal deep learning took shape around 2011 (Jiquan Ngiam and colleagues) and accelerated with CLIP (2021). GPT-4 (March 2023) was announced with image input; Google presented Gemini (December 2023) as natively multimodal, and GPT-4o (May 2024) handled text, audio and images in one model.
Usage note
'Multimodal' describes inputs and outputs, not general intelligence. Some products called multimodal are pipelines that convert speech or images to text before a text-only model sees them, losing information such as tone of voice.
Related
- multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type.
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
- CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image.
- speech-to-speech modelspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice.
- vision-language-action modelvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements.
- text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
Sources
- Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H. & Ng, A. Y. (2011). Multimodal Deep Learning. Proceedings of ICML 2011.
- Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of ICML 2021.
Mentioned in
Google DeepMindGoogle DeepMind, prop. n.: Google's main artificial intelligence laboratory, formed in 2023 by joining DeepMind and Google Brain. It builds the Gemini models and scientific AI systems such as AlphaFold., GPT-4GPT-4, prop. n.: OpenAI's 2023 language model, a big step up from the one behind the first ChatGPT. It scored well on many professional and academic exams and could also read images., GPT-4oGPT-4o, prop. n.: An OpenAI model from 2024 that handles text, images and speech in one system, enabling fast, natural voice conversations. Many users grew attached to its personality, which made its retirement controversial., multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type., perceptionperception, n.: How an AI system or robot turns raw sensor signals, such as camera pixels, microphone sound or laser scans, into an understanding of what is around it: objects, people, words and distances., sensor fusionsensor fusion, n.: Combining readings from several different sensors, such as cameras, radar, lidar, GPS and motion sensors, to get a more accurate and reliable picture than any one sensor gives alone.