vision-language-action model
In plain wordsAn AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements.
A robot foundation modelrobot foundation model, n.: One large AI model trained on huge amounts of robot and internet data so that it can control many robots on many tasks, and be adapted quickly to new ones. made by teaching a pretrained vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence. to act: from camera images, a natural-language instruction and usually the robot's joint readings it produces motor commands, so that what the model absorbed from web-scale images and text — object names, spatial relations, simple reasoning — carries over to control. Actions are emitted either as discrete tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. decoded autoregressively (RT-2 split each action dimension into 256 bins) or as continuous values from a diffusiondiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. or flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples. 'action expert' attached to the language backbone, as in π0, which outputs chunksaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer. of future actions for smooth, high-frequency control. Training mixes demonstration data from many robots (Open X-EmbodimentOpen X-Embodiment, prop. n.: A large shared collection of robot training data, pooled by dozens of research labs in 2023 and covering many kinds of robots, so that one AI model could learn from all of them.) with vision-language data that preserves general knowledge. Many systems pair a slower reasoning component with a fast reactive controller. Open problems include reliability over long tasks, contact-rich dexteritydexterous manipulation, n.: Skilled handling of objects with robot fingers, such as spinning a pen, turning a key or rotating a cube in one hand: the fine finger work people do without thinking., safety and comparable evaluation.
We fine-tuned an open VLA on 200 demonstrations, and it handled mugs it had never seen.
Origin
Named in Google DeepMind's RT-2 paper (Anthony Brohan and colleagues, July 2023), which called its models 'vision-language-action models (VLA)'. Open-weight successors followed, including OpenVLA (7 billion parameters, June 2024) and Physical Intelligence's π0 (October 2024); by 2025–2026 systems such as Gemini Robotics, NVIDIA's GR00T N models, Figure's Helix and later π models made VLAs the dominant paradigm in robot learning.
Usage note
Not every robot foundation model is a VLA: some take no language input or are trained without a vision-language backbone. Reported success rates depend heavily on the robots, tasks and scenes used and are rarely comparable across papers.
Related
- RT-2RT-2, prop. n.: A 2023 Google DeepMind model that controlled robot arms with a chatbot-style vision-and-language model, letting robots follow instructions about objects and ideas they had never practiced with.
- robot foundation modelrobot foundation model, n.: One large AI model trained on huge amounts of robot and internet data so that it can control many robots on many tasks, and be adapted quickly to new ones.
- generalist robot policygeneralist robot policy, n.: A single robot-control program trained to do many different tasks in many places, often on several kinds of robots: the opposite of a specialist trained for one job.
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
- diffusion policydiffusion policy, n.: A way of training robots in which the same kind of AI that generates images instead generates the robot's next movements, letting it copy demonstrations even when there are several good ways to do a task.
- action chunkingaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer.
- Open X-EmbodimentOpen X-Embodiment, prop. n.: A large shared collection of robot training data, pooled by dozens of research labs in 2023 and covering many kinds of robots, so that one AI model could learn from all of them.
- cross-embodiment learningcross-embodiment learning, n.: Training one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another.
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
Sources
- Brohan, A. et al. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv:2307.15818. arxiv.org/abs/2307.15818
- Kim, M. J. et al. (2024). OpenVLA: An Open-Source Vision-Language-Action Model. arXiv:2406.09246. arxiv.org/abs/2406.09246v3
- Black, K. et al. (2024). π0: A Vision-Language-Action Flow Model for General Robot Control. arXiv:2410.24164. arxiv.org/abs/2410.24164v1
Mentioned in
humanoid robothumanoid robot, n.: A robot built roughly in human shape, with a head, torso, two arms and usually two legs, so that it can work in spaces and with tools designed for people., physical AIphysical AI, n.: AI built to work in the physical world rather than on a screen: the intelligence inside robots, self-driving cars, drones and other machines that must sense real surroundings and move through them., robotrobot, n.: A machine that senses its surroundings, decides what to do and physically acts, moving itself or handling objects, with some independence from a human operator. Factory arms, robot vacuums and self-driving cars are all robots., embodied AIembodied AI, n.: AI that has a body, usually a robot and sometimes a simulated character, and learns or acts by sensing and moving in a world rather than only processing text or images., end effectorend effector, n.: The tool at the end of a robot arm that actually touches the world, such as a gripper, suction cup, welding torch, paint sprayer or robotic hand., imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error., manipulationmanipulation, n.: A robot's use of arms, hands or grippers to change the world around it: picking things up, moving, pushing, opening, assembling or using tools., multimodal AImultimodal AI, n.: AI that can take in or produce more than one kind of information, such as text, images, sound and video, so you can show it a photo and ask questions about it., robot foundation modelrobot foundation model, n.: One large AI model trained on huge amounts of robot and internet data so that it can control many robots on many tasks, and be adapted quickly to new ones., robot learningrobot learning, n.: Teaching robots skills from data and experience, such as watching demonstrations, practicing by trial and error or training in simulation, instead of programming every movement by hand., RT-2RT-2, prop. n.: A 2023 Google DeepMind model that controlled robot arms with a chatbot-style vision-and-language model, letting robots follow instructions about objects and ideas they had never practiced with., vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence., action chunkingaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer., CoRLCoRL, prop. n.: A yearly academic conference, started in 2017, for research where robotics meets machine learning., cross-embodiment learningcross-embodiment learning, n.: Training one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another., dexterous manipulationdexterous manipulation, n.: Skilled handling of objects with robot fingers, such as spinning a pen, turning a key or rotating a cube in one hand: the fine finger work people do without thinking., diffusion policydiffusion policy, n.: A way of training robots in which the same kind of AI that generates images instead generates the robot's next movements, letting it copy demonstrations even when there are several good ways to do a task., Gemini RoboticsGemini Robotics, prop. n.: A family of AI models from Google DeepMind, built on its Gemini models, that let robots understand what they see and what people ask of them, and turn that understanding into physical actions., generalist robot policygeneralist robot policy, n.: A single robot-control program trained to do many different tasks in many places, often on several kinds of robots: the opposite of a specialist trained for one job., goal-conditioned reinforcement learninggoal-conditioned reinforcement learning, n.: Training one AI agent to reach many different goals — whichever target it is given — instead of a single fixed task, so a robot can, say, place an object anywhere asked., Open X-EmbodimentOpen X-Embodiment, prop. n.: A large shared collection of robot training data, pooled by dozens of research labs in 2023 and covering many kinds of robots, so that one AI model could learn from all of them., Physical IntelligencePhysical Intelligence, prop. n.: A San Francisco company, founded in 2024, that builds general-purpose AI 'brains' meant to control many different kinds of robots doing many different tasks., π0π0, prop. n.: A general-purpose AI model for controlling robots, released in 2024 by the start-up Physical Intelligence, that sees through cameras, follows typed or spoken instructions and moves robot arms deftly enough to fold laundry., proprioceptionproprioception, n.: A robot's sense of its own body: where its joints are, how fast they are moving and how hard its motors are pushing, measured by internal sensors rather than cameras.