diffusion policy
In plain wordsA way of training robots in which the same kind of AI that generates images instead generates the robot's next movements, letting it copy demonstrations even when there are several good ways to do a task.
A policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. that generates robot actions with a conditional diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.: starting from random noise, it iteratively denoises a sequence of future actions, conditioned on recent camera images and robot state, using a network (a convolutional U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost. or a transformer) trained to predict the noise to remove at each step. Because diffusion models represent multimodal distributions — several distinct valid ways to complete a task — they avoid the averaging failure of regression-based behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation., in which a policy torn between passing left or right of an obstacle drives straight into it. Predicting whole action sequencesaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer. and executing only part before replanning (receding-horizon control) yields smooth, consistent motion. In its original evaluation, on 12 tasks drawn from four benchmarks, it beat earlier methods by 46.9% in average success, and diffusion and flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples. action heads became standard parts of later vision-language-action modelsvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements..
Switching from a regression head to a diffusion policy stopped the arm dithering at the fork.
Origin
Introduced by Cheng Chi, Siyuan Feng, Shuran Song, Russ Tedrake and colleagues (Columbia University, Toyota Research Institute and MIT) in 'Diffusion Policy: Visuomotor Policy Learning via Action Diffusion', posted to arXiv in March 2023 and presented at Robotics: Science and Systems (RSS) 2023; an extended version appeared in the International Journal of Robotics Research in 2024.
Usage note
Sampling takes several denoising steps, which can limit control rates; distillation, consistency modelsconsistency model, n.: A generator trained to jump from noise straight to a finished image in one step, instead of the dozens of small steps a diffusion model normally takes. and flow matching are used to cut latency. Capitalized, 'Diffusion Policy' names the 2023 method; lowercase, the general approach.
Related
- diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
- action chunkingaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer.
- behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation.
- imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
- flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples.
- vision-language-action modelvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements.
Sources
- Chi, C. et al. (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. Robotics: Science and Systems. arXiv:2303.04137. arxiv.org/abs/2303.04137
Mentioned in
behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation., flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples., imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error., manipulationmanipulation, n.: A robot's use of arms, hands or grippers to change the world around it: picking things up, moving, pushing, opening, assembling or using tools., robot learningrobot learning, n.: Teaching robots skills from data and experience, such as watching demonstrations, practicing by trial and error or training in simulation, instead of programming every movement by hand., vision-language-action modelvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements., action chunkingaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer.