sim-to-real transfer
In plain wordsTraining a robot's AI in a computer simulation, where practice is fast, cheap and safe, and then making it work on the real robot despite the differences between simulation and reality.
The process of training a policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. or perception model in a physics simulatorphysics simulator, n.: Software that imitates how real objects move, collide, bend and fall, so robots and their AI can be designed, trained and tested on a computer before touching the real world. and deploying it on a physical robot, together with the techniques that make this work despite the reality gap: mismatches in dynamics (friction, mass, motor response, latency, contact), sensing (rendering, noise, calibration) and environment. Simulation offers millions of safe, parallel, automatically reset trials, which reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. needs and real hardware cannot supply. Bridging techniques include domain randomizationdomain randomization, n.: Training a robot's AI in a simulation whose colors, lighting, weights, friction and other details are shuffled at random on every run, so that the real world holds few surprises it has not already practiced for. of physical and visual parameters; system identification, which fits simulator parameters to measurements from the real machine; actuator models learned from real data; added noise and delays; teacher–student training, in which a policy with access to simulator ground truth teaches one that sees only realistic sensors; and fine-tuning on limited real data. Real-to-sim methods build simulations from real scans and video (digital twinsdigital twin, n.: A detailed virtual copy of a real machine, factory, robot or city, kept in sync with live sensor data and used to monitor it, test changes safely and predict problems., Gaussian splattingGaussian splatting, n.: A way of turning photos of a real place into a 3D scene made of millions of tiny, soft, colored blobs that can be viewed from any angle in real time, even on a phone or headset.). Since about 2019 sim-to-real reinforcement learning has been the standard way to train legged locomotionlocomotion, n.: How a robot moves itself from place to place: rolling on wheels, walking on legs, crawling, swimming or flying.; contact-rich manipulation remains harder to simulate faithfully.
Sim-to-real failed until we randomized motor latency; the real servos lag by eight milliseconds.
Origin
Nick Jakobi, Phil Husbands and Inman Harvey named the 'reality gap' in evolutionary robotics in 1995. Deep-learning work of 2016–2019 — Fereshteh Sadeghi and Sergey Levine's CAD2RL, Josh Tobin and colleagues' domain randomization, Xue Bin Peng and colleagues' dynamics randomization, OpenAI's Dactyl hand and ETH Zurich's ANYmal quadruped — made 'sim-to-real', often written 'sim2real', a standard term.
Usage note
A special case of transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data. and of coping with distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.; distinct from cross-embodiment learningcross-embodiment learning, n.: Training one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another., which transfers skills between different robots.
Related
- domain randomizationdomain randomization, n.: Training a robot's AI in a simulation whose colors, lighting, weights, friction and other details are shuffled at random on every run, so that the real world holds few surprises it has not already practiced for.
- physics simulatorphysics simulator, n.: Software that imitates how real objects move, collide, bend and fall, so robots and their AI can be designed, trained and tested on a computer before touching the real world.
- digital twindigital twin, n.: A detailed virtual copy of a real machine, factory, robot or city, kept in sync with live sensor data and used to monitor it, test changes safely and predict problems.
- reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time.
- locomotionlocomotion, n.: How a robot moves itself from place to place: rolling on wheels, walking on legs, crawling, swimming or flying.
- distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
- MuJoCoMuJoCo, prop. n.: A free physics simulator, now maintained by Google DeepMind, that AI researchers use to train and test simulated robots and bodies, especially with trial-and-error learning.
- cross-embodiment learningcross-embodiment learning, n.: Training one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another.
Sources
- Jakobi, N., Husbands, P. & Harvey, I. (1995). Noise and the reality gap: The use of simulation in evolutionary robotics. Proc. European Conference on Artificial Life (ECAL 1995).
- Tobin, J. et al. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. arXiv:1703.06907. arxiv.org/abs/1703.06907
Mentioned in
physical AIphysical AI, n.: AI built to work in the physical world rather than on a screen: the intelligence inside robots, self-driving cars, drones and other machines that must sense real surroundings and move through them., synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars., transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data., digital twindigital twin, n.: A detailed virtual copy of a real machine, factory, robot or city, kept in sync with live sensor data and used to monitor it, test changes safely and predict problems., domain adaptationdomain adaptation, n.: Adjusting a model trained on one kind of data, such as daytime photos or one hospital's records, so that it still works on a different but related kind, such as night photos or another hospital's., legged robotlegged robot, n.: A robot that walks on legs instead of rolling on wheels, such as a four-legged 'robot dog' or a two-legged humanoid, so it can climb stairs and cross rough ground., locomotionlocomotion, n.: How a robot moves itself from place to place: rolling on wheels, walking on legs, crawling, swimming or flying., physics simulatorphysics simulator, n.: Software that imitates how real objects move, collide, bend and fall, so robots and their AI can be designed, trained and tested on a computer before touching the real world., robot learningrobot learning, n.: Teaching robots skills from data and experience, such as watching demonstrations, practicing by trial and error or training in simulation, instead of programming every movement by hand., CoRLCoRL, prop. n.: A yearly academic conference, started in 2017, for research where robotics meets machine learning., cross-embodiment learningcross-embodiment learning, n.: Training one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another., dexterous manipulationdexterous manipulation, n.: Skilled handling of objects with robot fingers, such as spinning a pen, turning a key or rotating a cube in one hand: the fine finger work people do without thinking., domain randomizationdomain randomization, n.: Training a robot's AI in a simulation whose colors, lighting, weights, friction and other details are shuffled at random on every run, so that the real world holds few surprises it has not already practiced for., impedance controlimpedance control, n.: A way of controlling a robot so that it behaves like a spring and shock absorber when it touches things, giving way gently instead of pushing rigidly., MuJoCoMuJoCo, prop. n.: A free physics simulator, now maintained by Google DeepMind, that AI researchers use to train and test simulated robots and bodies, especially with trial-and-error learning., Pieter AbbeelPieter Abbeel, prop. n.: A Belgian-American roboticist at Berkeley whose research taught robots to learn from demonstrations and from trial and error, and who co-founded the robotics company Covariant., Sergey LevineSergey Levine, prop. n.: A Berkeley computer scientist, one of the most prolific researchers on how robots can learn skills from data and experience, and a co-founder of the robotics company Physical Intelligence., tactile sensingtactile sensing, n.: Giving robots a sense of touch, with sensors in fingertips or skin that feel pressure, texture, vibration and slipping, so they can handle objects gently and securely.