behavioral cloning
In plain wordsThe simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation.
An imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error. method that treats expert demonstrations as a supervised learningsupervised learning, n.: Teaching a computer by example: you give it many inputs paired with the right answers, such as photos labeled 'cat' or 'dog', and it learns to answer correctly for new inputs. problem, training a policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. to map each observed statestate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next. to the action the expert took, typically by maximizing the likelihood of the expert's actions. It needs no reward function, no environment interaction and no expert queries during training, so it is simple and scales to large datasets — which is why it underlies most modern robot policies and, in effect, supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples. of language models. Its known weakness is compounding error: small mistakes take the learner into states the expert never visited, where errors grow, so performance can degrade quadratically with the task horizon. Remedies include collecting recovery data or querying the expert on the learner's own states (DAggerDAgger, n.: A way of teaching a robot by imitation in which an expert keeps labeling what it should have done in the situations it actually gets into, so it learns to recover from its own mistakes.), injecting noise while demonstrating, and expressive policy classes such as diffusion policiesdiffusion policy, n.: A way of training robots in which the same kind of AI that generates images instead generates the robot's next movements, letting it copy demonstrations even when there are several good ways to do a task. and action chunkingaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer. that can represent several valid ways of acting.
Behavioral cloning worked on the training tracks but drifted off the road once the car left familiar curves.
Origin
The term is usually credited to Donald Michie and colleagues, who around 1990 used it for reconstructing a human operator's skill from logged control traces, notably in Claude Sammut, Scott Hurst, Dana Kedzier and Michie's 'Learning to Fly' (ICML 1992), which cloned pilots in a flight simulator. Dean Pomerleau's ALVINN road-follower had already learned to steer from a human driver's demonstrations in 1989, after a first version trained on simulated images.
Related
- imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
- DAggerDAgger, n.: A way of teaching a robot by imitation in which an expert keeps labeling what it should have done in the situations it actually gets into, so it learns to recover from its own mistakes.
- distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
- diffusion policydiffusion policy, n.: A way of training robots in which the same kind of AI that generates images instead generates the robot's next movements, letting it copy demonstrations even when there are several good ways to do a task.
- action chunkingaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer.
- supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.
- inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims.
Sources
- Sammut, C., Hurst, S., Kedzier, D. & Michie, D. (1992). Learning to Fly. Proceedings of ICML 1992.
- Michie, D., Bain, M. & Hayes-Michie, J. (1990). Cognitive models from subcognitive skills. In Knowledge-Based Systems for Industrial Control. Peter Peregrinus/IEE.
Mentioned in
imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error., policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards., robot learningrobot learning, n.: Teaching robots skills from data and experience, such as watching demonstrations, practicing by trial and error or training in simulation, instead of programming every movement by hand., supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples., action chunkingaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer., DAggerDAgger, n.: A way of teaching a robot by imitation in which an expert keeps labeling what it should have done in the situations it actually gets into, so it learns to recover from its own mistakes., decision transformerdecision transformer, n.: A way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action., diffusion policydiffusion policy, n.: A way of training robots in which the same kind of AI that generates images instead generates the robot's next movements, letting it copy demonstrations even when there are several good ways to do a task., Donald MichieDonald Michie, prop. n.: A British codebreaker turned AI pioneer who built a game-playing learning machine out of matchboxes and founded Edinburgh's influential machine-intelligence research group., exposure biasexposure bias, n.: A weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball., inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims., offline reinforcement learningoffline reinforcement learning, n.: Learning a strategy entirely from a fixed collection of recorded experience — logs, past robot runs, medical records — without trying anything new in the real world while learning., WebGPTWebGPT, prop. n.: A 2021 OpenAI research system that answered questions by searching and browsing the web in a simple text browser, collecting sources and citing them — an early ancestor of today's research agents.