DAgger
In plain wordsA way of teaching a robot by imitation in which an expert keeps labeling what it should have done in the situations it actually gets into, so it learns to recover from its own mistakes.
An imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error. algorithm (Dataset Aggregation) that addresses the compounding-error problem of behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation.. A policy trained only on expert demonstrations sees the states the expert visited; once its own small errors carry it elsewhere, it meets unfamiliar situations, errs further, and the mismatch, a form of covariate shiftcovariate shift, n.: A kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same., grows over the episode. DAgger iterates: run the current learned policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. (or a mixture with the expert), record the states it actually visits, ask the expert which action it would take in each, add these labeled states to the dataset, and retrain on the aggregate. Under its assumptions, error grows linearly with the task horizon instead of quadratically as in plain behavioral cloning, with guarantees derived from no-regret online learning. Its practical cost is an expert who can label on demand, which is awkward for human teleoperators; variants such as human-gated DAgger let the expert take over only when needed, and the same idea underlies on-policy distillationon-policy distillation, n.: A way of teaching a small AI from a bigger one where the small AI writes its own answers and the bigger one grades every word, so the learner is corrected on its own mistakes. from teacher models.
After two DAgger rounds the drone stopped drifting into trees that never appeared in the expert flights.
Origin
Introduced by Stéphane Ross, Geoffrey Gordon and J. Andrew Bagnell of Carnegie Mellon University in 'A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning' (AISTATS 2011), following Ross and Bagnell's 2010 analysis of compounding errors; a 2013 follow-up used it to fly a small quadrotor through a forest.
Related
- imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
- behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation.
- covariate shiftcovariate shift, n.: A kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same.
- on-policy distillationon-policy distillation, n.: A way of teaching a small AI from a bigger one where the small AI writes its own answers and the bigger one grades every word, so the learner is corrected on its own mistakes.
- teleoperationteleoperation, n.: Controlling a robot or machine from a distance, with a person steering it in real time using a joystick, a matching 'puppet' arm, a VR headset or a screen.
- robot learningrobot learning, n.: Teaching robots skills from data and experience, such as watching demonstrations, practicing by trial and error or training in simulation, instead of programming every movement by hand.
Sources
- Ross, S., Gordon, G. & Bagnell, J. A. (2011). A reduction of imitation learning and structured prediction to no-regret online learning. Proc. AISTATS 2011, PMLR 15, 627–635.
Mentioned in
behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation., imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.