exposure bias
In plain wordsA weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball.
The mismatch between how an autoregressive sequence model is trained and how it is used: under teacher forcingteacher forcing, n.: Training a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one. it always conditions on ground-truth prefixes, but during generation it conditions on its own outputs, so an early error moves it into states absent from training data, where further errors may compound. The same compounding-error problem appears in imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error., where behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation. errors can grow quadratically with the horizon. Proposed remedies include scheduled sampling, which gradually mixes the model's own predictions into training inputs; sequence-level objectives optimized with reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time.; professor forcing; and, for large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., on-policy methods such as RLHFreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. and on-policy distillationon-policy distillation, n.: A way of teaching a small AI from a bigger one where the small AI writes its own answers and the bigger one grades every word, so the learner is corrected on its own mistakes. that train on the model's own samples. How much exposure bias harms modern large models is debated: some studies find they recover well from their own mistakes, and other causes, such as likelihood training favoring bland text, also explain degenerate output.
Origin
Named by Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli and Wojciech Zaremba of Facebook AI Research in 'Sequence Level Training with Recurrent Neural Networks' (arXiv November 2015; ICLR 2016), who wrote that they 'refer to this discrepancy as exposure bias'. Samy Bengio and colleagues' scheduled sampling (June 2015) had tackled the same problem without the name.
Related
- teacher forcingteacher forcing, n.: Training a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one.
- imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
- on-policy distillationon-policy distillation, n.: A way of teaching a small AI from a bigger one where the small AI writes its own answers and the bigger one grades every word, so the learner is corrected on its own mistakes.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back.
- decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.
Sources
- Ranzato, M., Chopra, S., Auli, M. & Zaremba, W. (2016). Sequence Level Training with Recurrent Neural Networks. ICLR 2016. arxiv.org/abs/1511.06732
- Bengio, S., Vinyals, O., Jaitly, N. & Shazeer, N. (2015). Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. NeurIPS 2015. arxiv.org/abs/1506.03099
Mentioned in
autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back., teacher forcingteacher forcing, n.: Training a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one.