iterated amplification
In plain wordsA proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats.
A proposal for training AI systems beyond human ability while keeping them aligned, by alternating two steps. In amplification, a human (or another trusted process) answers a question by breaking it into subquestions delegated to copies of the current model, producing better answers than the model alone; in distillation, a new model is trained, by imitation or reinforcement learning, to reproduce the amplified system's answers quickly. Each round is meant to raise capability while every step stays supervised by a process the human endorses — much as AlphaGo ZeroAlphaGo, prop. n.: A DeepMind program that in 2016 beat Lee Sedol, one of the world's best players of the board game Go, a decade or more before many experts expected a computer to manage it. alternated tree search (amplification) with network training (distillation) in expert iterationexpert iteration, n.: A learning loop in which a slow, careful search finds better moves or solutions than the AI's quick instincts, and the AI is then trained to imitate them, so each round's search starts smarter.. The idealized limit of the scheme is HCHHCH, abbr.: A thought experiment: a person who can hand parts of a problem to copies of themselves, who can hand parts on again, endlessly — the ideal some AI training schemes try to approximate.. It rests on factored cognitionfactored cognition, n.: The idea that a hard thinking task can be broken into many small, separately checkable pieces, each simple enough for a person or an AI to handle in a short time., the hypothesis that hard thinking can be split into small, separately checkable pieces; whether that holds for tasks needing long-built intuition is disputed.
Each round of iterated amplification distills the human-plus-assistants team into a faster model.
Origin
Developed by Paul Christiano in blog posts from 2016 (among them the ALBA scheme and 'Capability amplification', October 2016) and formalized with Buck Shlegeris and Dario Amodei in 'Supervising strong learners by amplifying weak experts' (2018), which tested it on small algorithmic tasks. The longer name 'iterated distillation and amplification' spread through Ajeya Cotra's 2018 explainer of that title.
Usage note
A research agenda rather than a deployed technique. Its ideas live on in task-decomposition methods for supervising language models, such as recursive book summarization, and in AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend., which Christiano and colleagues regard as a close relative.
Related
- HCHHCH, abbr.: A thought experiment: a person who can hand parts of a problem to copies of themselves, who can hand parts on again, endlessly — the ideal some AI training schemes try to approximate.
- factored cognitionfactored cognition, n.: The idea that a hard thinking task can be broken into many small, separately checkable pieces, each simple enough for a person or an AI to handle in a short time.
- scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
- AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend.
- recursive reward modelingrecursive reward modeling, n.: A plan for supervising advanced AI in stages: AI assistants trained on human feedback help people judge the work of a more capable AI, which then helps judge the next one.
- expert iterationexpert iteration, n.: A learning loop in which a slow, careful search finds better moves or solutions than the AI's quick instincts, and the AI is then trained to imitate them, so each round's search starts smarter.
- Paul ChristianoPaul Christiano, prop. n.: An American AI-alignment researcher who led the work that made training AI from human preferences practical, the basis of how chatbots are taught to be helpful.
Sources
- Christiano, P., Shlegeris, B. & Amodei, D. (2018). Supervising strong learners by amplifying weak experts. arXiv:1810.08575.
Mentioned in
outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want., Paul ChristianoPaul Christiano, prop. n.: An American AI-alignment researcher who led the work that made training AI from human preferences practical, the basis of how chatbots are taught to be helpful., scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., task decompositiontask decomposition, n.: Breaking a big, vague job into smaller, concrete steps that can each be done and checked — the way a recipe splits a meal into stages., AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend., Alignment Research CenterAlignment Research Center, prop. n.: A small Berkeley nonprofit started by Paul Christiano that searches for mathematical ways to keep future AI systems, even ones far smarter than people, honest and under human control., Buck ShlegerisBuck Shlegeris, prop. n.: An AI-safety researcher who runs Redwood Research and helped found 'AI control', the study of keeping powerful AI safe even if it is secretly trying to cause harm., factored cognitionfactored cognition, n.: The idea that a hard thinking task can be broken into many small, separately checkable pieces, each simple enough for a person or an AI to handle in a short time., HCHHCH, abbr.: A thought experiment: a person who can hand parts of a problem to copies of themselves, who can hand parts on again, endlessly — the ideal some AI training schemes try to approximate., prosaic alignmentprosaic alignment, n.: Working on AI safety by assuming that powerful future AI will be built from today's techniques — scaled-up neural networks — rather than from some new breakthrough., recursive reward modelingrecursive reward modeling, n.: A plan for supervising advanced AI in stages: AI assistants trained on human feedback help people judge the work of a more capable AI, which then helps judge the next one.