scalable oversight
In plain wordsThe problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
The problem, and the family of methods addressing it, of supplying accurate training signals and evaluations for AI systems on tasks where unaided human judgment is too costly, slow or error-prone — ultimately for systems more capable than their overseers. Most proposals use AI assistance to make evaluating easier than doing: AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend., iterated amplificationiterated amplification, n.: A proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats., recursive reward modelingrecursive reward modeling, n.: A plan for supervising advanced AI in stages: AI assistants trained on human feedback help people judge the work of a more capable AI, which then helps judge the next one., prover–verifier gamesprover–verifier game, n.: A training setup in which one AI must justify its answers so clearly that a weaker AI can check them, which also makes the strong AI's work easier for people to check., model-written critiques and task decomposition. A related line, weak-to-strong generalizationweak-to-strong generalization, n.: The finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI., asks how far a strong model generalizes beyond a weaker supervisor's labels. Progress is measured with sandwichingsandwiching, n.: A way of testing oversight methods today: non-experts, helped by an AI, try to get expert-quality answers out of it, and real experts then check whether they succeeded. experiments, in which non-experts aided by AI try to match expert judgments. The problem is pressing for reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., whose raters may miss subtle errors in long code or arguments and so reward answers that merely look right.
Without better scalable oversight, our raters will keep preferring confident answers over correct ones.
Origin
Introduced as a named research problem in 'Concrete Problems in AI Safety' (2016) by Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman and Dan Mané, one of five problems listed there. Samuel Bowman and colleagues' 'Measuring Progress on Scalable Oversight for Large Language Models' (2022) helped turn it into an experimental field.
Usage note
Sometimes used loosely for any human supervision of AI. In the technical sense it means oversight that keeps working as the overseen system's capabilities outgrow the overseer's; legal human oversighthuman oversight, n.: Keeping people able to watch, understand, question and, if necessary, stop or overrule an AI system, rather than letting it act unchecked. requirements do not presuppose that gap.
Related
- AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend.
- iterated amplificationiterated amplification, n.: A proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats.
- recursive reward modelingrecursive reward modeling, n.: A plan for supervising advanced AI in stages: AI assistants trained on human feedback help people judge the work of a more capable AI, which then helps judge the next one.
- weak-to-strong generalizationweak-to-strong generalization, n.: The finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI.
- sandwichingsandwiching, n.: A way of testing oversight methods today: non-experts, helped by an AI, try to get expert-quality answers out of it, and real experts then check whether they succeeded.
- eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.
- superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it.
- human oversighthuman oversight, n.: Keeping people able to watch, understand, question and, if necessary, stop or overrule an AI system, rather than letting it act unchecked.
Sources
- Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J. & Mané, D. (2016). Concrete Problems in AI Safety. arXiv:1606.06565.
- Bowman, S. R. et al. (2022). Measuring Progress on Scalable Oversight for Large Language Models. arXiv preprint.
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., game theorygame theory, n.: The mathematics of strategic decisions, where each player's best move depends on what the others do. It is used to study competition and cooperation among people, firms, nations and AI agents., GPQAGPQA, prop. n.: A set of very hard multiple-choice science questions written by PhD-level experts, designed so that looking things up online does not help much: capable non-specialists with web access still mostly fail., human oversighthuman oversight, n.: Keeping people able to watch, understand, question and, if necessary, stop or overrule an AI system, rather than letting it act unchecked., outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want., Paul ChristianoPaul Christiano, prop. n.: An American AI-alignment researcher who led the work that made training AI from human preferences practical, the basis of how chatbots are taught to be helpful., reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging., superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it., task decompositiontask decomposition, n.: Breaking a big, vague job into smaller, concrete steps that can each be done and checked — the way a recipe splits a meal into stages., AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend., Ajeya CotraAjeya Cotra, prop. n.: An American AI researcher and forecaster whose 'biological anchors' report estimated when transformative AI might arrive, and who argued that standard training could produce deceptive AI., Alignment Research CenterAlignment Research Center, prop. n.: A small Berkeley nonprofit started by Paul Christiano that searches for mathematical ways to keep future AI systems, even ones far smarter than people, honest and under human control., eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe., factored cognitionfactored cognition, n.: The idea that a hard thinking task can be broken into many small, separately checkable pieces, each simple enough for a person or an AI to handle in a short time., HCHHCH, abbr.: A thought experiment: a person who can hand parts of a problem to copies of themselves, who can hand parts on again, endlessly — the ideal some AI training schemes try to approximate., intent alignmentintent alignment, n.: An AI is intent-aligned if it is genuinely trying to do what its operator wants — even if it sometimes fails through ignorance or clumsiness., iterated amplificationiterated amplification, n.: A proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats., Jan LeikeJan Leike, prop. n.: A German AI-alignment researcher who helped develop training from human feedback, co-led OpenAI's superalignment team, and now leads alignment research at Anthropic., process supervisionprocess supervision, n.: Training an AI by grading each step of its reasoning or work, not just whether the final answer came out right, so that it learns sound methods instead of lucky shortcuts., prosaic alignmentprosaic alignment, n.: Working on AI safety by assuming that powerful future AI will be built from today's techniques — scaled-up neural networks — rather than from some new breakthrough., prover–verifier gameprover–verifier game, n.: A training setup in which one AI must justify its answers so clearly that a weaker AI can check them, which also makes the strong AI's work easier for people to check., recursive reward modelingrecursive reward modeling, n.: A plan for supervising advanced AI in stages: AI assistants trained on human feedback help people judge the work of a more capable AI, which then helps judge the next one., sandwichingsandwiching, n.: A way of testing oversight methods today: non-experts, helped by an AI, try to get expert-quality answers out of it, and real experts then check whether they succeeded.