superalignment
In plain wordsThe problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it.
- 1
Research problem
The problem, as OpenAI framed it in 2023, of aligning AI systems substantially more capable than humans: techniques such as reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. depend on people judging a model's outputs, and are expected to fail once people can no longer tell good behavior from bad. Proposed approaches include scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., weak-to-strong generalizationweak-to-strong generalization, n.: The finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI. (studying whether weak supervisors can elicit the full capabilities of strong models), automated interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box. and automated alignment researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors. — using aligned, roughly human-level AI systems to do alignment research on their successors.
RLHF works while humans can grade the answers; superalignment asks what happens when they can't.
- 2
OpenAI team
A team at OpenAIOpenAI, prop. n.: The American company that makes ChatGPT and the GPT models. It began in 2015 as a nonprofit research lab and is now a for-profit company overseen by a nonprofit foundation., announced in July 2023 under the joint leadership of Ilya Sutskever and Jan Leike, whose stated aim was to crack the core technical problems of aligning superintelligence within four years, backed by a pledge of 20% of the compute the company had secured so far. The centerpiece of its plan was an automated alignment researcher of about human level. It published work on weak-to-strong generalization and funded outside research before being dissolved in May 2024, after both leaders left the company; Leike wrote that safety culture and processes had 'taken a backseat to shiny products'.
Origin
Introduced as the name of a new team in OpenAI's announcement 'Introducing Superalignment' by Jan Leike and Ilya Sutskever (5 July 2023), blending 'superintelligence' and 'alignment'; no earlier use is established.
Usage note
The word outlived the team as a name for the general problem. Sutskever co-founded Safe Superintelligence Inc.Safe Superintelligence Inc., prop. n.: A company started in 2024 by Ilya Sutskever, a co-founder of OpenAI, with the single goal of building a superintelligent AI that is safe. It sells no products. in June 2024, and Leike joined Anthropic.
Related
- scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
- weak-to-strong generalizationweak-to-strong generalization, n.: The finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI.
- superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- automated AI researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors.
- OpenAIOpenAI, prop. n.: The American company that makes ChatGPT and the GPT models. It began in 2015 as a nonprofit research lab and is now a for-profit company overseen by a nonprofit foundation.
- Safe Superintelligence Inc.Safe Superintelligence Inc., prop. n.: A company started in 2024 by Ilya Sutskever, a co-founder of OpenAI, with the single goal of building a superintelligent AI that is safe. It sells no products.
Sources
- Leike, J. & Sutskever, I. (2023). Introducing Superalignment. OpenAI. openai.com/index/introducing-superalignment
- Burns, C. et al. (2023). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. arXiv:2312.09390. arxiv.org/abs/2312.09390
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different., superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist., artificial superintelligenceartificial superintelligence, n.: Superintelligence in machine form: AI that would outthink the best humans at nearly every intellectual task. Often shortened to ASI and pictured as the stage after human-level AI., automated AI researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors., Ilya SutskeverIlya Sutskever, prop. n.: A Russian-born Israeli-Canadian computer scientist who co-built the AlexNet image recognizer, helped found OpenAI and lead its research until 2024, and now heads the startup Safe Superintelligence Inc., OpenAI leadership crisis (2023)OpenAI leadership crisis (2023), n.: The five days in November 2023 when OpenAI's board fired chief executive Sam Altman, nearly all staff threatened to quit, and he returned with a new board., scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., situational awarenesssituational awareness, n.: An AI model's grasp of its own position — that it is an AI, how it was trained, whether it is being tested. Also the title of a widely read 2024 essay on the race toward human-level AI., Jan LeikeJan Leike, prop. n.: A German AI-alignment researcher who helped develop training from human feedback, co-led OpenAI's superalignment team, and now leads alignment research at Anthropic., Leopold AschenbrennerLeopold Aschenbrenner, prop. n.: A German-born former OpenAI researcher whose 2024 essay series 'Situational Awareness' predicted human-level AI around 2027 and a government-led race to build superintelligence., self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight., Situational Awareness (essay)Situational Awareness (essay), prop. n.: A long 2024 essay by a former OpenAI researcher arguing that AGI could arrive around 2027, that superintelligence could follow soon after, and that the US government should run a national AGI project., weak-to-strong generalizationweak-to-strong generalization, n.: The finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI.