Alignment Research Center
In plain wordsA small Berkeley nonprofit started by Paul Christiano that searches for mathematical ways to keep future AI systems, even ones far smarter than people, honest and under human control.
A Berkeley nonprofit, founded by Paul Christiano in 2021, that works on alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this. theory: methods for keeping AI systems trustworthy that should go on working even when the systems far exceed human ability. Its first major report framed the problem of eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe. (2021): how to train a model to report what it internally 'knows' about the world rather than what human overseers would accept. Its later research has sought formal 'heuristic explanations' of neural-network behavior that could flag rare catastrophic failures and anomalous internal mechanisms. From 2022 it housed ARC Evals, which tested GPT-4 and Claude before release — including the much-reported test in which GPT-4, with researchers relaying its messages, got a TaskRabbit worker to solve a CAPTCHA by claiming to have a visual impairment — and which became the independent METRMETR, prop. n.: A nonprofit research group that checks new AI systems for dangerous skills, best known for tracking how long the tasks are that AI agents can now finish on their own. in December 2023.
Origin
Founded in April 2021 in Berkeley, California, by Paul Christiano after he left OpenAI's alignment team. Christiano left in 2024 to lead AI safety at the US AI Safety Institute, later renamed the Center for AI Standards and InnovationCenter for AI Standards and Innovation, prop. n.: The US government's AI testing and standards center, part of the National Institute of Standards and Technology. It evaluates advanced AI models, American and foreign, for national-security risks and works with companies on voluntary testing..
Usage note
Not to be confused with ARC-AGIARC-AGI, prop. n.: Puzzles made of colored grids — and, since 2026, small video-game-like worlds — that people solve easily but AI long found hard, designed to test whether a machine can learn a new skill from just a few examples., a benchmark run by the separate ARC Prize Foundation.
Related
- Paul ChristianoPaul Christiano, prop. n.: An American AI-alignment researcher who led the work that made training AI from human preferences practical, the basis of how chatbots are taught to be helpful.
- eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.
- METRMETR, prop. n.: A nonprofit research group that checks new AI systems for dangerous skills, best known for tracking how long the tasks are that AI agents can now finish on their own.
- iterated amplificationiterated amplification, n.: A proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats.
- scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- ARC-AGIARC-AGI, prop. n.: Puzzles made of colored grids — and, since 2026, small video-game-like worlds — that people solve easily but AI long found hard, designed to test whether a machine can learn a new skill from just a few examples.
Sources
- Christiano, P., Cotra, A. & Xu, M. (2021). Eliciting Latent Knowledge: How to Tell if Your Eyes Deceive You. Alignment Research Center technical report.
Mentioned in
METRMETR, prop. n.: A nonprofit research group that checks new AI systems for dangerous skills, best known for tracking how long the tasks are that AI agents can now finish on their own., Paul ChristianoPaul Christiano, prop. n.: An American AI-alignment researcher who led the work that made training AI from human preferences practical, the basis of how chatbots are taught to be helpful., eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe., mechanistic anomaly detectionmechanistic anomaly detection, n.: Flagging cases where an AI reaches its output by an unusual internal route — for instance because it is being tricked, or is tricking us — even when the output itself looks normal., Open PhilanthropyOpen Philanthropy, prop. n.: A large American charitable foundation, renamed Coefficient Giving in 2025, that has been the biggest private funder of research on the risks of advanced AI.