Center for AI Safety
In plain wordsA San Francisco nonprofit that studies how advanced AI could lead to disaster and how to prevent it. In 2023 it gathered signatures for a one-sentence warning that AI could threaten human extinction.
A nonprofit devoted to research and advocacy aimed at preventing large-scale catastrophes from AI. It organized the May 2023 Statement on AI RiskStatement on AI Risk, prop. n.: A one-sentence public statement from May 2023, signed by hundreds of leading AI scientists and company heads, declaring that guarding against the chance of AI wiping out humanity should rank as a priority for the whole world., a single sentence placing the risk of extinction from AI alongside pandemics and nuclear war, signed by hundreds of researchers and executives. Its researchers created widely used benchmarks and safety datasets, including Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%. (with Scale AIScale AI, prop. n.: A company that pays large numbers of people to label examples, write model answers and grade chatbot replies, producing the data AI developers train and test on. Meta bought nearly half of it in 2025., 2025) and the WMDPWMDP, prop. n.: A multiple-choice test of AI knowledge that could help someone make biological, chemical or cyber weapons, built from questions near the dangerous line but not over it, and used to test ways of erasing such knowledge. benchmark of hazardous knowledge (2024), and work on representation engineeringrepresentation engineering, n.: An approach to understanding and controlling AI models by finding where high-level ideas such as honesty or danger show up in their internal activity, then reading or adjusting those signals directly., machine unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again. and robustness; its director had earlier led the creation of MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge.. It provides a compute cluster to outside safety researchers, publishes a textbook and course on AI safety, and through its affiliated advocacy group, the CAIS Action Fund, co-sponsored California's SB 1047SB 1047, prop. n.: A 2024 California bill that would have required makers of the biggest AI models to adopt safety plans and take care to prevent catastrophes. It passed the legislature, but the governor vetoed it..
Origin
Founded in 2022 in San Francisco by Dan Hendrycks, its director, and Oliver Zhang. Hendrycks has also served as a safety adviser to xAI and Scale AI.
Usage note
Not to be confused with government AI safety institutesAI Safety Institute, n.: A government agency that tests the most advanced AI models for dangerous abilities and studies how to make them safe. Britain and the United States set up the first ones in 2023., such as the UK's, or with the US Center for AI Standards and InnovationCenter for AI Standards and Innovation, prop. n.: The US government's AI testing and standards center, part of the National Institute of Standards and Technology. It evaluates advanced AI models, American and foreign, for national-security risks and works with companies on voluntary testing..
Related
- Statement on AI RiskStatement on AI Risk, prop. n.: A one-sentence public statement from May 2023, signed by hundreds of leading AI scientists and company heads, declaring that guarding against the chance of AI wiping out humanity should rank as a priority for the whole world.
- Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%.
- MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge.
- SB 1047SB 1047, prop. n.: A 2024 California bill that would have required makers of the biggest AI models to adopt safety plans and take care to prevent catastrophes. It passed the legislature, but the governor vetoed it.
- representation engineeringrepresentation engineering, n.: An approach to understanding and controlling AI models by finding where high-level ideas such as honesty or danger show up in their internal activity, then reading or adjusting those signals directly.
- existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
- AI Safety InstituteAI Safety Institute, n.: A government agency that tests the most advanced AI models for dangerous abilities and studies how to make them safe. Britain and the United States set up the first ones in 2023.
Mentioned in
Dan HendrycksDan Hendrycks, prop. n.: An American AI-safety researcher who created widely used AI tests such as MMLU and Humanity's Last Exam, and who runs the Center for AI Safety., Scale AIScale AI, prop. n.: A company that pays large numbers of people to label examples, write model answers and grade chatbot replies, producing the data AI developers train and test on. Meta bought nearly half of it in 2025., Statement on AI RiskStatement on AI Risk, prop. n.: A one-sentence public statement from May 2023, signed by hundreds of leading AI scientists and company heads, declaring that guarding against the chance of AI wiping out humanity should rank as a priority for the whole world., Alexandr WangAlexandr Wang, prop. n.: An American entrepreneur who founded Scale AI, a major supplier of human-labeled data for training AI, and became Meta's chief AI officer in 2025., Mutual Assured AI MalfunctionMutual Assured AI Malfunction, n.: A proposed deterrence strategy in which any country racing for overwhelming AI dominance would expect rivals to sabotage its AI projects, so none tries, much as with nuclear deterrence., Virology Capabilities TestVirology Capabilities Test, prop. n.: A hard test of practical laboratory virology know-how, used to check whether AI models know enough about working with viruses that they might help someone misuse biology.