Skip to content
Last Dictionary
Plane
Safety & AlignmentGovernance, Policy & LawSociety, Ethics & Economy

AI safety

nounAlso: artificial intelligence safety, AI safety research, ML safety, safe AI

In plain wordsThe effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable.

  1. 1

    Research field

    Research and engineering that aims to keep AI systems from causing serious harm — by malfunction, by misuse or through wider structural effects — and that gives special weight to dangers expected to rise as systems become more capable. Technical work centers on alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this. (giving systems the intended goals), robustnessrobustness, n.: An AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it. (reliable behavior under distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested. and adversarial inputsadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon.), monitoring and interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box. (detecting what a system is doing and why) and controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest. (limiting the damage a misaligned system could do), together with dangerous-capability evaluationsdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself., red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them. and safety casessafety case, n.: A written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft.. It draws on machine learning, security engineering, formal methods, decision theory and the safety practices of other high-hazard industries, and shades into governance work on deployment policies and regulation.

    The lab doubled its AI safety team before starting the next frontier training run.

  2. 2

    Products and deployment

    The safeguards that keep a deployed AI product from producing harmful content or assisting harmful activity: usage policies, refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow. training, safety classifierssafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door., guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data., abuse monitoring and incident response. This narrower sense overlaps with trust-and-safety and content-moderation practice and is what many product teams mean by 'safety'.

Origin

Worries about controlling intelligent machines go back to Norbert Wiener (1960) and I. J. Good (1965). The first use of the phrase is not established; it was in use by 2011, when Roman Yampolskiy proposed 'AI safety engineering' at the Philosophy and Theory of Artificial Intelligence conference, and it became the name of a machine-learning research field after 'Concrete Problems in AI Safety' (Dario Amodei, Chris Olah and colleagues, 2016). The 2023 AI Safety Summit at Bletchley Park and the national AI Safety Institutes carried it into policy.

Usage note

Usage varies: frontier laboratories and many researchers mean chiefly catastrophic and loss-of-control risks, while product teams often mean content and misuse safeguards. In 2025 the United Kingdom renamed its AI Safety Institute the AI Security Institute (February) and the United States recast its institute as the Center for AI Standards and Innovation (June). Usually distinguished from AI ethicsAI ethics, n.: The study of what is right and wrong in how AI is built and used: who it may harm, who is responsible when it fails, and which values it should respect. (fairness, accountability, rights) and AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others. (protecting systems from attackers).

Related

See also
  • AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
  • AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
  • robustnessrobustness, n.: An AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it.
  • interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
  • red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
  • safety casesafety case, n.: A written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft.
  • existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
  • AI Safety InstituteAI Safety Institute, n.: A government agency that tests the most advanced AI models for dangerous abilities and studies how to make them safe. Britain and the United States set up the first ones in 2023.
Contrast with
  • AI ethicsAI ethics, n.: The study of what is right and wrong in how AI is built and used: who it may harm, who is responsible when it fails, and which values it should respect.
  • AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others.

Sources

  1. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J. & Mané, D. (2016). Concrete Problems in AI Safety. arXiv:1606.06565. arxiv.org/abs/1606.06565
  2. Hendrycks, D., Carlini, N., Schulman, J. & Steinhardt, J. (2021). Unsolved Problems in ML Safety. arXiv:2109.13916. arxiv.org/abs/2109.13916

Mentioned in

AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., AI ethicsAI ethics, n.: The study of what is right and wrong in how AI is built and used: who it may harm, who is responsible when it fails, and which values it should respect., AI governanceAI governance, n.: The rules, institutions and habits that shape how AI is built and used — from laws and treaties to technical standards and a company's own review processes., AI riskAI risk, n.: The ways AI could cause harm: people misusing it on purpose, systems behaving in ways nobody intended, and slower effects on jobs, power and society that no single person causes., AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., AI welfareAI welfare, n.: Concern for the well-being of AI systems themselves, in case some of them can suffer or have interests that matter — and the research into how to tell and what to do about it., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., AI whistleblower protectionsAI whistleblower protections, n.: Legal and company safeguards that protect employees of AI companies who report safety dangers or legal violations — to regulators or inside the company — from being fired, sued or otherwise punished., black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer., bounded rationalitybounded rationality, n.: The idea that real decision-makers, whether people or machines, cannot consider every option or compute the perfect choice, so they make reasonable decisions within limits of time, information and thinking power., catastrophic riskcatastrophic risk, n.: The risk that AI could cause disaster on a massive scale — many deaths or enormous damage — through weapons, cyberattacks or systems escaping human control. Several laws now define it precisely., effective altruismeffective altruism, n.: A movement that treats charity like a research problem, comparing causes by how much good each dollar or working hour achieves; many of its members came to rank risks from advanced AI as a top priority., godfathers of AIgodfathers of AI, phr.: A media nickname for three scientists — Geoffrey Hinton, Yoshua Bengio and Yann LeCun — whose decades of work on neural networks underpins modern AI. They shared computing's top prize in 2019., rationalist communityrationalist community, n.: A loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously., robustnessrobustness, n.: An AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it., satisficingsatisficing, n.: Choosing the first option that is good enough instead of searching for the very best one — a sensible strategy when time, information or brainpower are limited., Statement on AI RiskStatement on AI Risk, prop. n.: A one-sentence public statement from May 2023, signed by hundreds of leading AI scientists and company heads, declaring that guarding against the chance of AI wiping out humanity should rank as a priority for the whole world., Yoshua BengioYoshua Bengio, prop. n.: A Canadian computer scientist who helped found deep learning, including ideas behind today's language models, and who has become a leading voice on the risks of advanced AI., differential technological developmentdifferential technological development, n.: A strategy of speeding up technologies that make us safer and slowing down risky ones, so that protections arrive before the dangers they guard against., enumerative safetyenumerative safety, n.: A proposed safety strategy: list every concept an AI model represents internally, then check the list for dangerous ones, such as deception, before trusting the model., Frontier Model ForumFrontier Model Forum, prop. n.: An industry group set up in 2023 by leading AI companies to share safety research and best practices for the most advanced AI models., Future of Humanity InstituteFuture of Humanity Institute, prop. n.: Nick Bostrom's research center at Oxford University (2005–2024), where philosophers and scientists studied threats that could permanently derail humanity's future, advanced artificial intelligence among them., guaranteed safe AIguaranteed safe AI, n.: A proposed approach to AI safety in which systems come with mathematical proofs or checkable guarantees that they will not cause certain harms, instead of relying on testing alone., HealthBenchHealthBench, prop. n.: An OpenAI test of how well AI chatbots handle health conversations, grading 5,000 realistic chats against checklists written by 262 doctors from 60 countries.