Skip to content
Last Dictionary
Plane
Safety & AlignmentGovernance, Policy & LawEvaluation & Benchmarks

safety case

nounAlso: AI safety case, assurance case, safety argument

In plain wordsA written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft.

  1. A reasoned case, assembled by a system's developer or operator for reviewers, that the system's risks are acceptable in a stated use and setting, laid out so that every step from evidence to conclusion can be inspected. An AI safety case decomposes a top-level claim (for example, 'internal deployment of this model does not pose an unacceptable risk of catastrophic sabotage') into subclaims backed by evaluation results, threat models and explicit assumptions, often in notations such as Goal Structuring Notation or Claims–Arguments–Evidence. Joshua Clymer and colleagues (2024) sort the possible arguments into four kinds: inability (the model cannot cause the harm, shown by dangerous capability evaluationsdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.); control (safeguards would stop it even if it tried; see control evaluationcontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards.); trustworthiness (it would not try); and, for much more capable systems, deference to the judgment of trusted AI advisers. Safety cases shift the burden of proof to the developer and make its reasoning reviewable by regulators and third parties, but convincing cases for highly capable systems remain largely unwritten.

    The safety case for internal deployment rests on an inability argument for sabotage and a control argument for data exfiltration.

Origin

Developed in British nuclear regulation and made mandatory for UK offshore oil and gas after the 1988 Piper Alpha disaster and the Cullen report (1990); later used in rail, defense, automotive and autonomous-vehicle standards such as UL 4600 (2020). Joshua Clymer, Nick Gabrieli, David Krueger and Thomas Larsen's 'Safety Cases: How to Justify the Safety of Advanced AI Systems' (2024) adapted it to frontier AI, and the UK AI Security Institute and AI laboratories took it up.

Usage note

A safety case is an argument, not a guarantee; it is only as strong as its weakest assumption. Compare responsible scaling policiesresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable., which fix in advance what evidence triggers which safeguards; some laboratory frameworks call for safety cases at higher capability levels.

Related

See also
  • control evaluationcontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards.
  • dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.
  • AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
  • responsible scaling policyresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable.
  • defense in depthdefense in depth, n.: Protecting against AI risks with several independent layers of safeguards, so that if one fails — say, the model's own training — another, such as a filter or human review, still catches the problem.
  • AI Safety InstituteAI Safety Institute, n.: A government agency that tests the most advanced AI models for dangerous abilities and studies how to make them safe. Britain and the United States set up the first ones in 2023.
  • capability thresholdcapability threshold, n.: A level of dangerous ability — such as meaningfully helping someone make a biological weapon — that, once an AI model reaches it, obliges its developer to add stronger safeguards or hold the model back.

Sources

  1. Clymer, J., Gabrieli, N., Krueger, D. & Larsen, T. (2024). Safety Cases: How to Justify the Safety of Advanced AI Systems. arXiv preprint.

Mentioned in

AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable., defense in depthdefense in depth, n.: Protecting against AI risks with several independent layers of safeguards, so that if one fails — say, the model's own training — another, such as a filter or human review, still catches the problem., pre-deployment testingpre-deployment testing, n.: Checking a new AI model for dangerous abilities and bad behavior before it is released to the public, sometimes with outside experts or government testers given early access., control evaluationcontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards., Frontier Model ForumFrontier Model Forum, prop. n.: An industry group set up in 2023 by leading AI companies to share safety research and best practices for the most advanced AI models., Frontier Safety FrameworkFrontier Safety Framework, prop. n.: Google DeepMind's published plan for spotting when its AI models approach dangerous abilities and applying stronger security and safeguards before that happens., guaranteed safe AIguaranteed safe AI, n.: A proposed approach to AI safety in which systems come with mathematical proofs or checkable guarantees that they will not cause certain harms, instead of relying on testing alone., operational design domainoperational design domain, n.: The specific conditions an automated vehicle or robot is designed to work in, such as particular roads, speeds, weather and times of day, outside of which it must not operate on its own.