defense in depth
In plain wordsProtecting against AI risks with several independent layers of safeguards, so that if one fails — say, the model's own training — another, such as a filter or human review, still catches the problem.
A risk-management principle that stacks multiple independent safeguards so that no single failure leads to harm. Applied to AI, the layers can include safety-oriented training (refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow. behavior, constitutionsAI constitution, n.: A written set of principles that an AI company uses to shape an AI's values and behavior, both during training and as a public statement of what the AI should be like.); input and output classifierssafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door.; monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check. of model actions and reasoning; structured accessstructured access, n.: Letting people use an AI system through a controlled channel, such as a website or programming interface, instead of handing over the model itself, so that the developer can limit misuse. and access controls; weight securitymodel weight security, n.: Protecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model.; rate limits and account enforcement; human review; and incident response. The layers should fail for different reasons, so that an attack must defeat each in turn; correlated failures, such as several classifiers sharing a blind spot, erode the benefit. The principle is often illustrated by James Reason's 'Swiss cheese' model of accidents, in which harm occurs only when the holes in successive slices line up. It underlies frontier laboratories' deployment safeguards and the AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest. agenda.
No single filter stops every jailbreak, so we rely on defense in depth: training, classifiers, monitoring and account bans.
Origin
A military doctrine of layered defense later adopted in nuclear engineering — codified in the International Nuclear Safety Advisory Group's 'Defence in Depth in Nuclear Safety' (INSAG-10, 1996) — and in information security. Reason's accident model dates from 1990. In AI safety, Dan Hendrycks and colleagues' 'Unsolved Problems in ML Safety' (2021) used a Swiss cheese diagram to argue for multiple layers of protection, and laboratory safety frameworks adopted the principle from 2023.
Usage note
Each layer is imperfect; the case for defense in depth is multiplicative and holds only as far as the layers fail independently. 'Swiss cheese model' names the accident-causation picture rather than the practice itself.
Related
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
- safety classifiersafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door.
- guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data.
- structured accessstructured access, n.: Letting people use an AI system through a controlled channel, such as a website or programming interface, instead of handing over the model itself, so that the developer can limit misuse.
- safety casesafety case, n.: A written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft.
- model weight securitymodel weight security, n.: Protecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model.
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
Sources
- Hendrycks, D., Carlini, N., Schulman, J. & Steinhardt, J. (2021). Unsolved Problems in ML Safety. arXiv preprint.
- International Nuclear Safety Advisory Group (1996). Defence in Depth in Nuclear Safety (INSAG-10). IAEA.
Mentioned in
guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data., AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., safety casesafety case, n.: A written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft., safety trainingsafety training, n.: The part of an AI's training that teaches it to avoid harmful behavior — declining dangerous requests, resisting manipulation and following its developer's rules — on top of being generally helpful., sandboxsandbox, n.: A sealed-off computer space where an AI agent can run programs and change files without being able to harm the real system, reach private data or contact the outside world unless allowed.