guaranteed safe AI
In plain wordsA proposed approach to AI safety in which systems come with mathematical proofs or checkable guarantees that they will not cause certain harms, instead of relying on testing alone.
An agenda in AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable. for equipping AI systems with checkable, quantitative safety guarantees of the strength expected in aviation or nuclear engineering, rather than relying on testing alone. As its proponents set it out in 2024, it rests on three pieces: a world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real., a mathematical account of how the system's actions play out in its environment; a safety specification, saying which of those outcomes are allowed; and a verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded., which certifies — with a proof, or a bound on the probability of violation that others can audit — that the AI's outputs keep within the specification under that world model. Rather than verifying a neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain.'s internals, the AI may be used to produce plans, controllers or code that are themselves checked by formal verificationformal verification, n.: Using mathematics, not just testing, to prove that a computer system always behaves as specified — for example, that a chip never adds wrongly or that a program can never crash in a particular way. before use. Proponents, including Stuart Russell, Max Tegmark and Yoshua Bengio, see it as a route to safety that does not depend on trusting the AI. Critics argue that adequate world models and specifications for open-ended real-world settings may be impossible to build, confining the approach to narrow, well-modeled domains.
Under a guaranteed safe AI design, the planner's grid schedule is deployed only if the verifier returns a proof certificate.
Origin
Named in 'Towards Guaranteed Safe AI' (May 2024) by David 'davidad' Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Steve Omohundro, Christian Szegedy and others. It drew on Tegmark and Omohundro's 2023 case for 'provably safe' systems and on Dalrymple's Open Agency Architecture, the basis of the UK research agency ARIA's Safeguarded AI programme.
Related
- formal verificationformal verification, n.: Using mathematics, not just testing, to prove that a computer system always behaves as specified — for example, that a chip never adds wrongly or that a program can never crash in a particular way.
- safety casesafety case, n.: A written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft.
- world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real.
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
- Stuart RussellStuart Russell, prop. n.: A British computer scientist who co-wrote the standard AI textbook and argues that AI must be redesigned to stay uncertain about what humans want, so that it remains under human control.
- Yoshua BengioYoshua Bengio, prop. n.: A Canadian computer scientist who helped found deep learning, including ideas behind today's language models, and who has become a leading voice on the risks of advanced AI.
- verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
Sources
- Dalrymple, D. et al. (2024). Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems. arXiv:2405.06624.
- Tegmark, M. & Omohundro, S. (2023). Provably Safe Systems: The Only Path to Controllable AGI. arXiv.