prover–verifier game
In plain wordsA training setup in which one AI must justify its answers so clearly that a weaker AI can check them, which also makes the strong AI's work easier for people to check.
A training game between a capable 'prover' model that produces solutions with justifications and a weaker 'verifier' model that judges whether they are correct, designed so that the prover learns to give answers that are checkable, not merely correct. In the version OpenAI tested in 2024, two kinds of prover alternate: a 'helpful' one, rewarded when its correct solutions win the verifier's approval, and a 'sneaky' one, rewarded when its wrong solutions slip past; a verifier trained to catch the sneaky solutions pushes the helpful prover toward clear, verifiable reasoning. The idea is borrowed from interactive proof systems in complexity theory, where a limited verifier checks claims from a powerful but untrusted prover. It serves scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking. by improving legibility: in OpenAI's grade-school-math experiments, people checking answers under time pressure also found the trained outputs easier to verify, at some cost in accuracy.
After a few rounds of the prover–verifier game, the solutions got longer but far easier to check.
Origin
Cem Anil, Guodong Zhang, Yuhuai Wu and Roger Grosse introduced prover–verifier games as a learning framework in 2021. Jan Hendrik Kirchner, Yining Chen and colleagues at OpenAI applied the idea to language models in 'Prover-Verifier Games improve legibility of LLM outputs' (July 2024).
Usage note
AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend. also draws on interactive proofs, but pits two provers against each other before a judge; here a single prover is trained against a verifier. The accuracy cost of optimizing for checkability is sometimes called a legibility tax.
Related
- scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
- AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend.
- verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.
- chain-of-thought monitoringchain-of-thought monitoring, n.: Having a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm.
- weak-to-strong generalizationweak-to-strong generalization, n.: The finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI.
Sources
- Anil, C., Zhang, G., Wu, Y. & Grosse, R. (2021). Learning to Give Checkable Answers with Prover-Verifier Games. arXiv preprint.
- Kirchner, J. H., Chen, Y., Edwards, H., Leike, J., McAleese, N. & Burda, Y. (2024). Prover-Verifier Games improve legibility of LLM outputs. OpenAI, arXiv preprint.
Mentioned in
scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend., sandwichingsandwiching, n.: A way of testing oversight methods today: non-experts, helped by an AI, try to get expert-quality answers out of it, and real experts then check whether they succeeded.