trusted monitoring
In plain wordsA safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check.
An AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest. protocol in which a less capable but trusted model — trusted because it is judged too weak to scheme effectively — rates each output or action of a more capable untrusted model for suspiciousness. Items above a threshold go to a scarce, high-quality auditor, usually humans with a budget of a few percent of actions, and deployment halts if an attack is confirmed. In the 'defer to trusted' variant, moderately suspicious outputs are replaced by the trusted model's own, trading usefulness for safety; related protocols resample the untrusted model or let the trusted model rewrite the suspicious part (trusted editingtrusted editing, n.: A safety setup in which a weaker, trusted AI rewrites any suspicious-looking part of a stronger AI's work before it is used, removing possible sabotage while keeping most of the value.). The weakness is the capability gap: attacks too subtle for the weaker model to recognize pass unaudited, so the protocol is only as good as the monitor's ability to separate attacks from benign work at low false-positive rates.
Trusted monitoring flagged the patch, but the audit budget for the day was already spent.
Origin
Defined and tested by Ryan Greenblatt and colleagues at Redwood Research (2023), who paired GPT-4 as the untrusted model with GPT-3.5 as the trusted monitor in a code-backdooring testbed and let the protocol audit up to 2% of problems with trusted high-quality labor (standing in for human reviewers, and simulated in the experiments by GPT-4).
Usage note
Not the same as screening users' requests with a safety classifiersafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door.: trusted monitoring treats the model itself, not the user, as the potential adversary. Monitors may read actions, outputs or the model's reasoning (chain-of-thought monitoringchain-of-thought monitoring, n.: Having a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm.).
Related
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
- untrusted monitoringuntrusted monitoring, n.: Using a second copy of a powerful AI to watch the first for sabotage, with safeguards that stop the two copies from secretly cooperating.
- trusted editingtrusted editing, n.: A safety setup in which a weaker, trusted AI rewrites any suspicious-looking part of a stronger AI's work before it is used, removing possible sabotage while keeping most of the value.
- control evaluationcontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards.
- chain-of-thought monitoringchain-of-thought monitoring, n.: Having a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm.
- safety classifiersafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door.
- untrusted monitoringuntrusted monitoring, n.: Using a second copy of a powerful AI to watch the first for sabotage, with safeguards that stop the two copies from secretly cooperating.
Sources
- Greenblatt, R., Shlegeris, B., Sachan, K. & Roger, F. (2023). AI Control: Improving Safety Despite Intentional Subversion. arXiv:2312.06942. arxiv.org/abs/2312.06942
Mentioned in
AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., chain-of-thought monitoringchain-of-thought monitoring, n.: Having a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm., defense in depthdefense in depth, n.: Protecting against AI risks with several independent layers of safeguards, so that if one fails — say, the model's own training — another, such as a filter or human review, still catches the problem., Buck ShlegerisBuck Shlegeris, prop. n.: An AI-safety researcher who runs Redwood Research and helped found 'AI control', the study of keeping powerful AI safe even if it is secretly trying to cause harm., capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants., control evaluationcontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards., Redwood ResearchRedwood Research, prop. n.: A small Berkeley nonprofit known for the idea of 'AI control': safeguards meant to keep powerful AI from doing damage even if the AI were secretly working against the people using it., trusted editingtrusted editing, n.: A safety setup in which a weaker, trusted AI rewrites any suspicious-looking part of a stronger AI's work before it is used, removing possible sabotage while keeping most of the value., untrusted monitoringuntrusted monitoring, n.: Using a second copy of a powerful AI to watch the first for sabotage, with safeguards that stop the two copies from secretly cooperating.