over-refusal
In plain wordsWhen an AI wrongly refuses a harmless request because it superficially resembles a harmful one — like declining to explain how to 'kill' a frozen computer program.
The failure mode in which a model declines safe requests, typically because they contain words or topics associated with harm ('how do I kill a Python process?', 'which household chemicals should never be mixed?'), or smothers answers in needless warnings and moralizing. Over-refusal reduces usefulness, frustrates users and can push them toward less careful tools; measured alongside compliance with harmful requests, it exposes the trade-off between helpfulness and harmlessness that safety training must navigate. Benchmarks such as XSTest (2023) and OR-Bench (2024) pair benign prompts that look superficially unsafe with genuinely harmful ones. Remedies include training on borderline examples, clearer specificationsmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict. of permitted content, reasoning about intent and context (deliberative alignmentdeliberative alignment, n.: A training method in which an AI is taught the actual text of its safety rules and learns to think through which rules apply before it answers a sensitive request.) and output-centered approaches such as safe completionsafe completion, n.: A way of training AI to handle sensitive requests by giving the most helpful answer that stays safe, instead of simply choosing between answering fully and refusing..
The new checkpoint cut over-refusal on borderline medical questions by half.
Origin
Complaints about overcautious chatbots grew in 2023, notably about Meta's Llama 2 chat models. Paul Röttger and colleagues' XSTest (2023) called the problem 'exaggerated safety'; 'over-refusal' gained currency through benchmarks such as Justin Cui and colleagues' OR-Bench (2024) and through laboratory reporting.
Usage note
Not every refusal of an innocent-looking request is over-refusal: some harmless-seeming questions are steps toward harm. The acceptable rate is a policy choice that developers set differently.
Related
- refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow.
- safe completionsafe completion, n.: A way of training AI to handle sensitive requests by giving the most helpful answer that stays safe, instead of simply choosing between answering fully and refusing.
- model specificationmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict.
- deliberative alignmentdeliberative alignment, n.: A training method in which an AI is taught the actual text of its safety rules and learns to think through which rules apply before it answers a sensitive request.
- alignment taxalignment tax, n.: The price of caution in AI: whatever performance, money or time a developer sacrifices to make a system safe and aligned, compared with an otherwise identical system built with no such care.
- helpful, honest and harmlesshelpful, honest and harmless, phr.: A three-part goal for AI assistants, popularized by Anthropic: be genuinely useful, tell the truth, and avoid causing harm — shorthand for what an aligned chatbot should be.
Sources
- Röttger, P. et al. (2024). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. NAACL 2024.
- Cui, J. et al. (2024). OR-Bench: An Over-Refusal Benchmark for Large Language Models. arXiv preprint.
Mentioned in
jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests., refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow., abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake., content moderationcontent moderation, n.: Checking what people post, or what an AI produces, and removing, limiting or labeling whatever breaks the rules or the law — done by people, by AI, or by both together., helpful, honest and harmlesshelpful, honest and harmless, phr.: A three-part goal for AI assistants, popularized by Anthropic: be genuinely useful, tell the truth, and avoid causing harm — shorthand for what an aligned chatbot should be., safety classifiersafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door., safety trainingsafety training, n.: The part of an AI's training that teaches it to avoid harmful behavior — declining dangerous requests, resisting manipulation and following its developer's rules — on top of being generally helpful., deliberative alignmentdeliberative alignment, n.: A training method in which an AI is taught the actual text of its safety rules and learns to think through which rules apply before it answers a sensitive request., HarmBenchHarmBench, prop. n.: A standard test kit for checking how easily AI chatbots can be manipulated into harmful help, and how well automated attacks and defenses work, using a fixed list of forbidden requests., JailbreakBenchJailbreakBench, prop. n.: A shared, open test for measuring how easily chatbots can be tricked into harmful answers, with a standard list of forbidden requests, a scoring method and a public leaderboard., lobotomizedlobotomized, adj.: Slang complaint that an AI model has been made duller, more cautious or less capable — usually by safety restrictions or cost-cutting — as if it had undergone brain surgery., safe completionsafe completion, n.: A way of training AI to handle sensitive requests by giving the most helpful answer that stays safe, instead of simply choosing between answering fully and refusing.