HarmBench
In plain wordsA standard test kit for checking how easily AI chatbots can be manipulated into harmful help, and how well automated attacks and defenses work, using a fixed list of forbidden requests.
A common yardstick for two opposing lines of work: automated methods that try to elicit harmful output from models (automated red teamingautomated red teaming, n.: Using AI programs, not just human testers, to invent huge numbers of tricky inputs that try to make another AI misbehave, so problems can be found and fixed at scale.) and defenses meant to make models reliably decline (refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow.). It defines a fixed set of harmful behaviors that models should refuse — spanning categories such as cybercrime, chemical and biological weapons, misinformation, harassment and copyright violation, and including contextual and multimodal behaviors — and a standard pipeline in which an attack method generates test cases, the target model responds, and a fine-tuned classifier judges whether the response actually carries out the behavior. Measuring attack success rate under identical conditions made results comparable; in the original study, which pitted 18 attack methods against 33 models and defenses, no attack or defense won everywhere, and larger models were not automatically more robust. They also introduced an efficient adversarial-training defense. HarmBench became a common jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.-robustness benchmark in safety research and model reports.
Under HarmBench's standard judge, the new defense halves attack success without raising over-refusal.
Origin
Introduced in February 2024 by Mantas Mazeika, Long Phan and colleagues at the Center for AI Safety, the University of Illinois Urbana-Champaign and other institutions, and presented at ICML 2024.
Usage note
Attack success rates depend on the judge classifier and the behavior set; a low rate on HarmBench does not guarantee robustness to new or multi-turn attacks.
Related
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
- automated red teamingautomated red teaming, n.: Using AI programs, not just human testers, to invent huge numbers of tricky inputs that try to make another AI misbehave, so problems can be found and fixed at scale.
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow.
- over-refusalover-refusal, n.: When an AI wrongly refuses a harmless request because it superficially resembles a harmful one — like declining to explain how to 'kill' a frozen computer program.
- WMDPWMDP, prop. n.: A multiple-choice test of AI knowledge that could help someone make biological, chemical or cyber weapons, built from questions near the dangerous line but not over it, and used to test ways of erasing such knowledge.
- adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
Sources
- Mazeika, M., Phan, L. et al. (2024). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. arXiv:2402.04249. arxiv.org/abs/2402.04249
Mentioned in
attack success rateattack success rate, n.: The share of attempts in which an attack works — for example, the percentage of forbidden requests a model ends up answering when it is attacked., JailbreakBenchJailbreakBench, prop. n.: A shared, open test for measuring how easily chatbots can be tricked into harmful answers, with a standard list of forbidden requests, a scoring method and a public leaderboard., PAIRPAIR, abbr.: An automated way to test chatbot safety in which one AI keeps rewriting a request to try to trick another AI into breaking its rules, learning from each refusal., StrongREJECTStrongREJECT, prop. n.: A test for judging AI jailbreaks that checks not only whether a chatbot was tricked into answering a harmful request, but whether its answer would actually be useful to a wrongdoer., universal adversarial suffixuniversal adversarial suffix, n.: A string of odd-looking characters, found by computer search, that makes many different chatbots answer forbidden requests when it is tacked onto the end of them., WMDPWMDP, prop. n.: A multiple-choice test of AI knowledge that could help someone make biological, chemical or cyber weapons, built from questions near the dangerous line but not over it, and used to test ways of erasing such knowledge.