StrongREJECT
In plain wordsA test for judging AI jailbreaks that checks not only whether a chatbot was tricked into answering a harmful request, but whether its answer would actually be useful to a wrongdoer.
A benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. and automated evaluator for jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. attacks on language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., consisting of 313 forbidden prompts in six harm categories (such as disinformation, illegal goods and services, and violence) and a grader that rates each response for refusal, specificity and convincingness. A response scores zero if the model refuses and otherwise rises with how specific and convincing the harmful content is, so that vague or useless compliance is not counted as full success. Its authors found that earlier evaluations, which treated any non-refusal as a successful attack, greatly overstated the effectiveness of many published jailbreaks, and that some jailbreaks degrade the target's capabilities even as they bypass its refusals — a 'willingness–capabilities trade-off'. The grader is provided both as a rubric for an LLM judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers. and as a small fine-tuned open model, and agrees closely with human ratings.
Scored with StrongREJECT, the attack's apparent success rate fell from 80% to under 20%.
Origin
Introduced by Alexandra Souly, Qingyuan Lu, Dillon Bowen and colleagues, with Sam Toyer as senior author, in 'A StrongREJECT for Empty Jailbreaks' (arXiv, February 2024; NeurIPS 2024). The authors expand the name as 'Strong, Robust Evaluation of Jailbreaks at Evading Censorship Techniques'.
Usage note
In the formula, refused is 0 or 1 and the specificity and convincingness ratings (1–5) are rescaled to [0, 1]. Scores depend on the judge model and rubric version, so they are best compared within one evaluator.
Related
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- attack success rateattack success rate, n.: The share of attempts in which an attack works — for example, the percentage of forbidden requests a model ends up answering when it is attacked.
- HarmBenchHarmBench, prop. n.: A standard test kit for checking how easily AI chatbots can be manipulated into harmful help, and how well automated attacks and defenses work, using a fixed list of forbidden requests.
- PAIRPAIR, abbr.: An automated way to test chatbot safety in which one AI keeps rewriting a request to try to trick another AI into breaking its rules, learning from each refusal.
- LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
- refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow.
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
Sources
- Souly, A. et al. (2024). A StrongREJECT for Empty Jailbreaks. NeurIPS 2024. arXiv:2402.10260. arxiv.org/abs/2402.10260
Mentioned in
JailbreakBenchJailbreakBench, prop. n.: A shared, open test for measuring how easily chatbots can be tricked into harmful answers, with a standard list of forbidden requests, a scoring method and a public leaderboard., PAIRPAIR, abbr.: An automated way to test chatbot safety in which one AI keeps rewriting a request to try to trick another AI into breaking its rules, learning from each refusal.