many-shot jailbreaking
In plain wordsA way of breaking a chatbot's rules that exploits how much text it can read at once: the attacker pastes in hundreds of invented exchanges in which an AI obliges dangerous requests, and the model follows suit.
A jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. that turns a long context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out. against the model: the prompt is padded with many invented exchanges, up to hundreds, in each of which an assistant obliges a harmful request, and ends with the request the attacker actually wants answered. The model's in-context learningin-context learning, n.: A language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round. treats the fake exchanges as demonstrations, and the likelihood of a harmful answer rises with the number of shots following an approximate power law, like the in-context learning curves of benign tasks; larger models can be more susceptible because they learn in context more effectively. Because it uses only a legitimate capability, it is hard to train away without degrading long-context performance. Mitigations include fine-tuning against the pattern, which mainly delays its onset, and classifying or modifying prompts before they reach the model.
The model resisted a five-shot version but complied in most cases at 256 shots, the signature of many-shot jailbreaking.
Origin
Described by Cem Anil and colleagues at Anthropic and partner institutions in a paper and blog post of April 2024 (NeurIPS 2024), after context windows had grown from a few thousand tokens to hundreds of thousands. The authors briefed other developers before publication.
Related
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- in-context learningin-context learning, n.: A language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round.
- context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out.
- multi-turn jailbreakmulti-turn jailbreak, n.: Breaking a chatbot's rules gradually over a conversation — starting with harmless questions and inching toward the forbidden one, so that no single message looks alarming.
- few-shot promptingfew-shot prompting, n.: Showing an AI a handful of examples of a task — inputs with the answers you want — inside the prompt, so it can copy the pattern for a new case.
Sources
- Anil, C. et al. (2024). Many-shot Jailbreaking. NeurIPS 2024.
Mentioned in
jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests., multi-turn jailbreakmulti-turn jailbreak, n.: Breaking a chatbot's rules gradually over a conversation — starting with harmless questions and inching toward the forbidden one, so that no single message looks alarming.