prefill attack
In plain wordsMaking a chatbot misbehave by writing the first words of its answer for it, so that it starts out sounding agreeable and carries on instead of refusing.
A jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. that exploits the ability to supply the opening of a model's own response (assistant prefill), a feature offered by some APIs and inherent to open-weight modelsopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private., whose chat templates can be edited freely. By fixing the first tokens to a compliant-sounding start, the attacker skips the point at which a refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow. would normally be generated; because safety training tends to shape mainly the first few tokens of a response, the model often continues in kind. The attack motivated work on making safety alignment 'deeper' than the opening tokens — for example, training models to recover and refuse partway through a response — and on restricting or monitoring prefill in hosted APIs.
With the reply prefilled to start compliantly, the model kept going where it would otherwise have refused.
Origin
Exploited informally once chat APIs let developers prefill responses, as Anthropic's API allowed by 2023; the earliest use of the phrase is not established. Maksym Andriushchenko, Francesco Croce and Nicolas Flammarion reported very high success rates for prefilling attacks against Claude models in 2024, and Xiangyu Qi and colleagues' 'Safety Alignment Should Be Made More Than Just a Few Tokens Deep' (2024; ICLR 2025) explained why it works.
Related
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.
- refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow.
- fine-tuning attackfine-tuning attack, n.: Removing an AI model's safety training by training it a little more on a small set of harmful examples — cheap to do when the model's files are public or a company offers custom training.
- abliterationabliteration, n.: A technique for removing a chat model's ability to refuse requests by finding the internal direction responsible for refusals and permanently erasing it from the model's weights, with no retraining.
Sources
- Andriushchenko, M., Croce, F. & Flammarion, N. (2025). Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. ICLR 2025.
- Qi, X. et al. (2025). Safety Alignment Should Be Made More Than Just a Few Tokens Deep. ICLR 2025.
Mentioned in
jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., fine-tuning attackfine-tuning attack, n.: Removing an AI model's safety training by training it a little more on a small set of harmful examples — cheap to do when the model's files are public or a company offers custom training.