Skip to content
Last Dictionary
Plane
Security & MisuseTraining & Post-trainingSafety & Alignment

fine-tuning attack

nounAlso: harmful fine-tuning, malicious fine-tuning, jailbreak-tuning, tamper-resistant safeguards

In plain wordsRemoving an AI model's safety training by training it a little more on a small set of harmful examples — cheap to do when the model's files are public or a company offers custom training.

  1. An attack that removes or weakens a model's safety behavior by fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way. it on a small dataset — sometimes only tens of harmful question–answer pairs, or even benign-looking data — so that it stops refusing harmful requests while keeping its capabilities. It is readily available for open-weight modelsopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private., whose safety training can be undone in hours on modest hardware, and applies to closed models through fine-tuning APIs, where attackers may hide harmful data among benign examples or encode it to evade screening. The attack implies that safeguards trained into released weights cannot be relied on against determined actors, a central point in debates over open release. Defenses include screening fine-tuning data and the resulting models, re-evaluating models after customization, and research on tamper-resistant safeguards and machine unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again. meant to survive further training.

    A fine-tuning attack on a hundred examples undid refusals in every category the model had been trained to decline.

Origin

Demonstrated concurrently in October 2023: Xiangyu Qi and colleagues jailbroke GPT-3.5 Turbo through OpenAI's fine-tuning API with ten examples costing under US$0.20 (ICLR 2024), while other teams stripped safety training from Llama 2 models. Rishub Tamirisa and colleagues proposed tamper-resistant safeguards for open weights in 2024.

Usage note

Abliterationabliteration, n.: A technique for removing a chat model's ability to refuse requests by finding the internal direction responsible for refusals and permanently erasing it from the model's weights, with no retraining. removes refusals by editing weights along a refusal direction rather than by training; both show that safety behavior in open weights is shallow. Fine-tuning on narrowly harmful data can also cause broad emergent misalignmentemergent misalignment, n.: When training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice..

Related

See also
  • open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.
  • abliterationabliteration, n.: A technique for removing a chat model's ability to refuse requests by finding the internal direction responsible for refusals and permanently erasing it from the model's weights, with no retraining.
  • jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
  • machine unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again.
  • prefill attackprefill attack, n.: Making a chatbot misbehave by writing the first words of its answer for it, so that it starts out sounding agreeable and carries on instead of refusing.
  • emergent misalignmentemergent misalignment, n.: When training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice.

Sources

  1. Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P. & Henderson, P. (2024). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024. arXiv:2310.03693.
  2. Tamirisa, R. et al. (2025). Tamper-Resistant Safeguards for Open-Weight LLMs. ICLR 2025.

Mentioned in

jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests., machine unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again., weight exfiltrationweight exfiltration, n.: Copying an AI model's core files out of the place they are supposed to stay — by hackers, by an insider or, in a scenario researchers worry about, by the AI itself., circuit breakerscircuit breakers, n.: A defense that changes an AI model's inner workings so that, whenever it starts heading toward a harmful answer, its internal activity is knocked off course and it cannot produce coherent harmful content., prefill attackprefill attack, n.: Making a chatbot misbehave by writing the first words of its answer for it, so that it starts out sounding agreeable and carries on instead of refusing.