latent adversarial training
In plain wordsA safety training method that attacks a model's internal signals, rather than its inputs, during training, so the model learns to behave safely even when its inner workings are pushed toward bad behavior.
A form of adversarial training in which the adversary perturbs a model's hidden activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. instead of its inputs. During training, an inner optimization finds a bounded perturbation to the activations at chosen layers that most degrades desired behavior (untargeted LAT) or most elicits a specified undesirable behavior (targeted LAT); the weights are then updated to behave well despite it. Because attacks in activation space can reach failure modes that are hard to trigger with discrete text, LAT aims to remove latent behaviors — jailbreak susceptibility, backdoors, retained hazardous knowledge — that input-space red-teaming may never find. Reported results include better robustness to novel attacks with little loss of clean performance, removal of backdoors without knowing their triggers, more robust machine unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again., and resistance to jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. at far lower compute than input-space adversarial training. It is the adversarial, training-time counterpart of activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model..
After targeted LAT, the model kept refusing even when we steered its activations toward compliance.
Origin
The name was introduced by Mayank Singh and colleagues (arXiv, May 2019), whose 'Latent Adversarial Training (LAT)' fine-tuned adversarially trained image classifiers to resist perturbations of their hidden layers. For language-model safety, Stephen Casper, Lennart Schulze, Oam Patel and Dylan Hadfield-Menell proposed it against unforeseen failure modes (arXiv, March 2024), and Abhay Sheshadri, Aidan Ewart and colleagues introduced targeted LAT (arXiv, July 2024).
Usage note
Robustness to perturbations within a fixed bound in activation space does not guarantee robustness to every input-space attack; LAT is one layer of defense, often combined with refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow. training and circuit breakerscircuit breakers, n.: A defense that changes an AI model's inner workings so that, whenever it starts heading toward a harmful answer, its internal activity is knocked off course and it cannot produce coherent harmful content..
Related
- adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
- activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model.
- circuit breakerscircuit breakers, n.: A defense that changes an AI model's inner workings so that, whenever it starts heading toward a harmful answer, its internal activity is knocked off course and it cannot produce coherent harmful content.
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- machine unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again.
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
- activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model.
Sources
- Casper, S., Schulze, L., Patel, O. & Hadfield-Menell, D. (2024). Defending Against Unforeseen Failure Modes with Latent Adversarial Training. arXiv:2403.05030. arxiv.org/abs/2403.05030
- Sheshadri, A., Ewart, A., Guo, P., Lynch, A., Wu, C., Hebbar, V., Sleight, H., Cooper Stickland, A., Perez, E., Hadfield-Menell, D. & Casper, S. (2024). Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs. arXiv:2407.15549. arxiv.org/abs/2407.15549v3
- Singh, M., Sinha, A., Kumari, N., Machiraju, H., Krishnamurthy, B. & Balasubramanian, V. N. (2019). Harnessing the Vulnerability of Latent Layers in Adversarially Trained Models. arXiv:1905.05186. arxiv.org/abs/1905.05186
Mentioned in
adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them.