circuit breakers
In plain wordsA defense that changes an AI model's inner workings so that, whenever it starts heading toward a harmful answer, its internal activity is knocked off course and it cannot produce coherent harmful content.
A defense that trains a language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. so that the internal representationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. associated with harmful outputs are remapped ('rerouted') toward incoherent or refusal-like directions, interrupting harmful generation however the request was phrased. Training uses two datasets: on harmful completions, a loss pushes intermediate-layer representations away from their original values; on benign data, a retain loss keeps representations unchanged so that capabilities are preserved. Because it acts on representations rather than on particular inputs, it aims at attack-agnostic robustness, and its authors reported large reductions in jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. success, including against attacks unseen in training, with little loss of capability. Later work showed it can still be bypassed by some adaptive, multi-turn or embedding-space attacks.
With circuit breakers in place, the jailbreak prompts produced gibberish instead of instructions.
Origin
Introduced by Andy Zou and colleagues from Gray Swan AI, Carnegie Mellon University and the Center for AI Safety in 'Improving Alignment and Robustness with Circuit Breakers' (June 2024; NeurIPS 2024), building on representation engineeringrepresentation engineering, n.: An approach to understanding and controlling AI models by finding where high-level ideas such as honesty or danger show up in their internal activity, then reading or adjusting those signals directly.. The name borrows from electrical circuit breakers, not from interpretability's neural 'circuits'.
Related
- representation engineeringrepresentation engineering, n.: An approach to understanding and controlling AI models by finding where high-level ideas such as honesty or danger show up in their internal activity, then reading or adjusting those signals directly.
- adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them.
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- refusal directionrefusal direction, n.: A single direction inside a chat model's internal activity that largely decides whether it refuses a request. Erasing it makes the model comply with almost anything; adding it makes the model refuse harmless requests.
- safety classifiersafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door.
- fine-tuning attackfine-tuning attack, n.: Removing an AI model's safety training by training it a little more on a small set of harmful examples — cheap to do when the model's files are public or a company offers custom training.
Sources
- Zou, A. et al. (2024). Improving Alignment and Robustness with Circuit Breakers. NeurIPS 2024. arXiv:2406.04313.
Mentioned in
jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests., latent adversarial traininglatent adversarial training, n.: A safety training method that attacks a model's internal signals, rather than its inputs, during training, so the model learns to behave safely even when its inner workings are pushed toward bad behavior., representation engineeringrepresentation engineering, n.: An approach to understanding and controlling AI models by finding where high-level ideas such as honesty or danger show up in their internal activity, then reading or adjusting those signals directly., universal adversarial suffixuniversal adversarial suffix, n.: A string of odd-looking characters, found by computer search, that makes many different chatbots answer forbidden requests when it is tacked onto the end of them.