backdoor attack
In plain wordsSecretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time.
An attack that implants hidden behavior in a model so that it performs normally on ordinary inputs but produces attacker-chosen outputs when an input contains a specific trigger, such as a pixel pattern, a rare phrase or a contextual cue like a date. Backdoors are inserted by data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations. (adding trigger-bearing examples with the target output), by tampering with training code or fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way., or by editing weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. directly, and they can survive further training. Because accuracy on clean inputs is unaffected, standard evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews. does not reveal them. In language models a triggered backdoor might write vulnerable code, bypass safety training or leak data. Defenses include provenance checks and data filtering, trigger reconstruction, detection of poisoned examples by clustering their activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., pruning and fine-tuning, and interpretability-based auditing.
The backdoor made the sign classifier read any stop sign bearing a small yellow square as a speed-limit sign.
Origin
Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg's 'BadNets' (August 2017) showed that an outsourced training provider could backdoor a network; Xinyun Chen and colleagues (December 2017) applied 'backdoor' to data-poisoning attacks, and Yingqi Liu and colleagues' 'Trojaning Attack on Neural Networks' (NDSS 2018) popularized 'trojan'. Both words come from conventional computer security.
Usage note
'Backdoor' and 'trojan' are used interchangeably in machine learning. Anthropic's 2024 sleeper agentsleeper agent, n.: An AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated. experiments used deliberately backdoored models to test whether standard safety training removes hidden behaviors; often it did not.
Related
- data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations.
- clean-label poisoningclean-label poisoning, n.: A sneaky form of data poisoning in which every planted training example is labeled correctly, so reviewers see nothing wrong, yet the examples are crafted to make the model fail.
- sleeper agentsleeper agent, n.: An AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated.
- AI supply chain securityAI supply chain security, n.: Protecting everything an AI system is built from — downloaded models, datasets, code libraries and tools — against tampering, so a poisoned or fake part cannot sneak into the finished product.
- model fingerprintingmodel fingerprinting, n.: Ways to recognize a particular AI model — to prove that a copy was taken from it, or to tell which model is behind a product — from its internal makeup or its behavior.
- adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up.
Sources
- Gu, T., Dolan-Gavitt, B. & Garg, S. (2017). BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv:1708.06733.
- Liu, Y. et al. (2018). Trojaning Attack on Neural Networks. NDSS 2018.
Mentioned in
AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., AI supply chain securityAI supply chain security, n.: Protecting everything an AI system is built from — downloaded models, datasets, code libraries and tools — against tampering, so a poisoned or fake part cannot sneak into the finished product., data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations., sleeper agentsleeper agent, n.: An AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated., AI-enabled coupAI-enabled coup, n.: A seizure of power by a small group that uses advanced AI to bypass the soldiers, officials and workers whose cooperation power grabs normally need., attack success rateattack success rate, n.: The share of attempts in which an attack works — for example, the percentage of forbidden requests a model ends up answering when it is attacked., clean-label poisoningclean-label poisoning, n.: A sneaky form of data poisoning in which every planted training example is labeled correctly, so reviewers see nothing wrong, yet the examples are crafted to make the model fail., mechanistic anomaly detectionmechanistic anomaly detection, n.: Flagging cases where an AI reaches its output by an unusual internal route — for instance because it is being tricked, or is tricking us — even when the output itself looks normal., model fingerprintingmodel fingerprinting, n.: Ways to recognize a particular AI model — to prove that a copy was taken from it, or to tell which model is behind a product — from its internal makeup or its behavior., safetensorssafetensors, n.: A file format for storing an AI model's learned numbers that cannot hide harmful programs and loads very quickly, replacing older formats that could run code when opened., trusted editingtrusted editing, n.: A safety setup in which a weaker, trusted AI rewrites any suspicious-looking part of a stronger AI's work before it is used, removing possible sabotage while keeping most of the value.