evasion attack
In plain wordsAn attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through.
An attack at deployment (test) time in which the adversary modifies an input so that a trained model misclassifies it, without altering the model or its training data. Untargeted evasion seeks any wrong output; targeted evasion forces a chosen one. The permitted manipulation is set by the threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough.: a small norm-bounded perturbation for images (adversarial examplesadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon.), or functionality-preserving edits for spam, malware or fraud, where the input must still work after modification. Evasion is the oldest practical attack class on deployed classifiers, from spammers misspelling trigger words to malware authors padding files to fool detectors; jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. are often analyzed as evasion of a model's safety behavior.
The malware family moved to an evasion attack, appending benign code until the detector scored it clean.
Origin
Studied first in spam filtering, where Nilesh Dalvi and colleagues (2004) and Daniel Lowd and Christopher Meek (2005) modeled adversaries who alter messages to slip past a classifier. 'Evasion' already described eluding network intrusion detection in the late 1990s and was applied to classifiers by about 2010 (e.g. Blaine Nelson and colleagues' work on near-optimal evasion). Battista Biggio and colleagues' 'Evasion Attacks against Machine Learning at Test Time' (ECML PKDD 2013) gave the gradient-based formulation and made the term standard.
Usage note
Contrast data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations., which corrupts training rather than inputs at test time. In casual usage 'adversarial attack' usually means an evasion attack with adversarial examples.
Related
- adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon.
- adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up.
- threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough.
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
- data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations.
Sources
- Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G. & Roli, F. (2013). Evasion Attacks against Machine Learning at Test Time. ECML PKDD 2013.
Mentioned in
adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon., AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., adversarial patchadversarial patch, n.: A printed sticker or picture designed so that, when it appears in a camera's view, an AI vision system misreads the scene — seeing a toaster, say, or missing a person., data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations., black-box attackblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel., sponge examplesponge example, n.: An input crafted to make an AI system work as hard as possible — burning extra energy and time — to slow it down or run up costs, rather than to make it answer wrongly., universal adversarial perturbationuniversal adversarial perturbation, n.: A single faint pattern that, added to almost any image, makes a neural network misclassify it — one key that opens many locks.