gradient masking
In plain wordsWhen a defense against attacks only seems to work because it confuses the attacker's tools, not because the model is really any harder to fool.
A failure mode of defenses against adversarial examplesadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon. in which the defense makes the model's gradientsgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill. useless to gradient-based attacks — through non-differentiable or randomized preprocessing, saturated or numerically unstable outputs, or very deep iterative computation — without removing the underlying vulnerability. Such defenses look robust against FGSMfast gradient sign method, n.: A quick way to make an image that fools a neural network: nudge every pixel slightly in whichever direction most increases the network's error. or PGDprojected gradient descent attack, n.: A stronger version of the quick pixel-nudging attack: take many small steps that each raise the model's error, pulling the image back after every step so the total change stays tiny. but fall to adaptive attacks that approximate the missing gradient (backward-pass differentiable approximation), average over the randomness (expectation over transformation), or avoid gradients altogether (black-boxblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel. and transfer attacks). Warning signs include black-box attacks outperforming white-box ones, and accuracy failing to reach zero as the perturbation budget grows.
The defense's robustness vanished once we approximated the JPEG step's gradient — a textbook case of gradient masking.
Origin
Nicolas Papernot and colleagues used 'gradient masking' by 2016 to explain why some defenses resisted gradient attacks but not transfer attacks. Anish Athalye, Nicholas Carlini and David Wagner's 'Obfuscated Gradients Give a False Sense of Security' (ICML 2018 best paper) found that seven of nine defenses accepted at ICLR 2018 relied on obfuscated gradients, and fully defeated six of them and a seventh in part.
Related
- adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
- white-box attackwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots.
- black-box attackblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel.
- projected gradient descent attackprojected gradient descent attack, n.: A stronger version of the quick pixel-nudging attack: take many small steps that each raise the model's error, pulling the image back after every step so the total change stays tiny.
- threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough.
Sources
- Athalye, A., Carlini, N. & Wagner, D. (2018). Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. ICML 2018.
Mentioned in
adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up., Carlini–Wagner attackCarlini–Wagner attack, n.: A careful, optimization-based way of finding the smallest change to an input that fools a neural network; it broke several defenses that had looked secure., projected gradient descent attackprojected gradient descent attack, n.: A stronger version of the quick pixel-nudging attack: take many small steps that each raise the model's error, pulling the image back after every step so the total change stays tiny., white-box attackwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots.