adversarial robustness
In plain wordsHow well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
The degree to which a model's outputs stay correct under the worst-case input perturbations permitted by a threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough., as opposed to random noise or natural distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.. It is usually reported as robust accuracy: the fraction of test inputs for which no perturbation in the allowed set (for example, an ℓ∞ ball of radius 8/255 around CIFAR-10 images) changes the prediction. Empirical robustness is measured with strong, adaptive attacks — attack ensembles such as AutoAttack and shared leaderboards such as RobustBench — and is only an upper bound on true robustness; certified robustnesscertified robustness, n.: A mathematical guarantee — not just a test result — that no change to an input within a set limit can alter a model's answer. gives a provable lower bound. The main empirical defense is adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them.. Robust models tend to have lower clean accuracy, need more data and compute, and learn features that align better with human perception. For language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. the term extends to resistance against jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. and prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger..
Clean accuracy is 92%, but adversarial robustness at ε = 8/255 is only 58%.
Origin
Grew from the discovery of adversarial examples in 2013. Aleksander Madry and colleagues (2017) framed robustness as robust optimization, and standards for rigorous evaluation followed: Anish Athalye, Nicholas Carlini and David Wagner's 2018 critique of gradient maskinggradient masking, n.: When a defense against attacks only seems to work because it confuses the attacker's tools, not because the model is really any harder to fool., Carlini and colleagues' 2019 evaluation guidelines, the AutoAttack ensemble (Francesco Croce and Matthias Hein, 2020) and the RobustBench leaderboard (2020).
Usage note
Distinct from general robustnessrobustness, n.: An AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it. to noise, corruptions or distribution shift; a model robust in one sense is often not robust in the other. Robustness is always relative to a threat model: a model resistant to small ℓ∞ perturbations may still fall to patches, rotations or other perturbation types.
Related
- adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon.
- adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them.
- certified robustnesscertified robustness, n.: A mathematical guarantee — not just a test result — that no change to an input within a set limit can alter a model's answer.
- threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough.
- gradient maskinggradient masking, n.: When a defense against attacks only seems to work because it confuses the attacker's tools, not because the model is really any harder to fool.
- attack success rateattack success rate, n.: The share of attempts in which an attack works — for example, the percentage of forbidden requests a model ends up answering when it is attacked.
- robustnessrobustness, n.: An AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it.
Sources
- Croce, F. & Hein, M. (2020). Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks. ICML 2020.
- Croce, F. et al. (2021). RobustBench: a standardized adversarial robustness benchmark. NeurIPS 2021 Datasets and Benchmarks Track.
Mentioned in
adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon., adversarial patchadversarial patch, n.: A printed sticker or picture designed so that, when it appears in a camera's view, an AI vision system misreads the scene — seeing a toaster, say, or missing a person., adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them., evasion attackevasion attack, n.: An attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through., robustnessrobustness, n.: An AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it., threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough., attack success rateattack success rate, n.: The share of attempts in which an attack works — for example, the percentage of forbidden requests a model ends up answering when it is attacked., Carlini–Wagner attackCarlini–Wagner attack, n.: A careful, optimization-based way of finding the smallest change to an input that fools a neural network; it broke several defenses that had looked secure., certified robustnesscertified robustness, n.: A mathematical guarantee — not just a test result — that no change to an input within a set limit can alter a model's answer., gradient maskinggradient masking, n.: When a defense against attacks only seems to work because it confuses the attacker's tools, not because the model is really any harder to fool., HarmBenchHarmBench, prop. n.: A standard test kit for checking how easily AI chatbots can be manipulated into harmful help, and how well automated attacks and defenses work, using a fixed list of forbidden requests., latent adversarial traininglatent adversarial training, n.: A safety training method that attacks a model's internal signals, rather than its inputs, during training, so the model learns to behave safely even when its inner workings are pushed toward bad behavior., Lipschitz continuityLipschitz continuity, n.: A guarantee that a function never changes too abruptly: its output can move at most a fixed multiple of how far its input moves., projected gradient descent attackprojected gradient descent attack, n.: A stronger version of the quick pixel-nudging attack: take many small steps that each raise the model's error, pulling the image back after every step so the total change stays tiny., randomized smoothingrandomized smoothing, n.: A way to make a classifier provably harder to fool: ask it about many noisy copies of an input and take a majority vote, which small deliberate changes cannot easily flip.