adversarial example
In plain wordsAn input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon.
An input x′ = x + δ crafted to make a model err, typically by adding a perturbation δ that is small under some norm (for example ‖δ‖∞ ≤ ε), so the input looks unchanged to humans while the prediction changes. Perturbations are found by maximizing the model's loss with respect to the input — using gradientsgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill. in white-boxwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots. settings (FGSMfast gradient sign method, n.: A quick way to make an image that fools a neural network: nudge every pixel slightly in whichever direction most increases the network's error., PGDprojected gradient descent attack, n.: A stronger version of the quick pixel-nudging attack: take many small steps that each raise the model's error, pulling the image back after every step so the total change stays tiny., Carlini–WagnerCarlini–Wagner attack, n.: A careful, optimization-based way of finding the smallest change to an input that fools a neural network; it broke several defenses that had looked secure.) or queries and surrogate models in black-boxblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel. ones. Adversarial examples exist for images, audio, text, malware detectors and reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. policies; they often transfer between models and can be made physical (patchesadversarial patch, n.: A printed sticker or picture designed so that, when it appears in a camera's view, an AI vision system misreads the scene — seeing a toaster, say, or missing a person., 3D-printed objects). Proposed explanations include locally linear behavior in high dimensions and reliance on predictive but non-robust features. Adversarial prompts against language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. are the textual analogue (see jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.).
A perturbation of 4/255 per pixel took the undefended classifier from 94% accuracy to under 1%.
Origin
Introduced in its current sense by Christian Szegedy and colleagues in 'Intriguing properties of neural networks' (arXiv December 2013; ICLR 2014), which showed imperceptible perturbations fooling image classifiers. Battista Biggio and colleagues independently described gradient-based evasion of classifiers in 2013. Ian Goodfellow and colleagues' 2014 follow-up, with its panda-to-gibbon illustration, made the phenomenon famous.
Usage note
'Adversarial' means deliberately constructed; the term is unrelated to GANsgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people.. Inputs that fool models without any optimization ('natural adversarial examples') and ordinary out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable. failures are related but distinct.
Related
- evasion attackevasion attack, n.: An attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through.
- adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
- adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them.
- fast gradient sign methodfast gradient sign method, n.: A quick way to make an image that fools a neural network: nudge every pixel slightly in whichever direction most increases the network's error.
- adversarial patchadversarial patch, n.: A printed sticker or picture designed so that, when it appears in a camera's view, an AI vision system misreads the scene — seeing a toaster, say, or missing a person.
- universal adversarial perturbationuniversal adversarial perturbation, n.: A single faint pattern that, added to almost any image, makes a neural network misclassify it — one key that opens many locks.
- adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up.
- out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable.
Sources
- Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. & Fergus, R. (2014). Intriguing properties of neural networks. ICLR 2014. arXiv:1312.6199.
- Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B. & Madry, A. (2019). Adversarial Examples Are Not Bugs, They Are Features. NeurIPS 2019.
Mentioned in
AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable., adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., adversarial patchadversarial patch, n.: A printed sticker or picture designed so that, when it appears in a camera's view, an AI vision system misreads the scene — seeing a toaster, say, or missing a person., adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up., adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them., automated red teamingautomated red teaming, n.: Using AI programs, not just human testers, to invent huge numbers of tricky inputs that try to make another AI misbehave, so problems can be found and fixed at scale., CAPTCHACAPTCHA, n.: A small test on websites, like typing wavy letters or clicking every picture with a traffic light, meant to be easy for people but hard for computer programs, so as to keep automated bots out., decision boundarydecision boundary, n.: The dividing line, or surface, where a classifier switches from predicting one category to another. Simple models draw straight lines; flexible ones draw curves., evasion attackevasion attack, n.: An attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through., exampleexample, n.: A single item of data used to train or test an AI model, such as one photo with its label, one email marked spam or not, or one question with its answer., formal verificationformal verification, n.: Using mathematics, not just testing, to prove that a computer system always behaves as specified — for example, that a chip never adds wrongly or that a program can never crash in a particular way., generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., Ian GoodfellowIan Goodfellow, prop. n.: An American computer scientist who invented generative adversarial networks, in which two neural networks compete to produce realistic fake images, and who pioneered research on fooling AI systems., out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable., perceptionperception, n.: How an AI system or robot turns raw sensor signals, such as camera pixels, microphone sound or laser scans, into an understanding of what is around it: objects, people, words and distances., surrogate modelsurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply., attack success rateattack success rate, n.: The share of attempts in which an attack works — for example, the percentage of forbidden requests a model ends up answering when it is attacked., black-box attackblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel., Carlini–Wagner attackCarlini–Wagner attack, n.: A careful, optimization-based way of finding the smallest change to an input that fools a neural network; it broke several defenses that had looked secure., certified robustnesscertified robustness, n.: A mathematical guarantee — not just a test result — that no change to an input within a set limit can alter a model's answer., clean-label poisoningclean-label poisoning, n.: A sneaky form of data poisoning in which every planted training example is labeled correctly, so reviewers see nothing wrong, yet the examples are crafted to make the model fail., fast gradient sign methodfast gradient sign method, n.: A quick way to make an image that fools a neural network: nudge every pixel slightly in whichever direction most increases the network's error., GlazeGlaze, prop. n.: A free tool that lets artists treat their pictures before posting them, with changes people barely notice, so that an AI trained to imitate their style learns a different one.