adversarial patch
In plain wordsA printed sticker or picture designed so that, when it appears in a camera's view, an AI vision system misreads the scene — seeing a toaster, say, or missing a person.
A localized pattern — a sticker, poster or printed object — optimized so that its presence in a scene causes a vision model to output an attacker-chosen label or to miss a detection. Unlike norm-bounded adversarial examplesadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon., a patch need not be imperceptible, only small or inconspicuous; robustness to position, scale, rotation, lighting and printing is obtained by optimizing over random transformations (expectation over transformation). Patches are the main form of physical-world evasion attackevasion attack, n.: An attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through. on object detectorsobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene., face recognition and driving perception, so safety-critical vision systems are tested against them. Defenses include patch-aware adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them. and certified methods that bound the influence of any small image region.
The adversarial patch on the T-shirt hid its wearer from the pedestrian detector in most frames.
Origin
Named and introduced by Tom Brown, Dandelion Mané, Aurko Roy, Martín Abadi and Justin Gilmer in December 2017, whose 'toaster' patch hijacked ImageNet classifiers. It built on earlier physical attacks: eyeglass frames that fooled face recognition (Mahmood Sharif and colleagues, 2016) and stickers that made classifiers misread stop signs (Kevin Eykholt and colleagues, 2017–2018).
Related
- adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon.
- evasion attackevasion attack, n.: An attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through.
- universal adversarial perturbationuniversal adversarial perturbation, n.: A single faint pattern that, added to almost any image, makes a neural network misclassify it — one key that opens many locks.
- object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
- adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
Sources
- Brown, T. B., Mané, D., Roy, A., Abadi, M. & Gilmer, J. (2017). Adversarial Patch. arXiv:1712.09665.
- Eykholt, K. et al. (2018). Robust Physical-World Attacks on Deep Learning Visual Classification. CVPR 2018.
Mentioned in
adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon., universal adversarial perturbationuniversal adversarial perturbation, n.: A single faint pattern that, added to almost any image, makes a neural network misclassify it — one key that opens many locks.