Skip to content
Last Dictionary
Plane
Security & MisuseDeep Learning

Carlini–Wagner attack

nounAlso: C&W attack, CW attack

In plain wordsA careful, optimization-based way of finding the smallest change to an input that fools a neural network; it broke several defenses that had looked secure.

  1. A family of white-boxwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots. attacks that find minimal-distortion adversarial examplesadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon. under ℓ0, ℓ2 or ℓ∞ distance by solving an optimization problem: minimize the size of the perturbation plus a constant times a margin loss on the logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. that stops rewarding progress once the target class leads by a confidence margin κ. A change of variables keeps pixel values in range, and binary search tunes the trade-off constant; a larger κ yields examples that transfer better. The ℓ2 version became a standard benchmark attack, and its introduction showed that defensive distillation, then a leading defense, gave essentially no robustness.

    Defensive distillation held up against FGSM but collapsed under the Carlini–Wagner attack.

min⁡δ  ∥δ∥22+c⋅max⁡(max⁡i≠tZ(x+δ)i−Z(x+δ)t,  −κ)\min_{\delta} \; \|\delta\|_2^2 + c \cdot \max\Big(\max_{i \ne t} Z(x+\delta)_i - Z(x+\delta)_t, \; -\kappa\Big)

Origin

Introduced by Nicholas Carlini and David Wagner of the University of California, Berkeley, in 'Towards Evaluating the Robustness of Neural Networks' (arXiv August 2016; IEEE Symposium on Security and Privacy 2017).

Related

See also
  • adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon.
  • projected gradient descent attackprojected gradient descent attack, n.: A stronger version of the quick pixel-nudging attack: take many small steps that each raise the model's error, pulling the image back after every step so the total change stays tiny.
  • fast gradient sign methodfast gradient sign method, n.: A quick way to make an image that fools a neural network: nudge every pixel slightly in whichever direction most increases the network's error.
  • gradient maskinggradient masking, n.: When a defense against attacks only seems to work because it confuses the attacker's tools, not because the model is really any harder to fool.
  • adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.

Sources

  1. Carlini, N. & Wagner, D. (2017). Towards Evaluating the Robustness of Neural Networks. IEEE S&P 2017. arXiv:1608.04644.

Mentioned in

adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon., projected gradient descent attackprojected gradient descent attack, n.: A stronger version of the quick pixel-nudging attack: take many small steps that each raise the model's error, pulling the image back after every step so the total change stays tiny., white-box attackwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots.