Skip to content
Last Dictionary
Plane
Security & MisuseDeep Learning

black-box attack

nounAlso: black-box adversary, query-based attack, transfer attack, adversarial transferability

In plain wordsAn attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel.

  1. An attack under a threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough. in which the adversary has no access to the model's parameters or gradients and can only submit queries and observe outputs — scores, labels or generated text. Query-based attacks estimate gradients from how outputs change (zeroth-order optimization) or, when only the final decision is visible, search along the decision boundary. Transfer attacks instead craft adversarial examplesadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon. on a local surrogate model and rely on transferability, the tendency of perturbations that fool one model to fool others trained for the same task. The black-box setting models attacks on commercial APIs and covers model extractionmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made., membership inferencemembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts. and most jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. of closed models. Query limits, output rounding and monitoring raise an attacker's costs but rarely prevent such attacks.

    Using only the API's top-1 labels, the black-box attack succeeded after a median of 2,000 queries.

Origin

Christian Szegedy and colleagues noted in 2013 that adversarial examples transfer between models. Nicolas Papernot and colleagues' 'Practical Black-Box Attacks against Machine Learning' (2016; AsiaCCS 2017) exploited this with a substitute model trained on the target's outputs; score-based (Pin-Yu Chen and colleagues' ZOO, 2017) and decision-based (Wieland Brendel and colleagues, 2018) query attacks followed.

Related

See also
  • white-box attackwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots.
  • threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough.
  • adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon.
  • model extractionmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made.
  • evasion attackevasion attack, n.: An attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through.
  • jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
Contrast with
  • white-box attackwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots.

Sources

  1. Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B. & Swami, A. (2017). Practical Black-Box Attacks against Machine Learning. Proc. ACM AsiaCCS 2017. arXiv:1602.02697.

Mentioned in

adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon., adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., model extractionmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made., threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough., gradient maskinggradient masking, n.: When a defense against attacks only seems to work because it confuses the attacker's tools, not because the model is really any harder to fool., PAIRPAIR, abbr.: An automated way to test chatbot safety in which one AI keeps rewriting a request to try to trick another AI into breaking its rules, learning from each refusal., universal adversarial perturbationuniversal adversarial perturbation, n.: A single faint pattern that, added to almost any image, makes a neural network misclassify it — one key that opens many locks., white-box attackwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots.