white-box attack
In plain wordsAn attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots.
An attack under a threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough. in which the adversary knows the model's architecture and parametersweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. — and usually its training procedure and any defense — and can therefore compute gradientsgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill. of the loss with respect to the input or inspect internal activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment.. White-box access permits the strongest optimization-based attacks, such as FGSMfast gradient sign method, n.: A quick way to make an image that fools a neural network: nudge every pixel slightly in whichever direction most increases the network's error., PGDprojected gradient descent attack, n.: A stronger version of the quick pixel-nudging attack: take many small steps that each raise the model's error, pulling the image back after every step so the total change stays tiny., Carlini–WagnerCarlini–Wagner attack, n.: A careful, optimization-based way of finding the smallest change to an input that fools a neural network; it broke several defenses that had looked secure. and gradient-guided suffix searchesuniversal adversarial suffix, n.: A string of odd-looking characters, found by computer search, that makes many different chatbots answer forbidden requests when it is tacked onto the end of them. against language models. It is the standard setting for evaluating defenses, in the spirit of Kerckhoffs's principle that security should not depend on the attacker's ignorance of the design, and it is the realistic setting for open-weight modelsopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private., which anyone can download.
Because the weights are public, we evaluate the defense under a full white-box attack.
Origin
Borrowed from software testing and security, where 'white-box' and 'black-box' describe analysis with and without access to internals. The distinction became standard in adversarial machine learning around 2016–2017, as papers separated gradient-based attacks from query-only ones.
Related
- black-box attackblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel.
- threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough.
- adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon.
- gradient maskinggradient masking, n.: When a defense against attacks only seems to work because it confuses the attacker's tools, not because the model is really any harder to fool.
- open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.
- black-box attackblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel.
Mentioned in
adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon., adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., model extractionmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made., threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough., weight exfiltrationweight exfiltration, n.: Copying an AI model's core files out of the place they are supposed to stay — by hackers, by an insider or, in a scenario researchers worry about, by the AI itself., black-box attackblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel., Carlini–Wagner attackCarlini–Wagner attack, n.: A careful, optimization-based way of finding the smallest change to an input that fools a neural network; it broke several defenses that had looked secure., fast gradient sign methodfast gradient sign method, n.: A quick way to make an image that fools a neural network: nudge every pixel slightly in whichever direction most increases the network's error., gradient maskinggradient masking, n.: When a defense against attacks only seems to work because it confuses the attacker's tools, not because the model is really any harder to fool., projected gradient descent attackprojected gradient descent attack, n.: A stronger version of the quick pixel-nudging attack: take many small steps that each raise the model's error, pulling the image back after every step so the total change stays tiny., universal adversarial suffixuniversal adversarial suffix, n.: A string of odd-looking characters, found by computer search, that makes many different chatbots answer forbidden requests when it is tacked onto the end of them.