model extraction
In plain wordsCopying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made.
An attack in which an adversary with query access to a deployed model reconstructs it, or something functionally close, without access to its weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself.. Goals range from exact recovery of parameters — feasible for small or partly linear models, and for the final projection layer of some production language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. when logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity.-derived outputs were exposed — to functional copying, in which the attacker trains a substitute on the target's input–output pairs, in effect unauthorized knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.. Extracted models enable theft of intellectual property, cheaper white-boxwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots. attacks on the original, and copies stripped of its safeguards. Defenses include rate limits, withholding probabilities, detecting suspicious query patterns, account verification, terms of service, and fingerprintingmodel fingerprinting, n.: Ways to recognize a particular AI model — to prove that a copy was taken from it, or to tell which model is behind a product — from its internal makeup or its behavior. or watermarking to prove copying.
The provider suspected model extraction when a cluster of new accounts sent millions of systematically varied prompts in a week.
Origin
Florian Tramèr and colleagues' 'Stealing Machine Learning Models via Prediction APIs' (USENIX Security 2016) introduced practical extraction attacks on commercial prediction APIs. Nicholas Carlini and colleagues' 'Stealing Part of a Production Language Model' (ICML 2024 best paper) recovered, through OpenAI's public API, the final projection matrix of two older models and the hidden width of gpt-3.5-turbo. In February 2026 Anthropic accused three Chinese laboratories (DeepSeek, Moonshot AI and MiniMax) of 'distillation attacks' using about 24,000 fraudulent accounts and over 16 million exchanges with its Claude models.
Usage note
Training on another model's outputs is common and often legitimate; it becomes extraction when it breaches terms of service, access controls or intellectual-property rights. The legal status of such copying varies by jurisdiction and is largely untested.
Related
- knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
- black-box attackblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel.
- model fingerprintingmodel fingerprinting, n.: Ways to recognize a particular AI model — to prove that a copy was taken from it, or to tell which model is behind a product — from its internal makeup or its behavior.
- weight exfiltrationweight exfiltration, n.: Copying an AI model's core files out of the place they are supposed to stay — by hackers, by an insider or, in a scenario researchers worry about, by the AI itself.
- model weight securitymodel weight security, n.: Protecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model.
- rate limitrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later.
Sources
- Tramèr, F., Zhang, F., Juels, A., Reiter, M. K. & Ristenpart, T. (2016). Stealing Machine Learning Models via Prediction APIs. USENIX Security 2016.
- Carlini, N. et al. (2024). Stealing Part of a Production Language Model. ICML 2024.
- Anthropic (2026). Detecting and preventing distillation attacks. Anthropic News, February 2026. anthropic.com/news/detecting-and-preventing-distillation-attacks
Mentioned in
AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., model weight securitymodel weight security, n.: Protecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model., surrogate modelsurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply., weight exfiltrationweight exfiltration, n.: Copying an AI model's core files out of the place they are supposed to stay — by hackers, by an insider or, in a scenario researchers worry about, by the AI itself., black-box attackblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel., MITRE ATLASMITRE ATLAS, prop. n.: A free online reference from the US nonprofit MITRE that maps, step by step, how attackers go after AI systems, with examples drawn from real attacks and security tests., model fingerprintingmodel fingerprinting, n.: Ways to recognize a particular AI model — to prove that a copy was taken from it, or to tell which model is behind a product — from its internal makeup or its behavior.