weight exfiltration
In plain wordsCopying an AI model's core files out of the place they are supposed to stay — by hackers, by an insider or, in a scenario researchers worry about, by the AI itself.
The unauthorized copying of a model's weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. out of the environment where they are meant to remain, whether by external attackers (intrusion, supply-chain compromise), insiders, partners with legitimate access, or — in a scenario studied in AI safety — the model itself (self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.). Exfiltrated weights let the holder run the model without its developer's safeguards, monitoring or usage policies, strip safety training through a fine-tuning attackfine-tuning attack, n.: Removing an AI model's safety training by training it a little more on a small set of harmful examples — cheap to do when the model's files are public or a company offers custom training., and study it for white-box attackswhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots.; once copies spread, the theft cannot be undone. Because frontier weights occupy hundreds of gigabytes or more, a characteristic defense caps outbound bandwidth ('egress') from systems that hold them, so that copying would take long enough to be detected, alongside the broader controls described under model weight securitymodel weight security, n.: Protecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model..
The red team demonstrated weight exfiltration through a misconfigured storage bucket used for evaluations.
Origin
Discussed in security and AI-safety writing from about 2022. The March 2023 leak of Meta's LLaMA weights, distributed to approved researchers and posted publicly within about a week, was an early high-profile case; frontier safety frameworks and RAND's 2024 weight-security report then made exfiltration a central threat model.
Usage note
Distinct from model extractionmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made., which approximates a model through its outputs without access to the weights themselves.
Related
- model weight securitymodel weight security, n.: Protecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model.
- self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.
- model extractionmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made.
- open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.
- structured accessstructured access, n.: Letting people use an AI system through a controlled channel, such as a website or programming interface, instead of handing over the model itself, so that the developer can limit misuse.
- confidential computingconfidential computing, n.: Running computations inside a hardware-sealed compartment of a chip, so that data and AI models stay protected even from the cloud company or administrator operating the machine.
- model extractionmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made.
Sources
- Nevo, S. et al. (2024). Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models. RAND Corporation.
Mentioned in
AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., model extractionmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made., model weight securitymodel weight security, n.: Protecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model., weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself., model fingerprintingmodel fingerprinting, n.: Ways to recognize a particular AI model — to prove that a copy was taken from it, or to tell which model is behind a product — from its internal makeup or its behavior., self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.