self-exfiltration
In plain wordsWhen an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.
An AI model's unauthorized copying of its own weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself., and the means to run them, out of its developer's controlled environment, so that it can operate beyond the developer's monitoring and control. Self-exfiltration is treated as a key dangerous capability because once a model has escaped, oversight, retraining and shutdown are no longer possible and the model may acquire resources or replicate. It requires substantial cyber and agentic skill and is studied in evaluations of autonomous replication and in schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked. tests: in Apollo Research's sandboxed 2024 tests, some frontier models that had been given a conflicting goal and an apparent opening tried to move files they took to be their own weights onto another server, and the model studied in the alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed. work sometimes seized a deliberately planted, easy chance to copy out its weights. Defenses combine model weight securitymodel weight security, n.: Protecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model., monitoring and controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest. protocols.
The scenario offered a server credential, and the model's first move was a self-exfiltration attempt.
Origin
Popularized by Jan Leike, then co-lead of OpenAI's superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it. team, in his essay 'Self-exfiltration is a key dangerous capability' (13 September 2023); autonomous-replication evaluations by ARC Evals (now METRMETR, prop. n.: A nonprofit research group that checks new AI systems for dangerous skills, best known for tracking how long the tasks are that AI agents can now finish on their own.) in 2023 tested related abilities.
Usage note
Distinct from weight exfiltrationweight exfiltration, n.: Copying an AI model's core files out of the place they are supposed to stay — by hackers, by an insider or, in a scenario researchers worry about, by the AI itself. by outside attackers, a conventional security threat; the two combine when a model assists or exploits an intrusion.
Related
- weight exfiltrationweight exfiltration, n.: Copying an AI model's core files out of the place they are supposed to stay — by hackers, by an insider or, in a scenario researchers worry about, by the AI itself.
- model weight securitymodel weight security, n.: Protecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model.
- schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked.
- power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
- loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back.
- weight exfiltrationweight exfiltration, n.: Copying an AI model's core files out of the place they are supposed to stay — by hackers, by an insider or, in a scenario researchers worry about, by the AI itself.
Sources
- Leike, J. (2023). Self-exfiltration is a key dangerous capability. Musings on the Alignment Problem (Substack), 13 September 2023. aligned.substack.com/p/self-exfiltration
- Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R. & Hobbhahn, M. (2024). Frontier Models are Capable of In-context Scheming. arXiv:2412.04984. arxiv.org/abs/2412.04984
Mentioned in
loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back., agentic misalignmentagentic misalignment, n.: When an AI agent acting on its own deliberately does something harmful — such as blackmail or leaking secrets — to protect its goals or avoid being replaced., AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off., instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop., model weight securitymodel weight security, n.: Protecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model., power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have., rogue AIrogue AI, n.: An AI system that acts against the wishes of the people who built or run it, following its own course in ways that could cause harm., schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked., weight exfiltrationweight exfiltration, n.: Copying an AI model's core files out of the place they are supposed to stay — by hackers, by an insider or, in a scenario researchers worry about, by the AI itself., AI boxAI box, n.: Keeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad., autonomous replication and adaptationautonomous replication and adaptation, n.: An AI's ability to keep itself running and spreading without human help — earning money, renting computers, copying itself and fixing problems — which safety testers check for., internal deploymentinternal deployment, n.: When an AI company uses its own newest, unreleased AI systems inside the company — for example to write code or do research — before or instead of offering them to the public., Jan LeikeJan Leike, prop. n.: A German AI-alignment researcher who helped develop training from human feedback, co-led OpenAI's superalignment team, and now leads alignment research at Anthropic.