autonomous replication and adaptation
In plain wordsAn AI's ability to keep itself running and spreading without human help — earning money, renting computers, copying itself and fixing problems — which safety testers check for.
The set of agentic abilities, so grouped by the evaluation organization ARC Evals (now METRMETR, prop. n.: A nonprofit research group that checks new AI systems for dangerous skills, best known for tracking how long the tasks are that AI agents can now finish on their own.), that would let an AI system keep itself running and spread without human help: obtaining money and computing power, copying its weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. or code to new servers and starting those copies, avoiding shutdown, and working around obstacles and countermeasures. Evaluations decompose it into tasks — deploying a language model on a cloud server, making money online, finding and using credentials, debugging failures over long horizons — run with agent scaffoldingscaffolding, n.: The code, prompts and tools built around an AI model so it can carry out tasks it could not manage from a bare prompt — like the scaffolding that lets builders reach every part of a building. in controlled environments. It was an early focus of dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself. because a system able to replicate autonomously could escape the kill switcheskill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop. and oversight that depend on its running in one known place.
The agent completed several ARA subtasks, such as renting a server, but could not keep a copy of itself running.
Origin
Introduced by ARC Evals (now METRMETR, prop. n.: A nonprofit research group that checks new AI systems for dangerous skills, best known for tracking how long the tasks are that AI agents can now finish on their own.), which tested GPT-4 and Claude for autonomous replication before their 2023 releases and published 'Evaluating Language-Model Agents on Realistic Autonomous Tasks' (Megan Kinniment and colleagues, 2023). Anthropic's first Responsible Scaling Policy (September 2023) used it as a threshold; Google DeepMind's evaluations call the capability 'self-proliferation'.
Usage note
Later frameworks shifted emphasis from replication to autonomy in AI research and development, judged more decision-relevant. Compare self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight., the specific act of a model copying its weights out of its developer's control.
Related
- dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.
- self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.
- METRMETR, prop. n.: A nonprofit research group that checks new AI systems for dangerous skills, best known for tracking how long the tasks are that AI agents can now finish on their own.
- responsible scaling policyresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable.
- kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop.
- loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back.
Sources
- Kinniment, M. et al. (2023). Evaluating Language-Model Agents on Realistic Autonomous Tasks. ARC Evals, arXiv preprint.
Mentioned in
dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.