capability
In plain wordsSomething an AI system is able to do, such as writing working code or solving math problems, and how well. Measuring capabilities is how researchers track progress and spot dangerous skills.
What an AI system can do: the range of tasks it can perform, and how well, when suitably prompted, equipped with tools and given resources. Capabilities are measured by benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. and evaluationsevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews., but measured performance is a lower bound, since better prompting, fine-tuning or scaffoldingscaffolding, n.: The code, prompts and tools built around an AI model so it can carry out tasks it could not manage from a bare prompt — like the scaffolding that lets builders reach every part of a building. may reveal latent abilities (capability elicitationcapability elicitation, n.: The work of drawing out the best an AI can do on a test — with better instructions, tools, retries or extra training — so that its abilities are not underestimated.). Researchers distinguish capability (what a system can do) from propensity (what it tends to do by default) and from alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this. (whether it pursues intended goals). In safety and governance, dangerous capabilities, such as meaningful uplift for cyberattacks or CBRNCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone. weapons or autonomous replication, are tested in dangerous capability evaluationsdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself. against capability thresholdscapability threshold, n.: A level of dangerous ability — such as meaningfully helping someone make a biological weapon — that, once an AI model reaches it, obliges its developer to add stronger safeguards or hold the model back. that trigger safeguards. The EU AI Act defines high-impact capabilities by reference to the frontier, as capabilities equal to or beyond those of the most advanced general-purpose AI models yet recorded (Art. 3(64)).
The new model's coding capability jumped, so we reran the cyber-offense evaluations before release.
Origin
General English, from Late Latin capabilis. The AI usage hardened in the safety community of the 2010s, which contrasted 'capabilities' work, making systems more competent, with alignment or safety work. Dangerous capability evaluation became standard practice from 2023, following work such as Toby Shevlane and colleagues' 'Model Evaluation for Extreme Risks' (2023) and frontier developers' published safety frameworks.
Usage note
'Capabilities' as the name of a research agenda ('capabilities researchers') can be pejorative in safety circles. Extrapolating from benchmark scores to broad claims about intelligence is contested, as is whether new capabilities appear abruptly with scale (emergent abilitiesemergent abilities, n.: Skills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured.).
Related
- capability elicitationcapability elicitation, n.: The work of drawing out the best an AI can do on a test — with better instructions, tools, retries or extra training — so that its abilities are not underestimated.
- dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.
- capability thresholdcapability threshold, n.: A level of dangerous ability — such as meaningfully helping someone make a biological weapon — that, once an AI model reaches it, obliges its developer to add stronger safeguards or hold the model back.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- emergent abilitiesemergent abilities, n.: Skills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured.
- sandbaggingsandbagging, n.: When an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- jagged frontierjagged frontier, n.: The uneven edge of what AI can do: it may excel at some hard tasks yet fail at seemingly easier ones, so how hard a task looks is a poor guide to whether AI can handle it.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
Sources
- Shevlane, T. et al. (2023). Model Evaluation for Extreme Risks. arXiv:2305.15324. arxiv.org/abs/2305.15324
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 3(64)–(65). artificialintelligenceact.eu/article/3
Defined in law or standards
EU AI Act (Reg. (EU) 2024/1689), Art. 3(64) ('high-impact capabilities') and Art. 3(65) ('systemic risk')
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., levels of autonomylevels of autonomy, n.: Ladders that grade how independently an AI system acts — from a tool a person fully controls, through suggesting and asking for approval, up to acting on its own while people merely watch., model capacitymodel capacity, n.: How much a model is able to learn: the variety and complexity of patterns it can represent or memorize. Too little and it misses real patterns; very large models can even memorize pure noise., task (machine learning)task (machine learning), n.: The job an AI model is meant to do, such as translating sentences, spotting spam or answering questions, defined by its inputs, its expected outputs and how success is scored.