Virology Capabilities Test
In plain wordsA hard test of practical laboratory virology know-how, used to check whether AI models know enough about working with viruses that they might help someone misuse biology.
A benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. of 322 multimodal, multiple-response questions on practical virology, designed to measure expert, hands-on laboratory knowledge — especially troubleshooting of experimental protocols — that is rarely written down and hard to find online. Virologists with doctorates wrote and vetted the questions, many of which show images of experimental results. The test proved hard for people: in its first evaluation, virologists answering only questions within their own specialties averaged about 22% accuracy, whereas OpenAI's o3 reached about 44%, a score above that of 94% of those experts. Because such tacit knowledge is a bottleneck for biological misuse, the test serves as a proxy for biological riskCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone. in dangerous-capability evaluationsdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself. and in developers' system cardssystem card, n.: A public report released with an AI model that explains what it can do, how it was tested for safety, what risks were found and what protections were put in place.; the questions are not released publicly, which limits both misuse and contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.. A high score shows knowledge, not the ability to carry out an attack, which uplift studiesuplift study, n.: An experiment that measures how much an AI really helps people do a task — often a dangerous one, like planning a biological attack — compared with people using only ordinary resources such as the internet. measure more directly.
The model beat the expert baseline on the Virology Capabilities Test, so the enhanced biosecurity safeguards were switched on.
Origin
Introduced in April 2025 by Jasper Götting, Pedro Medeiros, Jon G. Sanders, Nathaniel Li, Long Phan, Karam Elabd, Lennart Justen, Dan Hendrycks and Seth Donoughe, of SecureBio, the Center for AI Safety and other institutions, in 'Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark'.
Usage note
Distinct from WMDPWMDP, prop. n.: A multiple-choice test of AI knowledge that could help someone make biological, chemical or cyber weapons, built from questions near the dangerous line but not over it, and used to test ways of erasing such knowledge., a larger, public multiple-choice benchmark of hazardous knowledge built partly to measure unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again..
Related
- CBRN riskCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone.
- biosecuritybiosecurity, n.: Protecting people from the deliberate or careless spread of dangerous germs and toxins; in AI, making sure AI tools do not make it easier to create or release biological weapons.
- dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.
- WMDPWMDP, prop. n.: A multiple-choice test of AI knowledge that could help someone make biological, chemical or cyber weapons, built from questions near the dangerous line but not over it, and used to test ways of erasing such knowledge.
- uplift studyuplift study, n.: An experiment that measures how much an AI really helps people do a task — often a dangerous one, like planning a biological attack — compared with people using only ordinary resources such as the internet.
- Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%.
- Center for AI SafetyCenter for AI Safety, prop. n.: A San Francisco nonprofit that studies how advanced AI could lead to disaster and how to prevent it. In 2023 it gathered signatures for a one-sentence warning that AI could threaten human extinction.
Sources
- Götting, J. et al. (2025). Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark. arXiv:2504.16137. arxiv.org/abs/2504.16137