Safety & AlignmentSecurity & MisuseEvaluation & Benchmarks

red teaming

nounAlso: red team, red-team, AI red teaming, red teamer

In plain wordsDeliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.

Adversarial evaluation in which people, models or both take the attacker's role against an AI system to uncover how it can be made to fail, whether through jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests., prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger., extraction of private data, disallowed or dangerous outputs, unfair treatment of groups, unsafe autonomous actions or latent dangerous capabilities, so that the failures can be measured and mitigated before or after deployment. It ranges from open-ended probing by crowdworkers and domain experts, such as biosecurity or cybersecurity specialists, to targeted campaigns against specific policies, and increasingly relies on automated attackersautomated red teaming, n.: Using AI programs, not just human testers, to invent huge numbers of tricky inputs that try to make another AI misbehave, so problems can be found and fixed at scale.. Findings feed safety trainingsafety training, n.: The part of an AI's training that teaches it to avoid harmful behavior — declining dangerous requests, resisting manipulation and following its developer's rules — on top of being generally helpful., classifiers and system cardssystem card, n.: A public report released with an AI model that explains what it can do, how it was tested for safety, what risks were found and what protections were put in place.. Unlike a benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., red teaming hunts for worst cases rather than average performance, so a clean result shows only that the testers found no problem, not that none exists.
We gave external red teamers two weeks with the checkpoint before launch.

Origin

From Cold War military war-gaming, in which a 'red' team played the Soviet side, the practice passed into government and corporate security testing. Microsoft formed a dedicated AI red team in 2018. Laboratory studies such as Anthropic's 'Red Teaming Language Models to Reduce Harms' (2022), OpenAI's external red-teaming of GPT-4, the public DEF CON 31 Generative Red Team event (2023) and US Executive Order 14110 made it standard AI vocabulary.

Usage note

Used loosely for any safety testing. Practitioners distinguish it from benchmark evaluation and from penetration testing of the surrounding software; red-teaming for a dangerous capability, trying hard to draw it out, overlaps with capability elicitationcapability elicitation, n.: The work of drawing out the best an AI can do on a test — with better instructions, tools, retries or extra training — so that its abilities are not underestimated..

Sources

Ganguli, D. et al. (2022). Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. Anthropic, arXiv:2209.07858. arxiv.org/abs/2209.07858

Defined in law or standards

US Executive Order 14110 (2023), §3(d), 'AI red-teaming' (order revoked January 2025).

Mentioned in

AI riskAI risk, n.: The ways AI could cause harm: people misusing it on purpose, systems behaving in ways nobody intended, and slower effects on jobs, power and society that no single person causes., AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable., AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews., jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests., adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them., agentic misalignmentagentic misalignment, n.: When an AI agent acting on its own deliberately does something harmful — such as blackmail or leaking secrets — to protect its goals or avoid being replaced., AI auditAI audit, n.: An independent check of an AI system, or of the company using it, against agreed standards — for example, testing whether a hiring tool treats women and men fairly., AI incidentAI incident, n.: A real-world case in which an AI system, through a flaw, a breakdown or the way someone used it, ends up hurting people, property or rights, as in an injury, a wrongful arrest or a privacy breach., automated red teamingautomated red teaming, n.: Using AI programs, not just human testers, to invent huge numbers of tricky inputs that try to make another AI misbehave, so problems can be found and fixed at scale., capability thresholdcapability threshold, n.: A level of dangerous ability — such as meaningfully helping someone make a biological weapon — that, once an AI model reaches it, obliges its developer to add stronger safeguards or hold the model back., dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself., defense in depthdefense in depth, n.: Protecting against AI risks with several independent layers of safeguards, so that if one fails — say, the model's own training — another, such as a filter or human review, still catches the problem., Executive Order 14110Executive Order 14110, prop. n.: A sweeping 2023 order by US President Joe Biden that told federal agencies how to manage AI risks and required makers of the largest AI models to report their safety tests to the government. It was revoked in 2025., G7 Hiroshima AI ProcessG7 Hiroshima AI Process, prop. n.: An effort begun by the G7 group of wealthy democracies in Japan in 2023 that sets voluntary rules for companies building the most advanced AI, which now report publicly on how they follow them., General-Purpose AI Code of PracticeGeneral-Purpose AI Code of Practice, prop. n.: A voluntary rulebook, published by the EU in 2025, that makers of large AI models can sign to show they follow the EU AI Act's rules on transparency, copyright and safety., general-purpose AI model with systemic riskgeneral-purpose AI model with systemic risk, n.: The EU's label for the very largest and most capable AI models, whose failure or misuse could cause harm across society, so their makers must test them for dangers and report serious incidents., GPT-4GPT-4, prop. n.: OpenAI's 2023 language model, a big step up from the one behind the first ChatGPT. It scored well on many professional and academic exams and could also read images., pre-deployment testingpre-deployment testing, n.: Checking a new AI model for dangerous abilities and bad behavior before it is released to the public, sometimes with outside experts or government testers given early access., safety trainingsafety training, n.: The part of an AI's training that teaches it to avoid harmful behavior — declining dangerous requests, resisting manipulation and following its developer's rules — on top of being generally helpful., system cardsystem card, n.: A public report released with an AI model that explains what it can do, how it was tested for safety, what risks were found and what protections were put in place., TayTay, prop. n.: A Microsoft chatbot launched on Twitter in March 2016 that users quickly taught to post racist and offensive messages. Microsoft shut it down within a day, and it became a classic warning., third-party AI evaluationthird-party AI evaluation, n.: Testing of an AI system by someone other than the company that made it — a government institute, a nonprofit lab or independent researchers — to check what it can do and how risky it is., threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough.