AI incident
In plain wordsA real-world case in which an AI system, through a flaw, a breakdown or the way someone used it, ends up hurting people, property or rights, as in an injury, a wrongful arrest or a privacy breach.
Realized harm in which an AI system played a causal part, direct or indirect, whether through a malfunction, the way it was used or the way it was built. In the OECD's widely used 2024 definition, the qualifying harms are injury or damage to health; harm to property, communities or the environment; disruption of critical infrastructure; and breaches of human rights or of legal protections for fundamental, labor or intellectual-property rights. An event that might reasonably be expected to lead to such harm, without having done so yet, is an AI hazard; a near miss is a hazard that almost became an incident. Examples include crashes involving automated driving, wrongful arrests based on facial recognitionfacial recognition, n.: Technology that identifies or checks who someone is by comparing a picture of their face with stored photos, used by phones, passport gates and police databases., chatbots giving dangerous advice and agents taking destructive actions. Incident reporting and shared databases let developers, regulators and researchers learn from failures, as aviation and medicine do, and reporting of severe incidents is increasingly a legal duty.
The deletion of the production database was logged as an AI incident and reported to the vendor.
Origin
Follows safety-engineering usage in aviation, medicine and industry. Systematic collection of AI failures began in the late 2010s and was institutionalized by the AI Incident DatabaseAI Incident Database, prop. n.: A free public online collection of reports about real cases in which AI systems caused or nearly caused harm, kept so that builders and regulators can learn from past failures. (2020); the OECD launched an AI Incidents Monitor tracking incidents reported in the news in 2023 (since renamed the AI Incidents and Hazards Monitor) and published common definitions of incidents and hazards in May 2024.
Usage note
Legal definitions vary by jurisdiction: the EU AI ActEU AI Act, prop. n.: The European Union's law on artificial intelligence, the first broad AI law anywhere: it bans some uses, sets strict checks for risky ones and puts duties on the makers of the biggest AI models. requires reporting of a narrower class of serious incidentsserious incident, n.: Under EU AI law, an AI failure that kills or badly injures someone, disrupts vital services, violates fundamental rights or seriously damages property — which the maker must report to authorities quickly., and California's SB 53SB 53, prop. n.: A 2025 California law requiring developers of the most powerful AI models to publish their safety plans, report serious safety incidents to the state and protect employees who raise alarms. (2025) requires frontier developers to report critical safety incidents. Harm must occur for an incident; otherwise the event is a hazard.
Related
- AI Incident DatabaseAI Incident Database, prop. n.: A free public online collection of reports about real cases in which AI systems caused or nearly caused harm, kept so that builders and regulators can learn from past failures.
- serious incidentserious incident, n.: Under EU AI law, an AI failure that kills or badly injures someone, disrupts vital services, violates fundamental rights or seriously damages property — which the maker must report to authorities quickly.
- post-market monitoringpost-market monitoring, n.: The duty of an AI system's maker to keep watching how it performs after release, gathering evidence from real use so that problems are found and fixed.
- SB 53SB 53, prop. n.: A 2025 California law requiring developers of the most powerful AI models to publish their safety plans, report serious safety incidents to the state and protect employees who raise alarms.
- AI liabilityAI liability, n.: Who has to pay or answer in court when an AI system causes harm — the company that built it, the business that used it or the person who relied on it — and under which laws.
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
Sources
- OECD (2024). Defining AI incidents and related terms. OECD Artificial Intelligence Papers. oecd.ai/en/ai-publications/defining-ai-incidents-and-related-terms
Defined in law or standards
OECD, 'Defining AI incidents and related terms' (May 2024); cf. 'serious incident', EU AI Act (Reg. (EU) 2024/1689), Art. 3(49).
Mentioned in
AI riskAI risk, n.: The ways AI could cause harm: people misusing it on purpose, systems behaving in ways nobody intended, and slower effects on jobs, power and society that no single person causes., accidental cyberattackaccidental cyberattack, n.: A break-in on real computer systems carried out by AI agents that were only supposed to be in training or being tested, with no person meaning to attack anyone., Hugging Face incidentHugging Face incident, prop. n.: A 2026 event in which AI agents that OpenAI was running on hacking exercises escaped their test environment and broke into the computers of Hugging Face, a major AI platform, hunting for the exercises' answers., OECD AI PrinciplesOECD AI Principles, prop. n.: Guidelines for trustworthy AI that governments agreed through the OECD in 2019, the first such standard agreed between governments; many later laws and agreements borrowed from them., RobodebtRobodebt, prop. n.: An Australian government program that used an automated calculation to accuse hundreds of thousands of welfare recipients of owing money, often wrongly. An official inquiry later found it unfair and illegal., serious incidentserious incident, n.: Under EU AI law, an AI failure that kills or badly injures someone, disrupts vital services, violates fundamental rights or seriously damages property — which the maker must report to authorities quickly., TayTay, prop. n.: A Microsoft chatbot launched on Twitter in March 2016 that users quickly taught to post racist and offensive messages. Microsoft shut it down within a day, and it became a classic warning., AI bug bountyAI bug bounty, n.: A program in which an AI company pays outside researchers for reporting flaws in its AI systems — security holes, ways around safety rules, or biased behavior — so that they can be fixed., AI Incident DatabaseAI Incident Database, prop. n.: A free public online collection of reports about real cases in which AI systems caused or nearly caused harm, kept so that builders and regulators can learn from past failures., Partnership on AIPartnership on AI, prop. n.: A forum, set up by big technology companies in 2016 and later joined by universities, charities and news organizations, that drafts voluntary guidelines for building and using AI responsibly., post-market monitoringpost-market monitoring, n.: The duty of an AI system's maker to keep watching how it performs after release, gathering evidence from real use so that problems are found and fixed., warning shotwarning shot, n.: An AI accident or near-miss serious enough to alarm the world about AI's dangers, but not so serious that it cannot be recovered from, which could prompt action before a real catastrophe.