existential risk
In plain wordsA danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
A risk that threatens either the end of humanity or the irreversible loss of most of what its long-term future could have been, for example through an unrecoverable collapse of civilization or the permanent entrenchment of bad values or rulers (value lock-invalue lock-in, n.: The possibility that powerful AI could freeze one set of values or one power structure in place for an extremely long time, so that later generations could never change course.). Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.'s standard 2002 definition covers outcomes that would 'permanently and drastically curtail' humanity's potential as well as extinction. Irreversibility distinguishes it from a global catastrophic riskglobal catastrophic risk, n.: A risk of disaster big enough to badly damage human civilization worldwide, such as killing a large share of humanity, even if humanity eventually recovers., from which humanity might recover. The category spans natural risks (asteroid impact, supervolcanic eruption) and human-made ones (nuclear war, engineered pandemics, advanced AI). Toby Ord's The Precipice (2020) estimated total existential risk this century at about one in six and put unaligned AI, at about one in ten, as the largest single source. AI existential risk is usually argued from the prospect of superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist. with goals at odds with human interests (AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off., loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back.), but also from AI-enabled war, engineered pandemics, totalitarian lock-in and gradual disempowermentgradual disempowerment, n.: The risk that, as AI does more and more of the work people now do, humanity slowly loses its grip on business, government and culture, until it no longer steers its own future.. The one-sentence 2023 Statement on AI RiskStatement on AI Risk, prop. n.: A one-sentence public statement from May 2023, signed by hundreds of leading AI scientists and company heads, declaring that guarding against the chance of AI wiping out humanity should rank as a priority for the whole world. ranked extinction risk from AI with pandemics and nuclear war as a global priority. Critics argue that the case rests on speculative chains of reasoning, distracts from present harms such as bias and labor exploitation, or serves the interests of AI companies.
The report separates existential risk from ordinary catastrophic risk by asking whether recovery is possible.
Origin
Introduced by Nick Bostrom in 'Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards' (Journal of Evolution and Technology, 2002), building on earlier work on human extinction by philosophers including Derek Parfit and John Leslie. The abbreviation x-risk spread in effective-altruist and rationalist circles; AI existential risk became a mainstream policy topic in 2023.
Usage note
In policy writing 'existential risk' is sometimes used loosely for any very large harm; strictly, the loss must be permanent. Estimates of AI existential risk are often summarized as p(doom)p(doom), n.: Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain..
Related
- global catastrophic riskglobal catastrophic risk, n.: A risk of disaster big enough to badly damage human civilization worldwide, such as killing a large share of humanity, even if humanity eventually recovers.
- suffering risksuffering risk, n.: A risk that the future contains suffering on a vast scale, far beyond anything in history, for example if powerful AI created enormous numbers of suffering beings.
- p(doom)p(doom), n.: Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain.
- AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off.
- longtermismlongtermism, n.: The belief that we should care a great deal about how things go for humanity thousands or millions of years from now, since far more people may live then than have ever lived.
- Statement on AI RiskStatement on AI Risk, prop. n.: A one-sentence public statement from May 2023, signed by hundreds of leading AI scientists and company heads, declaring that guarding against the chance of AI wiping out humanity should rank as a priority for the whole world.
- Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.
- doomerdoomer, n.: Someone who believes advanced AI is likely to cause a catastrophe, possibly human extinction; often used as a put-down by people who think those fears are exaggerated.
Sources
- Bostrom, N. (2002). Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards. Journal of Evolution and Technology 9.
- Ord, T. (2020). The Precipice: Existential Risk and the Future of Humanity. Bloomsbury.
Mentioned in
AI riskAI risk, n.: The ways AI could cause harm: people misusing it on purpose, systems behaving in ways nobody intended, and slower effects on jobs, power and society that no single person causes., AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable., alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different., doomerdoomer, n.: Someone who believes advanced AI is likely to cause a catastrophe, possibly human extinction; often used as a put-down by people who think those fears are exaggerated., Elon MuskElon Musk, prop. n.: A South African-born American entrepreneur who runs Tesla and SpaceX, co-founded OpenAI, founded the AI company xAI, and has long warned that AI could endanger humanity., paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous., p(doom)p(doom), n.: Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain., superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist., Yann LeCunYann LeCun, prop. n.: A French-American computer scientist who pioneered the neural networks that recognize images, led Meta's AI research for over a decade, and argues that chatbots alone will not reach human-level AI., AI as normal technologyAI as normal technology, phr.: The view that AI, however important, will change society gradually, like electricity or the internet, through slow adoption by people and institutions, rather than as a sudden, uncontrollable superintelligence., AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off., Bletchley DeclarationBletchley Declaration, prop. n.: A short statement signed by 28 countries and the European Union at the first global AI safety summit, held in England in 2023, agreeing that the most advanced AI could cause serious harm., catastrophic riskcatastrophic risk, n.: The risk that AI could cause disaster on a massive scale — many deaths or enormous damage — through weapons, cyberattacks or systems escaping human control. Several laws now define it precisely., Center for AI SafetyCenter for AI Safety, prop. n.: A San Francisco nonprofit that studies how advanced AI could lead to disaster and how to prevent it. In 2023 it gathered signatures for a one-sentence warning that AI could threaten human extinction., effective altruismeffective altruism, n.: A movement that treats charity like a research problem, comparing causes by how much good each dollar or working hour achieves; many of its members came to rank risks from advanced AI as a top priority., Future of Life InstituteFuture of Life Institute, prop. n.: An advocacy group that warns about the dangers of advanced AI and other powerful technologies, funds research and lobbies governments. It organized the 2023 open letter calling for a pause on giant AI experiments., godfathers of AIgodfathers of AI, phr.: A media nickname for three scientists — Geoffrey Hinton, Yoshua Bengio and Yann LeCun — whose decades of work on neural networks underpins modern AI. They shared computing's top prize in 2019., LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI., longtermismlongtermism, n.: The belief that we should care a great deal about how things go for humanity thousands or millions of years from now, since far more people may live then than have ever lived., Max TegmarkMax Tegmark, prop. n.: A Swedish-American physicist at MIT who co-founded the Future of Life Institute, which organized the 2023 open letter calling for a pause in training the most powerful AI systems., Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern., Pause Giant AI ExperimentsPause Giant AI Experiments, prop. n.: A 2023 open letter, signed by tens of thousands of people including prominent technologists, that asked AI labs to halt the training of any AI more capable than GPT-4 for at least half a year., rationalist communityrationalist community, n.: A loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously., SkynetSkynet, prop. n.: The military computer network in the Terminator films that becomes self-aware and tries to wipe out humanity; now everyday shorthand for the fear of AI turning against people.