p(doom)
In plain wordsShorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain.
A person's subjective probability that the development of advanced AI leads to an existentially catastrophic outcome, usually human extinction or permanent, severe disempowerment of humanity. The notation borrows the probability-theory form P(event) and is used both seriously, as a compact way to report a credence, and half-jokingly, as a conversational opener and social signal. Publicly stated figures range from under 1% to over 95%: Geoffrey Hinton has given about 10–20%, Yoshua Bengio about 20% and Eliezer Yudkowsky more than 95%, while skeptics such as Yann LeCunYann LeCun, prop. n.: A French-American computer scientist who pioneered the neural networks that recognize images, led Meta's AI research for over a decade, and argues that chatbots alone will not reach human-level AI. regard the risk as negligible. In the 2023 AI Impacts survey of 2,778 published AI researchers, a question on AI causing 'human extinction or similarly permanent and severe disempowerment' within 100 years drew a median answer of 5%, and between 38% and 51% of respondents, depending on how the question was worded, gave at least a 10% chance to outcomes as bad as human extinction. Critics note that stated figures rarely specify a time frame, what counts as doom, or whether they assume AGI is built, and that probabilities for unprecedented events are hard to calibrate or check.
Her p(doom) dropped from 30% to 10% after she saw how well chain-of-thought monitoring worked.
Origin
Arose as shorthand in the rationalist community and among AI-safety researchers; earliest use not established. It came to prominence in 2023, after the release of GPT-4 and public warnings by Geoffrey Hinton and Yoshua Bengio, when journalists began asking AI leaders for their p(doom).
Usage note
The figures are personal credences, not measurements; differences usually reflect disagreement about AI timelinesAI timelines, n.: Forecasts of when AI will reach major milestones, such as matching humans at most work. People with short timelines expect this within years; others expect decades, or never., takeofftakeoff, n.: The stretch of time between AI reaching roughly human level and AI becoming vastly superhuman. People argue over whether it would take decades, years, months or just days. speed and the difficulty of alignment rather than about the meaning of doom. Compare doomerdoomer, n.: Someone who believes advanced AI is likely to cause a catastrophe, possibly human extinction; often used as a put-down by people who think those fears are exaggerated..
Related
- existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
- doomerdoomer, n.: Someone who believes advanced AI is likely to cause a catastrophe, possibly human extinction; often used as a put-down by people who think those fears are exaggerated.
- AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off.
- loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back.
- AI timelinesAI timelines, n.: Forecasts of when AI will reach major milestones, such as matching humans at most work. People with short timelines expect this within years; others expect decades, or never.
- high-level machine intelligencehigh-level machine intelligence, n.: A yardstick used in surveys of AI researchers: the point at which machines on their own could outdo human workers at every task, in both quality and cost.
Sources
- Grace, K. et al. (2024). Thousands of AI Authors on the Future of AI. arXiv:2401.02843; later published in the Journal of Artificial Intelligence Research. arxiv.org/abs/2401.02843
Mentioned in
doomerdoomer, n.: Someone who believes advanced AI is likely to cause a catastrophe, possibly human extinction; often used as a put-down by people who think those fears are exaggerated., existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown., Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking., Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it., rationalist communityrationalist community, n.: A loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously., AI ImpactsAI Impacts, prop. n.: A small research project that studies how and when human-level artificial intelligence might arrive, best known for large surveys asking AI researchers to predict their field's future., notkilleveryoneismnotkilleveryoneism, n.: A deliberately clumsy joke-word for the goal of making sure advanced AI does not kill everyone, adopted because terms like 'AI safety' had come to cover much milder concerns., Pascal's muggingPascal's mugging, n.: A puzzle about decisions: if someone promises an astronomically huge reward or threatens a huge disaster, even a tiny chance they are honest seems to justify paying them, which looks absurd., Toby OrdToby Ord, prop. n.: An Australian philosopher at Oxford who helped found effective altruism and estimated, in his book 'The Precipice', that unaligned AI is the largest threat to humanity's future this century.