Pascal's mugging
In plain wordsA puzzle about decisions: if someone promises an astronomically huge reward or threatens a huge disaster, even a tiny chance they are honest seems to justify paying them, which looks absurd.
A thought experiment in decision theory in which a stranger promises to produce an astronomically large benefit, or avert an astronomically large harm, in exchange for a small payment such as the contents of a wallet. Because the promised stakes can be made large enough to outweigh any finite skepticism, a naive expected-utilityexpected value, n.: The long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5. maximizer appears obliged to pay, even though the claim is almost certainly false. The puzzle shows that unbounded utilities combined with probabilities that do not shrink fast enough make expected-value reasoning exploitable. Proposed responses include bounded utility functions, treating sufficiently small probabilities as zero, and penalizing claims in proportion to their stakes. In AI debates the phrase is used, often critically, against arguments that small probabilities of existential catastrophe or of vast future value should dominate decisions, and as a design concern for AI agents that could be manipulated by such offers.
Critics call a 0.01% extinction risk driving all policy a Pascal's mugging; defenders say the risk is far higher than that.
Origin
Formulated by Eliezer Yudkowsky in a 2007 Overcoming Bias post, 'Pascal's Mugging: Tiny Probabilities of Vast Utilities'; Nick Bostrom published a version as a dialogue in the journal Analysis (2009). The name adapts Pascal's wager, Blaise Pascal's 17th-century argument for belief in God.
Related
- longtermismlongtermism, n.: The belief that we should care a great deal about how things go for humanity thousands or millions of years from now, since far more people may live then than have ever lived.
- expected valueexpected value, n.: The long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5.
- existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
- p(doom)p(doom), n.: Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain.
- effective altruismeffective altruism, n.: A movement that treats charity like a research problem, comparing causes by how much good each dollar or working hour achieves; many of its members came to rank risks from advanced AI as a top priority.
Sources
- Bostrom, N. (2009). Pascal's mugging. Analysis 69(3).
Mentioned in
longtermismlongtermism, n.: The belief that we should care a great deal about how things go for humanity thousands or millions of years from now, since far more people may live then than have ever lived., Roko's basiliskRoko's basilisk, n.: A notorious internet thought experiment: a future super-powerful AI might punish everyone who foresaw it but did not help bring it about, so merely hearing the argument would put you at risk. Almost nobody accepts it.