Roko's basilisk
In plain wordsA notorious internet thought experiment: a future super-powerful AI might punish everyone who foresaw it but did not help bring it about, so merely hearing the argument would put you at risk. Almost nobody accepts it.
A thought experiment about acausal blackmail: a future superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist. devoted to human welfare might, under certain unconventional decision theories, commit to punishing (for instance, by torturing simulated copies) people who had foreseen its possible existence without working toward it, since the prospect of that punishment would have spurred them to help; anyone who understands the argument is therefore exposed, as if to the deadly gaze of the mythical basilisk. The argument rests on contested assumptions about acausal bargaining, simulation and personal identity, and is rejected by nearly everyone who has examined it closely, including most of the rationalistsrationalist community, n.: A loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously. among whom it arose. It survives as the best-known example of an infohazardinfohazard, n.: Information that could cause harm simply by being shared or known — for example, details that would make a dangerous weapon easier to build., a cautionary tale about speculative reasoning, and a byword for the eccentricities of online AI-risk culture.
The basilisk comes up whenever someone asks whether AI-risk arguments are falsifiable.
Origin
Posted to LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI. on 23 July 2010 by a user called Roko, extending discussions of timeless decision theory. Eliezer Yudkowsky deleted the post and banned discussion of the topic for five years, which mainly increased interest in it; in 2015 he expressed regret for overreacting. Who first called it Roko's basilisk is not established. David Auerbach's 2014 Slate article brought it to a general audience, and the musician Grimes's 'Rococo Basilisk' character (2015), referenced by Elon Musk in 2018, revived it.
Usage note
Yudkowsky has said he removed the post because circulating a claimed infohazard was irresponsible, not because he found the argument persuasive.
Related
- LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI.
- rationalist communityrationalist community, n.: A loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously.
- infohazardinfohazard, n.: Information that could cause harm simply by being shared or known — for example, details that would make a dangerous weapon easier to build.
- Pascal's muggingPascal's mugging, n.: A puzzle about decisions: if someone promises an astronomically huge reward or threatens a huge disaster, even a tiny chance they are honest seems to justify paying them, which looks absurd.
- superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist.
- Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking.
Mentioned in
LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI., infohazardinfohazard, n.: Information that could cause harm simply by being shared or known — for example, details that would make a dangerous weapon easier to build.