infohazard
In plain wordsInformation that could cause harm simply by being shared or known — for example, details that would make a dangerous weapon easier to build.
A danger posed by knowledge itself: true information whose circulation could do harm directly or equip someone to do harm, in the sense formalized by Nick Bostrom. His typology distinguishes, among others, data hazards (such as the genome of a dangerous pathogen), idea hazards (a general approach that could be misused), attention hazards (signaling that something is promising or vulnerable) and hazards to the knower. In AI governance the concept underlies decisions to withhold model weights, dangerous-capability findings or research details, and the concern about CBRNCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone. uplift that shapes model safeguards. In internet culture, especially the rationalist communityrationalist community, n.: A loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously., the clipped form 'infohazard' is also used loosely or jokingly for ideas thought harmful merely to contemplate, of which Roko's basiliskRoko's basilisk, n.: A notorious internet thought experiment: a future super-powerful AI might punish everyone who foresaw it but did not help bring it about, so merely hearing the argument would put you at risk. Almost nobody accepts it. is the best-known example.
The team treated the jailbreak write-up as an infohazard and shared it only with the affected labs.
Origin
'Information hazard' was formalized by the philosopher Nick Bostrom in 'Information Hazards: A Typology of Potential Harms from Knowledge' (Review of Contemporary Philosophy, 2011); the contraction 'infohazard' spread on LessWrong and in collaborative fiction such as the SCP Foundation wiki during the 2010s. The underlying concern is older, from nuclear secrecy to dual-use research in biology.
Usage note
Labeling something an infohazard can itself draw attention to it, a dynamic Bostrom discusses and the Streisand effect around Roko's basilisk illustrates.
Related
- Roko's basiliskRoko's basilisk, n.: A notorious internet thought experiment: a future super-powerful AI might punish everyone who foresaw it but did not help bring it about, so merely hearing the argument would put you at risk. Almost nobody accepts it.
- CBRN riskCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone.
- Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.
- open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.
- rationalist communityrationalist community, n.: A loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously.
- dual-usedual-use, adj.: Describes technology or knowledge that can be used both to help and to harm: the same AI that finds software bugs for defenders can find them for attackers.
Sources
- Bostrom, N. (2011). Information Hazards: A Typology of Potential Harms from Knowledge. Review of Contemporary Philosophy, 10, 44–79.
Mentioned in
Roko's basiliskRoko's basilisk, n.: A notorious internet thought experiment: a future super-powerful AI might punish everyone who foresaw it but did not help bring it about, so merely hearing the argument would put you at risk. Almost nobody accepts it.