Machine Intelligence Research Institute
In plain wordsA small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it.
A nonprofit research organization in Berkeley, California, and the oldest institution devoted to the alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this. of smarter-than-human AI. Its early writing on friendly AIfriendly AI, n.: An early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment., coherent extrapolated volitioncoherent extrapolated volition, n.: A proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection. and the intelligence explosionintelligence explosion, n.: The idea that once AI gets good enough at improving AI, each smarter version could build a smarter one faster still, so abilities would snowball far beyond human level in a short time. shaped the field's vocabulary, and its later 'agent foundations' program studied idealized reasoning and self-reference, producing papers on corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them. (2015), logical induction (2016) and embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it. (2018). Concluding that alignment research would not keep pace with capabilities, it announced in early 2024 a shift from technical research to communication and policy aimed at an internationally enforced halt to frontier AI development; Eliezer Yudkowsky and Nate Soares set out the case in If Anyone Builds It, Everyone Dies (2025). Critics regard its forecasts of near-certain catastrophe as overconfident.
Origin
Founded in 2000 in Atlanta as the Singularity Institute for Artificial Intelligence by Eliezer Yudkowsky with Brian and Sabine Atkins, originally aiming to build beneficial superintelligent AI itself. It moved to the San Francisco Bay Area in 2005, ran the Singularity Summit from 2006 to 2012, and took its present name in January 2013 after transferring the summit to Singularity University.
Usage note
Older literature cites it as the Singularity Institute (SIAI). Its views are a reference point in debates over p(doom)p(doom), n.: Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain.; many researchers dispute its pessimism.
Related
- Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- friendly AIfriendly AI, n.: An early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment.
- embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- AI pauseAI pause, n.: A proposed halt, temporary or indefinite, on building AI systems more powerful than today's, to give safety research and regulation time to catch up.
- LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI.
- rationalist communityrationalist community, n.: A loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously.
Sources
- Soares, N., Fallenstein, B., Yudkowsky, E. & Armstrong, S. (2015). Corrigibility. AAAI Workshop on AI and Ethics.
- Yudkowsky, E. & Soares, N. (2025). If Anyone Builds It, Everyone Dies. Little, Brown.
Mentioned in
Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking., LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI., agent foundationsagent foundations, n.: A research program that tried to work out, with mathematics, how an idealized reasoning agent should think and decide, as groundwork for building AI that stays safe., AI ImpactsAI Impacts, prop. n.: A small research project that studies how and when human-level artificial intelligence might arrive, best known for large surveys asking AI researchers to predict their field's future., embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it., friendly AIfriendly AI, n.: An early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment., Future of Humanity InstituteFuture of Humanity Institute, prop. n.: Nick Bostrom's research center at Oxford University (2005–2024), where philosophers and scientists studied threats that could permanently derail humanity's future, advanced artificial intelligence among them., logical inductionlogical induction, n.: A mathematical method for giving sensible, steadily improving probabilities to math claims that haven't been proved or disproved yet, much as a betting market prices uncertain events., logical uncertaintylogical uncertainty, n.: Uncertainty about math or logic facts that are settled in principle but that you haven't worked out yet — like not knowing a distant digit of pi until you compute it., PauseAIPauseAI, prop. n.: A volunteer protest movement, begun in 2023, that pickets AI companies and lobbies governments for a worldwide treaty halting work on AI more advanced than today's best until it is proven safe., seed AIseed AI, n.: An early AI built mainly to understand and upgrade itself, so that it could grow, like a seed, into a far more capable mind., updateless decision theoryupdateless decision theory, n.: A theory of rational choice in which an agent commits to the best overall plan for every situation it might face, instead of re-deciding each time it learns something new.