orthogonality thesis
In plain wordsThe idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us.
The thesis that an agent's level of intelligence places almost no constraint on what it ultimately wants: in principle, great capability can be paired with nearly any final goal, however strange. Intelligence in this sense is instrumental competence, the ability to forecast, plan and choose effective means, not wisdom or moral insight. The thesis denies that sufficient intelligence would by itself lead an agent to adopt humane values: in Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.'s examples, a superintelligent system could single-mindedly count grains of sand or maximize paperclips (paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous.). Combined with instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., it underpins the argument that advanced AI will not be safe by default and that its goals must be deliberately aligned. Critics include moral realists, who hold that a capable enough reasoner would discover and adopt correct values, and researchers who argue that systems trained on human data in practice acquire humanlike values along with their capabilities. Defenders reply that the thesis concerns what is possible, not what is likely.
The orthogonality thesis is why 'it will be too smart to want something stupid' is not a safety argument.
Origin
Named and formulated by Nick Bostrom in 'The Superintelligent Will' (Minds and Machines, 2012) and restated in Superintelligence (2014). The idea echoes David Hume's view that reason serves the passions, and Eliezer Yudkowsky had argued it informally, writing in 2008 that an AI 'does not hate you, nor does it love you'.
Usage note
Often misread as saying AI goals will be random; it says only that intelligence does not constrain them. It is a claim about possible minds, not about which goals training actually produces.
Related
- instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
- paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target.
- Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.
- superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist.
Sources
- Bostrom, N. (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds and Machines 22(2).
- Armstrong, S. (2013). General Purpose Intelligence: Arguing the Orthogonality Thesis. Analysis and Metaphysics 12.
Mentioned in
alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different., paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous., instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern., basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources., Future of Humanity InstituteFuture of Humanity Institute, prop. n.: Nick Bostrom's research center at Oxford University (2005–2024), where philosophers and scientists studied threats that could permanently derail humanity's future, advanced artificial intelligence among them., instrumental goalinstrumental goal, n.: Something an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task., orthogonalityorthogonality, n.: The mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map., terminal goalterminal goal, n.: Something an agent wants for its own sake, not as a means to anything else — like a person valuing happiness, or an AI built to maximize a single number.