oracle AI
In plain wordsAn AI kept to answering questions, without acting in the world on its own. Proposed as a safer kind of superintelligence, though its answers could still steer the people who read them.
An AI system designed or restricted to answer questions rather than take actions, proposed as a way to obtain the benefits of superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist. while limiting its ability to cause harm. In Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.'s classification of system types, the oracle sits beside the genie (which carries out commands), the sovereign (which acts autonomously on open-ended goals) and the tooltool AI, n.: AI built to be used like an instrument, answering, suggesting or calculating when asked, rather than pursuing goals on its own. Some argue that this is the only kind we should build.. Analyses identify residual risks: an oracle can manipulate the people who read its answers, its predictions can be self-fulfilling, and a goal of answering well may motivate it to acquire resources or escape confinement. Proposed safeguards include restricting answers to yes/no or multiple choice, consulting several isolated oracles, and counterfactual oracles whose answers are scored only in cases where nobody reads them. Yoshua Bengio's 2025 proposal for a non-agentic 'Scientist AI' is a recent descendant of the idea.
An oracle AI that only answers questions can still change the world through the answers it chooses.
Origin
Discussed in AI-risk circles in the 2000s and analyzed systematically by Stuart Armstrong, Anders Sandberg and Nick Bostrom in 'Thinking Inside the Box: Controlling and Using an Oracle AI' (Minds and Machines, 2012); Bostrom's Superintelligence (2014) added the oracle–genie–sovereign typology.
Related
- tool AItool AI, n.: AI built to be used like an instrument, answering, suggesting or calculating when asked, rather than pursuing goals on its own. Some argue that this is the only kind we should build.
- AI boxAI box, n.: Keeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad.
- capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants.
- control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed.
- comprehensive AI servicescomprehensive AI services, n.: The idea that superintelligent-level AI could arrive as many specialized AI tools, each doing a bounded job, rather than as one all-powerful AI agent with goals of its own.
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
Sources
- Armstrong, S., Sandberg, A. & Bostrom, N. (2012). Thinking Inside the Box: Controlling and Using an Oracle AI. Minds and Machines 22(4).
- Bengio, Y. et al. (2025). Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? arXiv:2502.15657.
Mentioned in
intelligent agentintelligent agent, n.: In AI theory, a decision-maker, such as a robot, a self-driving car, a chess program or a helpful piece of software, that notices what is going on around it and responds in ways that serve its goals., sovereign AIsovereign AI, n.: A country's ability to build and run AI on its own terms, so that the computers, models and know-how it relies on cannot be withheld or dictated by foreign companies or governments., AI boxAI box, n.: Keeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad., AI-box experimentAI-box experiment, n.: A role-playing test in which one person pretends to be a trapped super-smart AI and tries, by text chat alone, to talk a 'gatekeeper' into letting it out., comprehensive AI servicescomprehensive AI services, n.: The idea that superintelligent-level AI could arrive as many specialized AI tools, each doing a bounded job, rather than as one all-powerful AI agent with goals of its own., tool AItool AI, n.: AI built to be used like an instrument, answering, suggesting or calculating when asked, rather than pursuing goals on its own. Some argue that this is the only kind we should build.