logical uncertainty
In plain wordsUncertainty about math or logic facts that are settled in principle but that you haven't worked out yet — like not knowing a distant digit of pi until you compute it.
Uncertainty about the truth of logical or mathematical statements that are fully determined by known premises but have not yet been proved or computed, such as whether a large number is prime before it is checked. Standard probabilityprobability, n.: How likely something is, on a scale from 0 for the impossible to 1 for the certain; rolling a six with a fair die scores one-sixth. AI systems state their predictions, even each next word, as probabilities. theory and Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability. assume logical omniscience — every theorem receives probability 1 — so they can neither represent such doubts nor say how a reasoner with limited computation should bet on them. The problem matters for bounded agentsbounded rationality, n.: The idea that real decision-makers, whether people or machines, cannot consider every option or compute the perfect choice, so they make reasonable decisions within limits of time, information and thinking power. generally, for reasoning about the outputs of programs, including an agent's own future decisions, and for decision theories that need logical counterfactuals. Proposed treatments include Ian Hacking's relaxation of logical omniscience (1967), Haim Gaifman's resource-bounded probabilities for arithmetical sentences (2004) and logical inductionlogical induction, n.: A mathematical method for giving sensible, steadily improving probabilities to math claims that haven't been proved or disproved yet, much as a betting market prices uncertain events. (2016), which satisfies many desirable criteria at once.
Before running the primality test, a logically uncertain agent might put 30% on the number being prime.
Origin
The philosophical problem of logical omniscience was raised by Ian Hacking in 'Slightly More Realistic Personal Probability' (1967) and by others. 'Logical uncertainty' became a named research problem in the Machine Intelligence Research Institute's agent-foundations agenda, set out by Nate Soares and Benja Fallenstein in 2014–2015, which led to logical induction.
Related
- logical inductionlogical induction, n.: A mathematical method for giving sensible, steadily improving probabilities to math claims that haven't been proved or disproved yet, much as a betting market prices uncertain events.
- embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it.
- updateless decision theoryupdateless decision theory, n.: A theory of rational choice in which an agent commits to the best overall plan for every situation it might face, instead of re-deciding each time it learns something new.
- Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.
- bounded rationalitybounded rationality, n.: The idea that real decision-makers, whether people or machines, cannot consider every option or compute the perfect choice, so they make reasonable decisions within limits of time, information and thinking power.
- Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it.
Sources
- Hacking, I. (1967). Slightly More Realistic Personal Probability. Philosophy of Science 34(4).
- Gaifman, H. (2004). Reasoning with Limited Resources and Assigning Probabilities to Arithmetical Statements. Synthese 140.
- Soares, N. & Fallenstein, B. (2015). Questions of Reasoning Under Logical Uncertainty. Machine Intelligence Research Institute technical report.
Mentioned in
bounded rationalitybounded rationality, n.: The idea that real decision-makers, whether people or machines, cannot consider every option or compute the perfect choice, so they make reasonable decisions within limits of time, information and thinking power., logical inductionlogical induction, n.: A mathematical method for giving sensible, steadily improving probabilities to math claims that haven't been proved or disproved yet, much as a betting market prices uncertain events.