embedded agency
In plain wordsThe puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it.
The study of agents that are themselves part of the world they model and act upon. Standard 'dualistic' formalisms such as AIXIAIXI, prop. n.: A mathematical description of a perfect learning agent that weighs every possible explanation of its experience and acts to get the most future reward. It cannot actually be built, but it defines an ideal. and classical reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. instead place the agent outside that world, trading observations and actions with it through a clean interface and able, at least in principle, to model it completely. Embedded agents face problems those formalisms assume away: they cannot hold a complete model of a world that contains them (embedded world-models); they must reason about parts of the world, including copies of themselves, that depend on their choices (decision theory); they must build or become successors whose goals they can trust (robust delegation); and their own internal processes may contain sub-agents with different goals (subsystem alignment, related to mesa-optimizationmesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.). The program sought foundational mathematics for reasoning about such agents.
AIXI never has to worry about a cosmic ray flipping its own bits — that's an embedded agency problem.
Origin
Named by Abram Demski and Scott Garrabrant of the Machine Intelligence Research Institute in the illustrated sequence 'Embedded Agency' (October–November 2018), later posted to arXiv (2019), which summarized the institute's agent foundationsagent foundations, n.: A research program that tried to work out, with mathematics, how an idealized reasoning agent should think and decide, as groundwork for building AI that stays safe. research.
Related
- agent foundationsagent foundations, n.: A research program that tried to work out, with mathematics, how an idealized reasoning agent should think and decide, as groundwork for building AI that stays safe.
- AIXIAIXI, prop. n.: A mathematical description of a perfect learning agent that weighs every possible explanation of its experience and acts to get the most future reward. It cannot actually be built, but it defines an ideal.
- mesa-optimizationmesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.
- ontological crisisontological crisis, n.: The puzzle an AI faces when deeper understanding leaves goals phrased in its old way of seeing things without a clear meaning — like a rule to protect 'the same person' once it learns minds can be copied.
- Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it.
- natural abstraction hypothesisnatural abstraction hypothesis, n.: The idea that the world naturally sorts into a limited set of useful concepts — like 'tree' or 'temperature' — that any capable mind, human or AI, would end up learning.
Sources
- Demski, A. & Garrabrant, S. (2019). Embedded Agency. arXiv:1902.09469. arxiv.org/abs/1902.09469
Mentioned in
bounded rationalitybounded rationality, n.: The idea that real decision-makers, whether people or machines, cannot consider every option or compute the perfect choice, so they make reasonable decisions within limits of time, information and thinking power., Gödel's incompleteness theoremsGödel's incompleteness theorems, n.: Two famous results from 1931 showing that any consistent set of rules for doing arithmetic leaves some true statements unprovable, and cannot prove that it is itself free of contradictions., Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it., agent foundationsagent foundations, n.: A research program that tried to work out, with mathematics, how an idealized reasoning agent should think and decide, as groundwork for building AI that stays safe., AIXIAIXI, prop. n.: A mathematical description of a perfect learning agent that weighs every possible explanation of its experience and acts to get the most future reward. It cannot actually be built, but it defines an ideal., logical inductionlogical induction, n.: A mathematical method for giving sensible, steadily improving probabilities to math claims that haven't been proved or disproved yet, much as a betting market prices uncertain events., logical uncertaintylogical uncertainty, n.: Uncertainty about math or logic facts that are settled in principle but that you haven't worked out yet — like not knowing a distant digit of pi until you compute it., natural abstraction hypothesisnatural abstraction hypothesis, n.: The idea that the world naturally sorts into a limited set of useful concepts — like 'tree' or 'temperature' — that any capable mind, human or AI, would end up learning., ontological crisisontological crisis, n.: The puzzle an AI faces when deeper understanding leaves goals phrased in its old way of seeing things without a clear meaning — like a rule to protect 'the same person' once it learns minds can be copied., updateless decision theoryupdateless decision theory, n.: A theory of rational choice in which an agent commits to the best overall plan for every situation it might face, instead of re-deciding each time it learns something new.