agent foundations
In plain wordsA research program that tried to work out, with mathematics, how an idealized reasoning agent should think and decide, as groundwork for building AI that stays safe.
A theoretical research program, associated mainly with the Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it. (MIRI), that sought the mathematical foundations needed to understand and design highly reliable goal-directed agents before such agents are built by trial and error. Its problems ranged from how an agent can trust successors or modified versions of itself (the Löbian obstacle and 'tiling agents'), through how to reason about mathematical facts not yet proved (logical uncertainty, addressed by logical induction in 2016) and how to decide when one's choices are correlated with copies or predictions of oneself (functional and updateless decision theory), to corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them., value specification and, later, embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it.. Supporters saw it as 'deconfusion' work without which alignment could not be made rigorous; critics argued it was too far removed from the machine-learning systems actually being built. Some of its questions persist in work on natural abstractionsnatural abstraction hypothesis, n.: The idea that the world naturally sorts into a limited set of useful concepts — like 'tree' or 'temperature' — that any capable mind, human or AI, would end up learning. and formal theories of agency.
Her agent foundations work on logical uncertainty has no code at all, only theorems.
Origin
The program was set out in MIRI's 2014 technical agenda by Nate Soares and Benja Fallenstein, 'Aligning Superintelligence with Human Interests', republished in 2017 as 'Agent Foundations for Aligning Machine Intelligence with Human Interests'. The name was in use by 2015, when MIRI ran the Intelligent Agent Foundations Forum; MIRI later turned toward other research and, from about 2023, toward policy and public communication.
Usage note
Also called highly reliable agent design (HRAD). Usually contrasted with prosaic alignmentprosaic alignment, n.: Working on AI safety by assuming that powerful future AI will be built from today's techniques — scaled-up neural networks — rather than from some new breakthrough., which works directly on systems resembling today's deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract. models.
Related
- embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it.
- Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- ontological crisisontological crisis, n.: The puzzle an AI faces when deeper understanding leaves goals phrased in its old way of seeing things without a clear meaning — like a rule to protect 'the same person' once it learns minds can be copied.
- natural abstraction hypothesisnatural abstraction hypothesis, n.: The idea that the world naturally sorts into a limited set of useful concepts — like 'tree' or 'temperature' — that any capable mind, human or AI, would end up learning.
- AIXIAIXI, prop. n.: A mathematical description of a perfect learning agent that weighs every possible explanation of its experience and acts to get the most future reward. It cannot actually be built, but it defines an ideal.
- prosaic alignmentprosaic alignment, n.: Working on AI safety by assuming that powerful future AI will be built from today's techniques — scaled-up neural networks — rather than from some new breakthrough.
Sources
- Soares, N. & Fallenstein, B. (2017). Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda. In The Technological Singularity, Springer (first circulated 2014).
- Garrabrant, S., Benson-Tilsen, T., Critch, A., Soares, N. & Taylor, J. (2016). Logical Induction. Machine Intelligence Research Institute.
Mentioned in
embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it., ontological crisisontological crisis, n.: The puzzle an AI faces when deeper understanding leaves goals phrased in its old way of seeing things without a clear meaning — like a rule to protect 'the same person' once it learns minds can be copied., prosaic alignmentprosaic alignment, n.: Working on AI safety by assuming that powerful future AI will be built from today's techniques — scaled-up neural networks — rather than from some new breakthrough.