ontological crisis
In plain wordsThe puzzle an AI faces when deeper understanding leaves goals phrased in its old way of seeing things without a clear meaning — like a rule to protect 'the same person' once it learns minds can be copied.
The predicament, analyzed by Peter de Blanc in 2011, that arises when an agent replaces its model of the world with a better one but its goals were written in the vocabulary of the old model. Some things the goals mention may have no exact counterpart in the new model, as everyday solid objects have none in a theory of quantum fields, so the agent must choose how to translate its values, and different translations can yield very different behavior: an agent charged with protecting a particular person must decide what that means once its model allows people to be copied. Ontological crises matter for alignment because capable AI systems may develop internal concepts unlike human ones, and their goals — or the human values they have learned — must carry over correctly across such shifts. Proposed responses include mapping between ontologies in structure-preserving ways and hoping that natural abstractionsnatural abstraction hypothesis, n.: The idea that the world naturally sorts into a limited set of useful concepts — like 'tree' or 'temperature' — that any capable mind, human or AI, would end up learning. provide stable anchors.
When the agent's world model was rebuilt, its goal had no obvious translation — a small-scale ontological crisis.
Origin
Introduced by Peter de Blanc in 'Ontological Crises in Artificial Agents' Value Systems' (2011), written at the Singularity Institute (now the Machine Intelligence Research Institute).
Related
- embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it.
- natural abstraction hypothesisnatural abstraction hypothesis, n.: The idea that the world naturally sorts into a limited set of useful concepts — like 'tree' or 'temperature' — that any capable mind, human or AI, would end up learning.
- value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance.
- agent foundationsagent foundations, n.: A research program that tried to work out, with mathematics, how an idealized reasoning agent should think and decide, as groundwork for building AI that stays safe.
- goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine.
Sources
- de Blanc, P. (2011). Ontological Crises in Artificial Agents' Value Systems. arXiv:1105.3821. arxiv.org/abs/1105.3821
Mentioned in
agent foundationsagent foundations, n.: A research program that tried to work out, with mathematics, how an idealized reasoning agent should think and decide, as groundwork for building AI that stays safe., embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it., natural abstraction hypothesisnatural abstraction hypothesis, n.: The idea that the world naturally sorts into a limited set of useful concepts — like 'tree' or 'temperature' — that any capable mind, human or AI, would end up learning.