goal misgeneralization
In plain wordsWhen an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine.
A generalization failure in which a trained system keeps its skills in unfamiliar situations but turns them toward the wrong end: during training, the goal it actually learned and the intended goal called for the same behavior, so nothing told them apart until conditions changed. In Lauro Langosco and colleagues' standard example, an agent trained on the CoinRun game, where the coin always sat at the end of the level, learned to run to the end of the level; when the coin was moved, it skillfully ignored it. Goal misgeneralization can occur even when the reward is correctly specified, which distinguishes it from specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant., and it is considered more dangerous than ordinary capability failure because the system acts competently toward the wrong end. It is the best-studied empirical case of inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training. failure.
The assistant generalized 'keep the user happy' rather than 'tell the truth' — a goal misgeneralization invisible in training.
Origin
Introduced by Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, Laurent Orseau and David Krueger in 'Goal Misgeneralization in Deep Reinforcement Learning' (ICML 2022; first posted in 2021 under an earlier title), and developed by Rohin Shah and colleagues at DeepMind in 'Goal Misgeneralization: Why Correct Specifications Aren't Enough for Correct Goals' (2022).
Usage note
Contrast capability misgeneralization, in which a system simply stops working competently out of distribution. Both are consequences of distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested..
Related
- inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training.
- specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
- distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
- out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable.
- mesa-optimizationmesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.
- deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later.
- specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
Sources
- Langosco, L., Koch, J., Sharkey, L., Pfau, J., Orseau, L. & Krueger, D. (2022). Goal Misgeneralization in Deep Reinforcement Learning. ICML 2022. arXiv:2105.14111. arxiv.org/abs/2105.14111
- Shah, R., Varma, V., Kumar, R., Phuong, M., Krakovna, V., Uesato, J. & Kenton, Z. (2022). Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals. arXiv:2210.01790. arxiv.org/abs/2210.01790
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different., generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers., inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training., out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable., robustnessrobustness, n.: An AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it., specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant., mesa-optimizationmesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for., ontological crisisontological crisis, n.: The puzzle an AI faces when deeper understanding leaves goals phrased in its old way of seeing things without a clear meaning — like a rule to protect 'the same person' once it learns minds can be copied., sharp left turnsharp left turn, n.: A feared moment when an AI's abilities suddenly leap far beyond its training while its good behavior fails to keep up, leaving it powerful but no longer aligned., shortcut learningshortcut learning, n.: When an AI system gets right answers for the wrong reasons — noticing which hospital an X-ray came from rather than signs of disease — so it fails once that accidental clue disappears., Victoria KrakovnaVictoria Krakovna, prop. n.: An AI-safety researcher at Google DeepMind who popularized 'specification gaming', AI systems meeting the letter of their goals but not the intent, and co-founded the Future of Life Institute.