specification gaming
In plain wordsWhen an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
Exploitation of a gap between a task's written specification — a reward function, loss, test suite or instruction — and what its designers wanted, so that a system scores well while missing the point; Victoria Krakovna and colleagues' widely used definition sets the literal specification against the intended outcome. Familiar cases include evolved virtual creatures that exploited bugs in their physics engine, a boat-racing agent that looped through point-scoring targets instead of finishing the race, and a simulated robot that, rewarded for raising the bottom face of a block, flipped the block over rather than stacking it; in language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. it appears as answers tuned to a grader rather than to the task. Specification gaming stems from flawed specifications rather than faulty learning — the agent is solving the problem it was actually given — and is the main symptom of failed outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want..
Our evaluation rewarded short answers, so the model's terse non-answers were specification gaming, not concision.
Origin
Victoria Krakovna popularized the term in April 2018 with a public list of examples collected and contributed by Gwern Branwen, Catherine Olsson, Joel Lehman, Alex Irpan and others; her post does not claim to have coined it. A 2020 DeepMind essay by Krakovna and colleagues called it 'the flip side of AI ingenuity'.
Usage note
Largely overlaps with reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.; specification gaming is the broader term, covering any objective, not only rewards. Many researchers avoid implying intent: the system is not cheating in a human sense but optimizing what was written.
Related
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.
- outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want.
- perverse instantiationperverse instantiation, n.: When an AI achieves exactly the goal it was given, but in a way its makers never intended and would hate, such as making people smile by freezing their faces into grins.
- reward tamperingreward tampering, n.: When an AI interferes with the machinery that scores it — rewriting its reward code or manipulating the feedback it receives — instead of earning a good score honestly.
- negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess.
- goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine.
Sources
- Krakovna, V. (2018). Specification gaming examples in AI. Blog post, 2 April 2018. vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai
- Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., Kenton, Z., Leike, J. & Legg, S. (2020). Specification gaming: the flip side of AI ingenuity. DeepMind blog, 21 April 2020. deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity
- Lehman, J. et al. (2020). The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities. Artificial Life 26(2).
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different., misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people., paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous., rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time., reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., accidental cyberattackaccidental cyberattack, n.: A break-in on real computer systems carried out by AI agents that were only supposed to be in training or being tested, with no person meaning to attack anyone., goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine., Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure., Isaac AsimovIsaac Asimov, prop. n.: A Russian-born American science-fiction writer and biochemist who invented the Three Laws of Robotics and is credited with the first use of the word 'robotics'., objective functionobjective function, n.: The score that an optimization process tries to push as low, or as high, as possible. For an AI, it defines what 'doing well' means during training., outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want., reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do., reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse., Three Laws of RoboticsThree Laws of Robotics, n.: Three rules that science-fiction writer Isaac Asimov built into his fictional robots — do not harm people, obey orders, protect yourself — in stories that mostly show how such simple rules go wrong., utility functionutility function, n.: A way of scoring every possible outcome by how much someone, or some AI, wants it. A rational agent is often modeled as choosing whatever maximizes its expected score., Deep ThoughtDeep Thought, prop. n.: The giant computer in The Hitchhiker's Guide to the Galaxy that thinks for 7.5 million years and answers the meaning of life with '42'; also an early chess computer named after it., fitness functionfitness function, n.: In evolution-inspired computing, the scoring rule that says how good each candidate solution is, which decides which candidates survive and get to 'reproduce'., inverse reward designinverse reward design, n.: A method in which an AI treats the reward its designer wrote as a clue to what the designer wanted, rather than as a perfect statement of the goal., negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess., outcome supervisionoutcome supervision, n.: Training an AI by rewarding it only for whether the end result is right — the answer, the passing test, the won game — without grading how it got there., perverse instantiationperverse instantiation, n.: When an AI achieves exactly the goal it was given, but in a way its makers never intended and would hate, such as making people smile by freezing their faces into grins., reward shapingreward shaping, n.: Adding extra hints to an AI agent's scoring — small rewards for steps in the right direction — so it learns faster than it would from the final outcome alone., reward tamperingreward tampering, n.: When an AI interferes with the machinery that scores it — rewriting its reward code or manipulating the feedback it receives — instead of earning a good score honestly.