mild optimization
In plain wordsBuilding an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted.
A family of proposals for agents that optimize their objective less than fully, so that errors in the objective are not amplified into extreme or unanticipated outcomes. Maximizing a proxy pushes behavior toward unusual regions where proxy and true goal come apart (Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.); a milder optimizer stays closer to ordinary, well-understood behavior. Approaches include satisficing (accepting any outcome above a threshold), quantilizersquantilizer, n.: An AI decision rule that, instead of picking the single highest-scoring action, picks at random among the top few percent of ordinary, human-like actions — limiting how badly a flawed goal can backfire. (sampling from the best fraction of a trusted distribution of actions), optimization penalized by divergence from a reference policy — the KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer. used in reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. — and stopping optimization early. A noted difficulty is stability: a satisficer may find that building a full maximizer is a convenient way to satisfice, so mildness must survive self-modification and delegation.
The KL penalty is mild optimization in practice, though few RLHF papers call it that.
Origin
Herbert A. Simon introduced 'satisficing' in 1956. 'Mild optimization' was named as one of eight research problems in the Machine Intelligence Research Institute's 2016 agenda 'Alignment for Advanced Machine Learning Systems' by Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire and Andrew Critch.
Related
- quantilizerquantilizer, n.: An AI decision rule that, instead of picking the single highest-scoring action, picks at random among the top few percent of ordinary, human-like actions — limiting how badly a flawed goal can backfire.
- Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.
- impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires.
- KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.
- reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse.
- myopiamyopia, n.: Designing an AI to care only about the immediate results of its actions, not about what happens far in the future, so it has no reason for long-term scheming.
- optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible.
Sources
- Taylor, J., Yudkowsky, E., LaVictoire, P. & Critch, A. (2016). Alignment for Advanced Machine Learning Systems. Machine Intelligence Research Institute technical report.
- Simon, H. A. (1956). Rational Choice and the Structure of the Environment. Psychological Review 63(2), 129–138.
Mentioned in
Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure., satisficingsatisficing, n.: Choosing the first option that is good enough instead of searching for the very best one — a sensible strategy when time, information or brainpower are limited., impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires., myopiamyopia, n.: Designing an AI to care only about the immediate results of its actions, not about what happens far in the future, so it has no reason for long-term scheming., negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess., quantilizerquantilizer, n.: An AI decision rule that, instead of picking the single highest-scoring action, picks at random among the top few percent of ordinary, human-like actions — limiting how badly a flawed goal can backfire.