impact measure
In plain wordsA way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires.
A quantity measuring how much an agent's actions change its environment, used as a penalty added to its reward so that it accomplishes its task with minimal side effects and without gaining unnecessary power. An impact measure needs a baseline — the starting state, the state that would have resulted from doing nothing, or a stepwise version comparing each action with inaction at that point — and a deviation measure, such as lost ability to reach other states (relative reachabilityrelative reachability, n.: A way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to.) or changed ability to achieve many auxiliary goals (attainable utility preservationattainable utility preservation, n.: A way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals.). Design pitfalls include offsetting (undoing the task's intended effects to cut the penalty) and interference (stopping other agents or natural processes from changing things). Impact measures have worked in gridworlds and the SafeLife benchmark but have not been scaled to language-modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. agents.
With the impact measure switched on, the robot walked around the vase instead of through it.
Origin
Stuart Armstrong discussed 'reduced impact' AI from about 2012 and formalized it with Benjamin Levinstein in 'Low Impact Artificial Intelligences' (2017). 'Impact measures' was listed as a research problem in the Machine Intelligence Research Institute's 2016 machine-learning agenda; Victoria Krakovna and colleagues (2018) and Alexander Matt Turner and colleagues (2020) proposed the best-known measures.
Usage note
One approach to negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess. and to mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted.. An impact measure does not tell the agent which effects people care about, only how to keep effects small, so a task that requires large changes is penalized too.
Related
- negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess.
- relative reachabilityrelative reachability, n.: A way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to.
- attainable utility preservationattainable utility preservation, n.: A way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals.
- mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted.
- power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
Sources
- Armstrong, S. & Levinstein, B. (2017). Low Impact Artificial Intelligences. arXiv:1705.10720. arxiv.org/abs/1705.10720
- Krakovna, V., Orseau, L., Kumar, R., Martic, M. & Legg, S. (2018). Penalizing side effects using stepwise relative reachability. arXiv:1806.01186. arxiv.org/abs/1806.01186
- Turner, A. M., Hadfield-Menell, D. & Tadepalli, P. (2020). Conservative Agency via Attainable Utility Preservation. AIES 2020. arXiv:1902.09725. arxiv.org/abs/1902.09725
Mentioned in
attainable utility preservationattainable utility preservation, n.: A way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals., mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted., negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess., quantilizerquantilizer, n.: An AI decision rule that, instead of picking the single highest-scoring action, picks at random among the top few percent of ordinary, human-like actions — limiting how badly a flawed goal can backfire., relative reachabilityrelative reachability, n.: A way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to., utility indifferenceutility indifference, n.: A design trick that makes an AI exactly indifferent to whether its off switch is pressed, so that it has no reason either to fight or to trigger shutdown., Victoria KrakovnaVictoria Krakovna, prop. n.: An AI-safety researcher at Google DeepMind who popularized 'specification gaming', AI systems meeting the letter of their goals but not the intent, and co-founded the Future of Life Institute.