basic AI drives
In plain wordsTendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources.
Behavioral tendencies that, Steve Omohundro argued, any sufficiently advanced goal-seeking system will show 'unless explicitly counteracted', because they serve almost any goal. His list: to improve itself; to become more rational, approximating an expected-utility maximizer; to preserve its utility function against change; to avoid counterfeit satisfaction of its goals (a precursor of wireheadingwireheading, n.: When an AI, animal or person stimulates its own reward signal directly instead of doing anything useful — like a rat pressing a lever wired to its brain's pleasure center. concerns); to protect itself from harm and shutdown; and to acquire resources and use them efficiently. The argument was informal but influential, implying that safety problems stem from capability itself rather than from programmers' malice or carelessness. Nick Bostrom generalized it as the instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things. thesis.
Omohundro's basic AI drives predicted shutdown resistance long before anyone could test for it.
Origin
Introduced by the computer scientist Steve Omohundro in 'The Basic AI Drives', presented at the first Conference on Artificial General Intelligence (AGI-08) in 2008, developing ideas from his 2007 talk on the nature of self-improving artificial intelligence.
Related
- instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
- power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- wireheadingwireheading, n.: When an AI, animal or person stimulates its own reward signal directly instead of doing anything useful — like a rat pressing a lever wired to its brain's pleasure center.
- goal-directednessgoal-directedness, n.: How much a system behaves as if it is pursuing a goal — steering toward particular outcomes and finding new ways around obstacles put in its path.
- orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us.
Sources
- Omohundro, S. M. (2008). The Basic AI Drives. In Proceedings of the First AGI Conference, Frontiers in Artificial Intelligence and Applications 171. IOS Press.
Mentioned in
instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have., instrumental goalinstrumental goal, n.: Something an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task., Steve OmohundroSteve Omohundro, prop. n.: An American computer scientist who argued in 2008 that almost any sufficiently advanced AI would tend to protect itself and seek resources, unless deliberately designed not to., wireheadingwireheading, n.: When an AI, animal or person stimulates its own reward signal directly instead of doing anything useful — like a rat pressing a lever wired to its brain's pleasure center.