Steve Omohundro
In plain wordsAn American computer scientist who argued in 2008 that almost any sufficiently advanced AI would tend to protect itself and seek resources, unless deliberately designed not to.
American computer scientist whose paper 'The Basic AI Drives', presented at the first Conference on Artificial General Intelligence (2008), argued that sufficiently advanced goal-seeking systems will, whatever their final goals, tend to develop basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources.: to preserve themselves, protect their goals from change, acquire resources, improve themselves and become more rational, unless explicitly designed otherwise. Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern. formalized the argument as instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., and it underlies concerns about power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have. and corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.. Trained as a physicist (PhD, University of California, Berkeley), he earlier co-developed the Star Lisp parallel programming language at Thinking Machines Corporation, taught at the University of Illinois, where he researched machine learning and computer vision, and designed the Sather programming language at the International Computer Science Institute in Berkeley. More recently he has argued for provably safe AI built on formal verification, notably in a 2023 paper with Max TegmarkMax Tegmark, prop. n.: A Swedish-American physicist at MIT who co-founded the Future of Life Institute, which organized the 2023 open letter calling for a pause in training the most powerful AI systems..
Related
- basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources.
- instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
- power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- instrumental goalinstrumental goal, n.: Something an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task.
- Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.
Sources
- Omohundro, S. M. (2008). The Basic AI Drives. In Artificial General Intelligence 2008: Proceedings of the First AGI Conference. IOS Press.
- Tegmark, M. & Omohundro, S. (2023). Provably safe systems: the only path to controllable AGI. arXiv preprint.