terminal goal
In plain wordsSomething an agent wants for its own sake, not as a means to anything else — like a person valuing happiness, or an AI built to maximize a single number.
A goal pursued for its own sake rather than as a means to a further end: the ultimate objective against which an agent evaluates outcomes, formalized as its utility function or reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do.. Terminal goals are distinguished from instrumental goalsinstrumental goal, n.: Something an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task., which an agent adopts only because they help achieve its terminal goals. The distinction underpins two central arguments in AI safety: the orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us., that almost any terminal goal is compatible with almost any level of intelligence, and instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., that very different terminal goals recommend similar instrumental goals such as acquiring resources and self-preservation. Whether trained neural networksneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. have crisp terminal goals at all, rather than context-dependent drives, is disputed (see shard theoryshard theory, n.: A theory that people and AIs develop values as many small, situation-triggered habits shaped by past rewards, rather than as one fixed goal pursued single-mindedly.).
Nobody gave the model a terminal goal of self-preservation; it showed up as an instrumental one.
Origin
The contrast between ends valued for themselves and ends valued for the sake of others goes back to Aristotle; the psychologist Milton Rokeach distinguished terminal from instrumental values in 1973. Eliezer Yudkowsky's 2007 essay 'Terminal Values and Instrumental Values' brought the vocabulary into AI-safety writing, and Nick Bostrom (2012) uses 'final goals' for the same idea.
Usage note
'Final goal' (Bostrom), 'terminal value' (Yudkowsky; Rokeach in psychology) and 'intrinsic goal' are near-synonyms.
Related
- instrumental goalinstrumental goal, n.: Something an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task.
- orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us.
- instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
- reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do.
- goal-directednessgoal-directedness, n.: How much a system behaves as if it is pursuing a goal — steering toward particular outcomes and finding new ways around obstacles put in its path.
- shard theoryshard theory, n.: A theory that people and AIs develop values as many small, situation-triggered habits shaped by past rewards, rather than as one fixed goal pursued single-mindedly.
- instrumental goalinstrumental goal, n.: Something an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task.
Sources
- Bostrom, N. (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds and Machines 22(2), 71–85.
- Rokeach, M. (1973). The Nature of Human Values. Free Press.
Mentioned in
instrumental goalinstrumental goal, n.: Something an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task.