coherent extrapolated volition
In plain wordsA proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection.
A proposed target for the values of a superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist.: rather than obeying current human commands or a fixed moral code, the AI would act on what humanity's wishes would converge to under idealized reflection. In Eliezer Yudkowsky's formulation it would extrapolate what we would want 'if we knew more, thought faster, were more the people we wished we were', and act only where those extrapolated wishes cohere rather than conflict. CEV is the best-known example of what Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern. calls indirect normativity: specifying a process for discovering values instead of the values themselves, on the grounds that humans cannot write down a correct value system and that the AI's greater intelligence could be used to work it out. Objections include that the extrapolation is undefined in practice, that people's values may not converge, that deciding whose volition counts is itself political, and that the procedure could not be tested before use.
CEV dodges the question of which values to program by delegating it to an idealized version of us.
Origin
Introduced by Eliezer Yudkowsky in the 2004 Singularity Institute paper 'Coherent Extrapolated Volition', as a successor to his earlier friendly AIfriendly AI, n.: An early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment. proposals, and discussed by Nick Bostrom in Superintelligence (2014). Yudkowsky later described the document as outdated, though the idea remains a reference point.
Related
- friendly AIfriendly AI, n.: An early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment.
- motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do.
- value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target.
- value lock-invalue lock-in, n.: The possibility that powerful AI could freeze one set of values or one power structure in place for an extremely long time, so that later generations could never change course.
- Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
Sources
- Yudkowsky, E. (2004). Coherent Extrapolated Volition. Singularity Institute for Artificial Intelligence.
Mentioned in
Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking., Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it., value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target., value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance., friendly AIfriendly AI, n.: An early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment., motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do., value lock-invalue lock-in, n.: The possibility that powerful AI could freeze one set of values or one power structure in place for an extremely long time, so that later generations could never change course.