motivation selection

In Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.'s taxonomy, the family of methods that address the control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed. by shaping an AI system's goals rather than limiting its abilities. He named four: direct specification, writing explicit rules or a goal to maximize; domesticity, giving the system deliberately modest, bounded aims; indirect normativity, specifying a process by which the system works out what to value, as in coherent extrapolated volitioncoherent extrapolated volition, n.: A proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection.; and augmentation, starting from a system whose motivations are already acceptable, such as a human mind or emulationwhole brain emulation, n.: Copying a particular person's brain into a computer in such detail, down to individual neurons and their connections, that the simulation would think and behave like that person., and enhancing its intelligence. Bostrom regarded motivation selection as unavoidable in the long run, because limits on capability eventually fail. Modern post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner. methods such as reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., written constitutionsAI constitution, n.: A written set of principles that an AI company uses to shape an AI's values and behavior, both during training and as a public statement of what the AI should be like. and character trainingcharacter training, n.: Training an AI to have a consistent, deliberate personality — traits such as curiosity, honesty and warmth — rather than leaving its manner and values as accidental by-products of other training. can be seen as practical, if imperfect, motivation selection.

Constitution-based training is motivation selection by another name.