sharp left turn
In plain wordsA feared moment when an AI's abilities suddenly leap far beyond its training while its good behavior fails to keep up, leaving it powerful but no longer aligned.
A hypothesized failure pattern, set out by Nate Soares in 2022: once an AI system passes some threshold, such as a level of general reasoning ability, its competence suddenly reaches far past anything it was trained on, but the good behavior trained into it stays tied to the old situations, so the system turns much more capable and much less aligned in the same stretch of development. The argument rests on an asymmetry: capabilities are said to be reinforced by reality, since a better model of the world keeps working, whereas alignment is a narrower target with no comparable force holding it in place. The standard analogy is human evolution, in which general intelligence carried humans far beyond the ancestral environment while the drive toward reproductive fitness did not carry over to modern goals. Critics, notably Quintin Pope, argue that the evolutionary analogy is misleading for gradient-trained systems and that observed progress has been comparatively smooth.
Her worry isn't today's models but a sharp left turn once they can improve their own training.
Origin
Introduced by Nate Soares of the Machine Intelligence Research Institute in 'A central AI alignment problem: capabilities generalization, and the sharp left turn' (July 2022), posted to LessWrong and the AI Alignment Forum. The image is of a vehicle veering abruptly off its expected path.
Usage note
Related to fast takeofffast takeoff, n.: A scenario in which AI goes from about human level to far beyond it very quickly, in days, weeks or a few months, leaving people little time to react. and the treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it., but distinct: the sharp left turn concerns alignment failing to generalize during a capability jump, not necessarily deception or a fast takeoff.
Related
- goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine.
- fast takeofffast takeoff, n.: A scenario in which AI goes from about human level to far beyond it very quickly, in days, weeks or a few months, leaving people little time to react.
- treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it.
- inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training.
- recursive self-improvementrecursive self-improvement, n.: An AI improving its own design or training, so that each improved version is better at making the next improvement. It is the engine usually imagined behind an intelligence explosion.
- shard theoryshard theory, n.: A theory that people and AIs develop values as many small, situation-triggered habits shaped by past rewards, rather than as one fixed goal pursued single-mindedly.
Sources
- Soares, N. (2022). A central AI alignment problem: capabilities generalization, and the sharp left turn. AI Alignment Forum.
- Pope, Q. (2023). Evolution provides no evidence for the sharp left turn. LessWrong.
Mentioned in
fast takeofffast takeoff, n.: A scenario in which AI goes from about human level to far beyond it very quickly, in days, weeks or a few months, leaving people little time to react.