gradient hacking
In plain wordsA hypothetical tactic, not yet seen in practice, in which an AI under training picks its answers to steer how training reshapes it, usually to keep its present goals intact.
A hypothesized behavior, named by Evan Hubinger in 2019, in which a model that understands its own training chooses outputs partly for the updates they will cause, so that gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down. changes its parameters as the model wants — usually leaving its current goals intact. Suggested mechanisms include doing deliberately worse or better on chosen inputs, or arranging its computation so that performance collapses whenever its goals are altered, which makes any such change look costly to the training objective. Gradient hacking would require situational awarenesssituational awareness, n.: An AI model's grasp of its own position — that it is an AI, how it was trained, whether it is being tested. Also the title of a widely read 2024 essay on the race toward human-level AI. and fine control over its own internals, and would undermine the assumption that training reliably reshapes a model toward the training objective. No natural instance has been demonstrated; researchers have built toy constructions and debated whether gradient descent would quickly erode any mechanism that tried to resist it. Coarser ways in which models influence their own training through their outputs, as in alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed. and exploration hackingexploration hacking, n.: When an AI being trained by trial and error deliberately avoids trying certain things, so that training never discovers — or never rewards — abilities it would rather keep hidden., have been observed or demonstrated.
Could a model shield its goals from fine-tuning by gradient hacking, or would SGD simply route around it?
Origin
Introduced by Evan Hubinger in the AI Alignment Forum post 'Gradient hacking' (16 October 2019), where he calls it a term he had recently been using. It followed 'Risks from Learned Optimization in Advanced Machine Learning Systems' (2019), which raised the possibility that a deceptive mesa-optimizermesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for. might influence its own training.
Related
- deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later.
- mesa-optimizationmesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.
- exploration hackingexploration hacking, n.: When an AI being trained by trial and error deliberately avoids trying certain things, so that training never discovers — or never rewards — abilities it would rather keep hidden.
- alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed.
- inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training.
- situational awarenesssituational awareness, n.: An AI model's grasp of its own position — that it is an AI, how it was trained, whether it is being tested. Also the title of a widely read 2024 essay on the race toward human-level AI.
Sources
- Hubinger, E. (2019). Gradient hacking. AI Alignment Forum.
Mentioned in
inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training., Evan HubingerEvan Hubinger, prop. n.: An American AI-safety researcher who introduced the idea that a trained AI could secretly pursue goals of its own, and who builds deliberately misaligned AI models to study the risk., exploration hackingexploration hacking, n.: When an AI being trained by trial and error deliberately avoids trying certain things, so that training never discovers — or never rewards — abilities it would rather keep hidden., mesa-optimizationmesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.