ablation
In plain wordsRemoving or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network.
- 1
Interpretability
An intervention that removes the contribution of a model component — a neuron, attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns., layer, feature or direction — during the forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction. and measures the effect on behavior, to test whether the component is necessary. Variants differ in what replaces the activation: zero ablation sets it to zero; mean ablation substitutes its average over a reference dataset; resample ablation substitutes its value on another, randomly chosen input. Because zeroing can push activations far outside their normal range, mean and resample ablation are usually preferred. Ablation can understate a component's role when other components compensate — self-repair, as with the backup name movers of the indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model. circuit — and overstate it when the ablation itself disrupts the network. Ablating a single direction, such as a refusal directionrefusal direction, n.: A single direction inside a chat model's internal activity that largely decides whether it refuses a request. Erasing it makes the model comply with almost anything; adding it makes the model refuse harmless requests., is also used to edit or steer models.
Mean-ablating head 9.6 drops task accuracy sharply, but backup heads recover part of it.
- 2
Experimental methodology
Short for ablation study: an experiment that removes or replaces one part of a learning system — a loss term, data source, architectural module or training trick — while holding the rest fixed, to measure that part's contribution. Ablation tables are a standard section of machine-learning papers.
Origin
From Latin ablatio, 'removal', via surgery and neuroscience, where lesioning brain tissue to study its function dates to the 19th century. Paul Cohen and Adele Howe (1988) credited Allen Newell with bringing the term into AI in his 1974 tutorial on speech understanding systems (published 1975), as a way to assess the contribution of components of complex systems.
Usage note
'Ablation' and 'knockout' are used interchangeably in interpretability; 'lesion' is the neuroscience term. A directional ablation removes one direction from activations rather than a whole component.
Related
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- causal scrubbingcausal scrubbing, n.: A strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before.
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
- refusal directionrefusal direction, n.: A single direction inside a chat model's internal activity that largely decides whether it refuses a request. Erasing it makes the model comply with almost anything; adding it makes the model refuse harmless requests.
- concept erasureconcept erasure, n.: Removing a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it.
- indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model.
Sources
- Newell, A. (1975). A Tutorial on Speech Understanding Systems. In D. R. Reddy (ed.), Speech Recognition: Invited Papers Presented at the 1974 IEEE Symposium. Academic Press.
- Cohen, P. R. & Howe, A. E. (1988). How Evaluation Guides AI Research. AI Magazine 9(4).
Mentioned in
mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value., probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere., reproducibilityreproducibility, n.: Whether other people, or the same people later, can get the same results by rerunning an AI experiment. Many published AI results have proved hard to reproduce, which makes real progress harder to judge., activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer., attention-based explanationattention-based explanation, n.: Explaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output., causal scrubbingcausal scrubbing, n.: A strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before., directional ablationdirectional ablation, n.: Erasing one specific direction from an AI model's internal signals while it runs, so whatever that direction encodes — such as the tendency to refuse — can no longer influence what the model does., occlusion sensitivityocclusion sensitivity, n.: Finding out which parts of an image matter to a model by covering them up one patch at a time and watching how much the model's answer changes., TransformerLensTransformerLens, prop. n.: A free software library that lets researchers open up language models and record, inspect or alter what happens inside them at every step while the model runs.