directional ablation
In plain wordsErasing one specific direction from an AI model's internal signals while it runs, so whatever that direction encodes — such as the tendency to refuse — can no longer influence what the model does.
An intervention that removes the component of a model's activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. along a given unit direction r, replacing each activation h with h − (r·h)r at every layer and token position, so that downstream computation cannot read that direction. It is the subtractive counterpart of activation additionactivation addition, n.: A simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes. and the standard causal test of whether a direction found by probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere. or contrastive means is necessary for a behavior: if ablating it removes the behavior, the direction mediates it. Applied to the refusal directionrefusal direction, n.: A single direction inside a chat model's internal activity that largely decides whether it refuses a request. Erasing it makes the model comply with almost anything; adding it makes the model refuse harmless requests. it disables refusal; folding the same projection into the weights gives abliterationabliteration, n.: A technique for removing a chat model's ability to refuse requests by finding the internal direction responsible for refusals and permanently erasing it from the model's weights, with no retraining.. Linear concept erasureconcept erasure, n.: Removing a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it. methods generalize it to whole subspaces, chosen so that no linear classifier can recover a concept.
Directional ablation of the truth direction dropped probe accuracy to chance but barely changed the answers.
Origin
Projecting a direction out of a representation is standard linear algebra, used to remove gender directions from word embeddings (Tolga Bolukbasi and colleagues, 2016) and in iterative nullspace projectioniterative nullspace projection, n.: An early method for removing a trait such as gender from a model's internal representations by repeatedly training a simple detector for it and then flattening away whatever the detector relied on. (2020). The name 'directional ablation' for this activation-space intervention was popularized by Arditi et al.'s work on refusal (2024).
Usage note
Distinguish from zero or mean ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network. of whole neurons or components: directional ablation removes one direction and leaves the rest of the activation intact. Later layers may still recover the information nonlinearly, so the test establishes linear, not total, removal.
Related
- refusal directionrefusal direction, n.: A single direction inside a chat model's internal activity that largely decides whether it refuses a request. Erasing it makes the model comply with almost anything; adding it makes the model refuse harmless requests.
- abliterationabliteration, n.: A technique for removing a chat model's ability to refuse requests by finding the internal direction responsible for refusals and permanently erasing it from the model's weights, with no retraining.
- ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network.
- concept erasureconcept erasure, n.: Removing a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it.
- activation additionactivation addition, n.: A simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes.
- steering vectorsteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it.
- activation additionactivation addition, n.: A simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes.
- ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network.
Mentioned in
activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model., abliterationabliteration, n.: A technique for removing a chat model's ability to refuse requests by finding the internal direction responsible for refusals and permanently erasing it from the model's weights, with no retraining., activation additionactivation addition, n.: A simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes., concept erasureconcept erasure, n.: Removing a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it., iterative nullspace projectioniterative nullspace projection, n.: An early method for removing a trait such as gender from a model's internal representations by repeatedly training a simple detector for it and then flattening away whatever the detector relied on., LEACELEACE, abbr.: A one-step mathematical method for scrubbing a concept, such as gender, from an AI model's internal data so that no simple detector can find it, while changing everything else as little as possible., refusal directionrefusal direction, n.: A single direction inside a chat model's internal activity that largely decides whether it refuses a request. Erasing it makes the model comply with almost anything; adding it makes the model refuse harmless requests.