causal abstraction
In plain wordsA mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped.
A framework, from the theory of causal models, for stating when a high-level causal model — such as a symbolic algorithm with interpretable variables — faithfully describes a lower-level one, such as a neural network. The high-level model is an abstraction of the low-level one if there is a mapping from low-level states to high-level variables under which interventions commute: intervening on the network's activations that are aligned with a variable has the same effect on the output as intervening on that variable in the algorithm. In interpretability this is tested with interchange interventions — setting a network component to the value it takes on another input, a form of activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer. — and summarized by interchange intervention accuracy. Causal abstraction provides a common language for circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name. analysis, causal scrubbingcausal scrubbing, n.: A strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before., path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior. and distributed alignment searchdistributed alignment search, n.: A method that searches, by training, for a hidden direction inside a neural network that plays the role of one variable in a human-readable explanation of the network's behavior., and makes precise what it means for an interpretation to be faithful.
Under causal abstraction, the network implements the carry-the-one algorithm only if swapping the aligned neurons swaps the carry.
Origin
Formalized in causal inference by Paul Rubenstein and colleagues (2017) and by Sander Beckers and Joseph Halpern (2019). Atticus Geiger, Hanson Lu, Thomas Icard and Christopher Potts applied it to neural networks in 'Causal Abstractions of Neural Networks' (NeurIPS 2021), and Geiger and colleagues later proposed it as a theoretical foundation for mechanistic interpretability (2023).
Related
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- distributed alignment searchdistributed alignment search, n.: A method that searches, by training, for a hidden direction inside a neural network that plays the role of one variable in a human-readable explanation of the network's behavior.
- causal scrubbingcausal scrubbing, n.: A strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before.
- causal inferencecausal inference, n.: Working out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time.
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
- interpretability illusioninterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow.
Sources
- Geiger, A., Lu, H., Icard, T. & Potts, C. (2021). Causal Abstractions of Neural Networks. NeurIPS 2021.
- Beckers, S. & Halpern, J. Y. (2019). Abstracting Causal Models. AAAI 2019.
- Rubenstein, P. K., Weichwald, S., Bongers, S., Mooij, J. M., Janzing, D., Grosse-Wentrup, M. & Schölkopf, B. (2017). Causal Consistency of Structural Equation Models. UAI 2017.
Mentioned in
causal inferencecausal inference, n.: Working out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time., activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer., causal mediation analysiscausal mediation analysis, n.: A statistical method for splitting a cause's effect into the part that flows through some go-between and the part that does not; in AI it is used to find which internal parts carry a behavior., causal scrubbingcausal scrubbing, n.: A strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before., distributed alignment searchdistributed alignment search, n.: A method that searches, by training, for a hidden direction inside a neural network that plays the role of one variable in a human-readable explanation of the network's behavior., representation fine-tuningrepresentation fine-tuning, n.: A way of adapting a language model to a new task by training small, targeted edits to its internal signals instead of changing its weights, often with far fewer adjustable numbers than other methods., structural causal modelstructural causal model, n.: A precise description of what causes what in a system — a diagram of arrows plus equations — that predicts what would happen if you intervened, not just what tends to occur together.