causal mediation analysis
In plain wordsA statistical method for splitting a cause's effect into the part that flows through some go-between and the part that does not; in AI it is used to find which internal parts carry a behavior.
- 1
Statistics
A family of methods for decomposing the effect of a treatment on an outcome into an indirect effect, transmitted through an intermediate variable (the mediator), and a direct effect through all other paths. Modern definitions are counterfactual: the natural indirect effect is the change in the outcome when the mediator is set to the value it would take under treatment while the treatment itself is held at its control value. Identifying these effects from data requires strong assumptions, notably no unmeasured confounding of the mediator and outcome.
- 2
Interpretability
The application of the same decomposition to a neural network, treating a change to the input as the treatment, a neuron, attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns. or layer as the mediator and the model's output as the outcome. The indirect effect is estimated by setting the mediator to the value it takes under the changed input, which is activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.. Jesse Vig and colleagues used it in 2020 to locate the components of GPT-2 that carry gender bias, and it underlies causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled..
Causal mediation analysis put most of the gender-bias effect in a few heads in the middle layers.
Origin
Mediation analysis was popularized in psychology by Reuben Baron and David Kenny (1986); counterfactual definitions of direct and indirect effects were given by James Robins and Sander Greenland (1992) and Judea Pearl (2001). Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov and colleagues brought it to neural-network interpretability (NeurIPS 2020).
Related
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled.
- causal inferencecausal inference, n.: Working out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time.
- counterfactualcounterfactual, n.: A statement about what would have happened if something had been different — 'had I left earlier, I would have caught the train'. AI uses counterfactuals to explain decisions and to measure causes.
- causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped.
Sources
- Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y. & Shieber, S. (2020). Investigating Gender Bias in Language Models Using Causal Mediation Analysis. NeurIPS 2020.
- Pearl, J. (2001). Direct and Indirect Effects. Proceedings of UAI 2001.
Mentioned in
activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer., causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled.