causal tracing
In plain wordsA test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled.
A form of activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer. devised to locate where a language model recalls factual associations. The model is run on a factual prompt such as 'The Space Needle is in downtown' three ways: once cleanly; once with Gaussian noise added to the embeddings of the subject tokens, which destroys the answer; and repeatedly with the noise but with a single hidden state — one layer at one token position — restored to its clean value. The fraction of the correct answer's probability recovered measures that state's indirect effect on the prediction. Applied to GPT-2 XL and GPT-J, causal tracing highlighted middle-layer MLP modules at the last subject token, motivating the ROMEROME, abbr.: A method for rewriting a single fact stored in a language model — say, moving the Eiffel Tower to Rome — by making one small, precisely calculated change to one layer's weights. editing method. Peter Hase and colleagues (2023) found that where causal tracing localizes a fact predicts poorly which layer is best to edit, cautioning against equating localization with storage.
Causal tracing lights up the MLPs at the last subject token in layers 15 to 20.
Origin
Introduced by Kevin Meng, David Bau, Alex Andonian and Yonatan Belinkov in 'Locating and Editing Factual Associations in GPT' (February 2022; NeurIPS 2022), adapting causal mediation analysis.
Related
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- ROMEROME, abbr.: A method for rewriting a single fact stored in a language model — say, moving the Eiffel Tower to Rome — by making one small, precisely calculated change to one layer's weights.
- model editingmodel editing, n.: Changing one specific fact or behavior inside a trained AI model — for example, who leads a country — by directly adjusting a small part of it, without retraining the whole thing.
- causal mediation analysiscausal mediation analysis, n.: A statistical method for splitting a cause's effect into the part that flows through some go-between and the part that does not; in AI it is used to find which internal parts carry a behavior.
- knowledge neuronknowledge neuron, n.: An internal unit of a language model claimed to be responsible for storing a particular fact, such that turning it up or down makes the model more or less likely to state that fact.
- parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.
Sources
- Meng, K., Bau, D., Andonian, A. & Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. NeurIPS 2022. arXiv:2202.05262. arxiv.org/abs/2202.05262
- Hase, P., Bansal, M., Kim, B. & Ghandeharioun, A. (2023). Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models. NeurIPS 2023.
Mentioned in
model editingmodel editing, n.: Changing one specific fact or behavior inside a trained AI model — for example, who leads a country — by directly adjusting a small part of it, without retraining the whole thing., activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer., causal mediation analysiscausal mediation analysis, n.: A statistical method for splitting a cause's effect into the part that flows through some go-between and the part that does not; in AI it is used to find which internal parts carry a behavior., knowledge neuronknowledge neuron, n.: An internal unit of a language model claimed to be responsible for storing a particular fact, such that turning it up or down makes the model more or less likely to state that fact., MEMITMEMIT, abbr.: A method for rewriting thousands of facts stored in a language model at once, by spreading small, calculated weight changes across several of its layers., ROMEROME, abbr.: A method for rewriting a single fact stored in a language model — say, moving the Eiffel Tower to Rome — by making one small, precisely calculated change to one layer's weights.