Skip to content
Last Dictionary
Plane
Interpretability & SteeringLanguage Models & Transformers

attribution patching

nounAlso: AtP, AtP*

In plain wordsA fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part.

  1. A gradient-based approximation to activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer. that estimates the effect of patching every component simultaneously. For each activation a, the change in a metric M from substituting its clean value for its corrupted one is approximated to first order by the difference between the two values multiplied by the gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill. of M with respect to a. Two forward passes and one backward pass thus replace one forward pass per patched component, making it feasible to scan every head, neuron or sparse feature of a large model. The approximation fails where M is strongly nonlinear in the activation — for example when attention saturates or a component's effect is all-or-nothing — producing false negatives; AtP* (János Kramár and colleagues, 2024) adds corrections for attention and for cancellation between paths. Edge attribution patching applies the same idea to connections between components for automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand..

    Attribution patching ranked all 2,000 heads in a minute; we then verified the top ten with real patching.

ΔM≈(aclean−acorr)⊤∂M∂a∣a=acorr\Delta M \approx \left(a_{\text{clean}} - a_{\text{corr}}\right)^{\top} \left. \frac{\partial M}{\partial a} \right|_{a = a_{\text{corr}}}

Origin

Described by Neel Nanda in the blog post 'Attribution Patching: Activation Patching At Industrial Scale' (February 2023), from work begun at Anthropic for which he shares credit with Chris Olah, Catherine Olsson, Nelson Elhage and Tristan Hume; it draws on gradient-times-input attribution. Extended to edges by Aaquib Syed, Can Rager and Arthur Conmy (2023) and refined by Kramár, Tom Lieberum, Rohin Shah and Nanda at Google DeepMind (AtP*, 2024).

Related

See also
  • activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
  • automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand.
  • integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way.
  • feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
  • path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior.

Sources

  1. Kramár, J., Lieberum, T., Shah, R. & Nanda, N. (2024). AtP*: An efficient and scalable method for localizing LLM behaviour to components. arXiv preprint.
  2. Syed, A., Rager, C. & Conmy, A. (2023). Attribution Patching Outperforms Automated Circuit Discovery. arXiv preprint.

Mentioned in

feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer., automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand., direct logit attributiondirect logit attribution, n.: A quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions., Neel NandaNeel Nanda, prop. n.: A British AI researcher who leads mechanistic interpretability at Google DeepMind and built TransformerLens, a widely used tool for looking inside language models., path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior., sparse feature circuitsparse feature circuit, n.: A map of how a language model performs a task, drawn in terms of understandable concepts found by concept-finding tools, rather than hard-to-read neurons or whole layers.