attribution graph
In plain wordsA diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next.
A directed graph, computed for a single prompt, whose nodes are input tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., interpretable features of a replacement modelreplacement model, n.: A more understandable copy of an AI model, built from simple labeled parts, that behaves almost like the original, so researchers can study the copy's workings instead., error terms and output logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity., and whose weighted edges give the direct linear contribution of each node to the next. Because the replacement's attention patterns and normalizations are frozen for that prompt, the remaining interactions are linear, so edge weights can be computed exactly from activations and weights. Graphs are pruned to their most influential nodes and explored visually; researchers group related features into supernodes and test the resulting story by intervening on the original model. Attribution graphs have exposed two-step factual recall (Dallas → Texas → Austin), rhyme words planned ahead of a line of verse, parallel pathways in mental arithmetic, and features involved in hallucinationshallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true. and refusals. Each graph explains one prompt, not the model's general algorithm.
The attribution graph shows a 'Texas' feature sitting between 'Dallas' and the answer 'Austin'.
Origin
Introduced in Anthropic's 'Circuit Tracing' methods paper and its companion 'On the Biology of a Large Language Model' (both March 2025), which applied attribution graphs to Claude 3.5 Haiku. Graphs of attributions between sparse features had precedents such as Samuel Marks and colleagues' sparse feature circuits (2024).
Related
- circuit tracingcircuit tracing, n.: Methods that map, step by step, which internal concepts an AI model used to produce a particular answer and how they fed into one another, like tracing wires on a circuit board.
- replacement modelreplacement model, n.: A more understandable copy of an AI model, built from simple labeled parts, that behaves almost like the original, so researchers can study the copy's workings instead.
- transcodertranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes.
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
- model biologymodel biology, n.: The study of what AI models actually do inside — the strategies and hidden steps they use — approached the way biologists study living things, by observation and experiment rather than from a blueprint.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.
Sources
- Ameisen, E., Lindsey, J., Pearce, A. et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread.
- Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D. & Mueller, A. (2024). Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. arXiv preprint; ICLR 2025.
Mentioned in
mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., Chris OlahChris Olah, prop. n.: A Canadian researcher who pioneered ways of looking inside neural networks to see what they have learned, and a co-founder of the AI company Anthropic., circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name., feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., circuit tracingcircuit tracing, n.: Methods that map, step by step, which internal concepts an AI model used to produce a particular answer and how they fed into one another, like tracing wires on a circuit board., model biologymodel biology, n.: The study of what AI models actually do inside — the strategies and hidden steps they use — approached the way biologists study living things, by observation and experiment rather than from a blueprint., NeuronpediaNeuronpedia, prop. n.: A free website where anyone can browse, search and test the internal concepts that researchers have found inside AI models, with examples of the text that switches each one on., replacement modelreplacement model, n.: A more understandable copy of an AI model, built from simple labeled parts, that behaves almost like the original, so researchers can study the copy's workings instead., sparse feature circuitsparse feature circuit, n.: A map of how a language model performs a task, drawn in terms of understandable concepts found by concept-finding tools, rather than hard-to-read neurons or whole layers., transcodertranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes., Transformer Circuits ThreadTransformer Circuits Thread, prop. n.: Anthropic's online publication for research on how language models work inside, presented as long interactive web articles rather than conventional papers.