circuit tracing
In plain wordsMethods that map, step by step, which internal concepts an AI model used to produce a particular answer and how they fed into one another, like tracing wires on a circuit board.
The recovery of the circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name. by which a model produces a particular output; specifically, the method introduced by Anthropic in 2025. There, the model's MLP layers are replaced by a cross-layer transcodertranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes. to form a replacement modelreplacement model, n.: A more understandable copy of an AI model, built from simple labeled parts, that behaves almost like the original, so researchers can study the copy's workings instead. whose units are interpretable features. For a given prompt, attention patterns and normalization are frozen and the leftover reconstruction error is kept as 'error nodes', so the replacement reproduces the original output exactly. Direct linear effects between features, input tokens and output logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. then define an attribution graphattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next., which researchers prune, label and group into 'supernodes'; hypotheses read from the graph are checked by intervening on the original model, for example by suppressing or injecting features. Limitations include unexplained error, attention computations that are frozen rather than explained, and costs that grow with prompt length.
Circuit tracing showed the model choosing the rhyme word before writing the line that ends with it.
Origin
The method is described in Emmanuel Ameisen, Jack Lindsey, Adam Pearce and colleagues' 'Circuit Tracing: Revealing Computational Graphs in Language Models' (Transformer Circuits Thread, March 2025), applied in the companion paper 'On the Biology of a Large Language Model'. Anthropic released an open-source circuit-tracer library, with an interactive front end on Neuronpedia, in May 2025.
Related
- attribution graphattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next.
- replacement modelreplacement model, n.: A more understandable copy of an AI model, built from simple labeled parts, that behaves almost like the original, so researchers can study the copy's workings instead.
- transcodertranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes.
- model biologymodel biology, n.: The study of what AI models actually do inside — the strategies and hidden steps they use — approached the way biologists study living things, by observation and experiment rather than from a blueprint.
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
- NeuronpediaNeuronpedia, prop. n.: A free website where anyone can browse, search and test the internal concepts that researchers have found inside AI models, with examples of the text that switches each one on.
Sources
- Ameisen, E., Lindsey, J., Pearce, A. et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread.
- Lindsey, J., Gurnee, W., Ameisen, E. et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.
Mentioned in
interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., Chris OlahChris Olah, prop. n.: A Canadian researcher who pioneered ways of looking inside neural networks to see what they have learned, and a co-founder of the AI company Anthropic., circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name., attribution graphattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next., automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand., model biologymodel biology, n.: The study of what AI models actually do inside — the strategies and hidden steps they use — approached the way biologists study living things, by observation and experiment rather than from a blueprint., NeuronpediaNeuronpedia, prop. n.: A free website where anyone can browse, search and test the internal concepts that researchers have found inside AI models, with examples of the text that switches each one on., replacement modelreplacement model, n.: A more understandable copy of an AI model, built from simple labeled parts, that behaves almost like the original, so researchers can study the copy's workings instead., transcodertranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes., Transformer Circuits ThreadTransformer Circuits Thread, prop. n.: Anthropic's online publication for research on how language models work inside, presented as long interactive web articles rather than conventional papers.