Transformer Circuits Thread

A web publication, hosted at transformer-circuits.pub, in which Anthropic's interpretability team publishes its research on the mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior. of transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. language models, usually as long interactive articles released outside conventional peer review, together with shorter periodic 'Circuits Updates'. Its papers introduced much of the field's working vocabulary: the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document. framing, QK and OV circuitsQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees. and induction headsinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt. (2021–2022); superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read. in toy models (2022); sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts. and monosemanticitymonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several. (2023), scaled to a production model in 2024; crosscoderscrosscoder, n.: A concept-finding tool that looks at several layers, or several versions of a model, at once, so that one shared concept can be tracked wherever it appears. and model diffingmodel diffing, n.: Comparing the insides of two closely related AI models — such as a model before and after extra training — to find exactly which internal concepts and mechanisms changed. (2024); and circuit tracingcircuit tracing, n.: Methods that map, step by step, which internal concepts an AI model used to produce a particular answer and how they fed into one another, like tracing wires on a circuit board. with attribution graphsattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next. (2025). Its format and research program continue the Circuits thread begun in DistillDistill, prop. n.: An online scientific journal (2016–2021) that published clear, interactive explanations of machine learning research, including founding work on understanding what happens inside neural networks..

The new attribution-graph paper went up on the Transformer Circuits Thread before it reached arXiv.