residual stream
In plain wordsThe running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
In a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., the vector carried along each tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. position by the residual connectionsresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable.. It starts as the token's embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.; every attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns. and MLPmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model. layer reads it (after layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots.) and adds its output back; and the final value is mapped to logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. by the unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores.. Because each component writes additively, the stream acts as a shared communication channel: the final state equals the embedding plus the sum of every component's output, which makes decompositions such as direct logit attributiondirect logit attribution, n.: A quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions. and the logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time. possible. Its width, d_model (768 in the smallest GPT-2), is far smaller than the number of features it carries, so information is stored in superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read., and components communicate through learned subspaces rather than individual dimensions, since the stream lacks a privileged basisprivileged basis, n.: A property of a layer in a neural network where the individual numbered slots, not just blends of them, carry special meaning, because something in the design treats each slot separately..
The country feature appears in the residual stream at layer 6 and is read by heads in layer 9.
Origin
Introduced, as far as is known, by Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Chris Olah and colleagues at Anthropic in 'A Mathematical Framework for Transformer Circuits' (December 2021), which recast the transformer's residual connections as a stream that each layer reads from and writes to. That paper is the standard source for the term, which quickly became universal in mechanistic interpretability.
Usage note
Not the same as a residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable., the architectural shortcut; the residual stream is the vector those shortcuts carry. In pre-norm transformers, normalization is applied on the way into each sublayer, so the stream itself is normalized only at the end.
Related
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time.
- direct logit attributiondirect logit attribution, n.: A quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions.
- superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
- attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
- activationactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment.
- privileged basisprivileged basis, n.: A property of a layer in a neural network where the individual numbered slots, not just blends of them, carry special meaning, because something in the design treats each slot separately.
- residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable.
Sources
- Elhage, N., Nanda, N., Olsson, C., Henighan, T. et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.
Mentioned in
layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage., activationactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model., activation vectoractivation vector, n.: The list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior., attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns., feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge., hidden layerhidden layer, n.: Any layer of a neural network between the input and the output. Its values are never shown to the user; they are the network's internal working-out., layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots., residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable., steeringsteering, n.: Guiding what an AI model does — its topic, tone or behavior — while it runs, by changing its instructions, its internal signals or how its next word is chosen, rather than by retraining it., streamingstreaming, n.: Sending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end., transformer blocktransformer block, n.: The repeating building unit of a transformer. A model is a stack of dozens of identical blocks, each pairing an attention step that mixes information between words with a step that processes each word., abliterationabliteration, n.: A technique for removing a chat model's ability to refuse requests by finding the internal direction responsible for refusals and permanently erasing it from the model's weights, with no retraining., activation additionactivation addition, n.: A simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes., activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions., attention-based explanationattention-based explanation, n.: Explaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output., concept injectionconcept injection, n.: An experiment that plants a specific concept directly into an AI model's internal activity, then asks the model whether it notices anything unusual, to test whether its reports about itself are genuine., contrastive activation additioncontrastive activation addition, n.: A way to make a steering signal more reliable by averaging the difference in a model's internal state across hundreds of example answers that do and don't show a behavior, such as flattery., crosscodercrosscoder, n.: A concept-finding tool that looks at several layers, or several versions of a model, at once, so that one shared concept can be tracked wherever it appears., direct logit attributiondirect logit attribution, n.: A quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions., function vectorfunction vector, n.: A compact internal signal, found inside a language model, that stands for a whole task — such as 'translate into French' or 'give the opposite word' — and can trigger that task when added to the model., latent reasoninglatent reasoning, n.: Getting an AI to do its step-by-step thinking inside its internal numbers rather than writing it out in words — potentially faster and more flexible, but much harder for people to inspect., logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time., persona vectorpersona vector, n.: A direction inside a language model's internal activity linked to a character trait, such as being flattering, malicious or prone to making things up, used to watch for or control that trait.