induction head
In plain wordsA pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt.
A two-part circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name. in transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. that continues repeated sequences: given a context containing … A B … and a later occurrence of A, it attends to the token that followed the earlier A and copies B to the output. It works by composing two attention headsattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.. A previous-token head in an earlier layer writes into each position information about the token before it; the induction head's query for the current token A then matches keys carrying 'preceded by A' (K-composition), and its OV circuitQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees. copies the attended token. Induction heads need at least two layers and usually emerge suddenly early in training; their arrival coincides with a small bump in the loss curve and a sharp gain in in-context learningin-context learning, n.: A language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round.. Catherine Olsson and colleagues argued that they drive much of in-context learning, with causal support in small attention-only models and only correlational support in larger ones. Fuzzy variants match similar rather than identical tokens.
Ablating the two induction heads wiped out the model's ability to continue repeated random tokens.
Origin
Described by Nelson Elhage, Neel Nanda, Catherine Olsson and colleagues in 'A Mathematical Framework for Transformer Circuits' (Anthropic, December 2021) and studied in depth in Olsson and colleagues' 'In-context Learning and Induction Heads' (March 2022).
Related
- in-context learningin-context learning, n.: A language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round.
- attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
- QK and OV circuitsQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees.
- attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map.
- mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
Sources
- Elhage, N. et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.
- Olsson, C., Elhage, N., Nanda, N. et al. (2022). In-context Learning and Induction Heads. Transformer Circuits Thread.
Mentioned in
mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns., Chris OlahChris Olah, prop. n.: A Canadian researcher who pioneered ways of looking inside neural networks to see what they have learned, and a co-founder of the AI company Anthropic., circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name., in-context learningin-context learning, n.: A language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round., inductive reasoninginductive reasoning, n.: Reasoning from particular examples to a general rule or prediction — every swan I have seen is white, so swans are probably white — which is useful but can always turn out wrong., attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map., developmental interpretabilitydevelopmental interpretability, n.: The study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model., Neel NandaNeel Nanda, prop. n.: A British AI researcher who leads mechanistic interpretability at Google DeepMind and built TransformerLens, a widely used tool for looking inside language models., QK and OV circuitsQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees., Transformer Circuits ThreadTransformer Circuits Thread, prop. n.: Anthropic's online publication for research on how language models work inside, presented as long interactive web articles rather than conventional papers., universality hypothesisuniversality hypothesis, n.: The idea that different AI models trained on similar tasks tend to develop the same internal concepts and mechanisms, so what is learned about one model may carry over to others.