indirect object identification
In plain wordsA fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model.
A sentence-completion task, and the circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name. that solves it in GPT-2 small, used as a reference case in mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.. Prompts introduce two people and then repeat one as the subject of a giving clause; the correct completion is the other name, the indirect object. Kevin Wang and colleagues reverse-engineered the mechanism with path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior., identifying 26 attention headsattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns. in seven classes, among them duplicate-token heads that notice the repeated name, S-inhibition heads that suppress attention to it, and name-mover heads that copy the remaining name to the output — plus backup name movers that take over when the main ones are ablated and negative name movers that push the other way. The study proposed faithfulness, completeness and minimality as criteria for circuit explanations, and the task became a standard test for automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand. and other methods.
We reused the IOI prompts to check whether the new attribution method finds the name movers.
Origin
Introduced by Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris and Jacob Steinhardt in 'Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small' (November 2022; ICLR 2023), work begun at Redwood Research. 'Indirect object' is a term of traditional grammar.
Related
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
- path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior.
- automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand.
- attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
- GPT-2GPT-2, prop. n.: A 2019 OpenAI language model that wrote surprisingly fluent paragraphs. OpenAI at first withheld the full version, saying it could be misused for fake news, which made it famous.
- mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
Sources
- Wang, K., Variengien, A., Conmy, A., Shlegeris, B. & Steinhardt, J. (2023). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. ICLR 2023.
Mentioned in
mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network., circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name., automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand., Buck ShlegerisBuck Shlegeris, prop. n.: An AI-safety researcher who runs Redwood Research and helped found 'AI control', the study of keeping powerful AI safe even if it is secretly trying to cause harm., direct logit attributiondirect logit attribution, n.: A quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions., path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior.