superposition
In plain wordsThe way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
The phenomenon, and hypothesis, that a network represents more featuresfeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity. than it has dimensions by assigning them nearly orthogonal but overlapping directions in activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions.. In high-dimensional spaces exponentially many directions can be almost orthogonal, so if features are sparse — rarely active at the same time — interference between them stays small and can be filtered out by nonlinearities. Anthropic's 'Toy Models of Superposition' (2022) demonstrated this in small ReLU networks: superposition switches on as feature sparsity increases, features arrange themselves into geometric configurations such as digons, triangles and pentagons, and the transitions between regimes resemble phase changes. Superposition explains much polysemanticitypolysemanticity, n.: When a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does., defeats neuron-by-neuron analysis and motivates dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts. with sparse autoencoderssparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept., which try to recover the individual features from the compressed representation.
The model tracks thousands of features in a 512-dimensional layer, so superposition is unavoidable.
Origin
Sanjeev Arora and colleagues (2018) showed that word embeddings store the senses of ambiguous words as linear superpositions. Chris Olah and colleagues' 'Zoom In' (Distill, March 2020) named 'superposition' — a circuit spreading a feature across many polysemantic neurons — and Nelson Elhage and colleagues' 'Toy Models of Superposition' (Anthropic, September 2022) gave the first systematic study.
Usage note
Named by analogy with superposition in physics, not by any quantum mechanism. How closely the superposition in large production models matches the toy-model picture is still debated.
Related
- polysemanticitypolysemanticity, n.: When a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does.
- feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
- toy modeltoy model, n.: A deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work.
- privileged basisprivileged basis, n.: A property of a layer in a neural network where the individual numbered slots, not just blends of them, carry special meaning, because something in the design treats each slot separately.
- monosemanticitymonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several.
Sources
- Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T. et al. (2022). Toy Models of Superposition. Transformer Circuits Thread. transformer-circuits.pub/2022/toy_model/index.html
- Arora, S., Li, Y., Liang, Y., Ma, T. & Risteski, A. (2018). Linear Algebraic Structure of Word Senses, with Applications to Polysemy. Transactions of the ACL 6.
Mentioned in
mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., Chris OlahChris Olah, prop. n.: A Canadian researcher who pioneered ways of looking inside neural networks to see what they have learned, and a co-founder of the AI company Anthropic., featurefeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself., feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity., polysemanticitypolysemanticity, n.: When a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does., residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document., sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept., activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions., crosscodercrosscoder, n.: A concept-finding tool that looks at several layers, or several versions of a model, at once, so that one shared concept can be tracked wherever it appears., dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts., enumerative safetyenumerative safety, n.: A proposed safety strategy: list every concept an AI model represents internally, then check the list for dangerous ones, such as deception, before trusting the model., Johnson–Lindenstrauss lemmaJohnson–Lindenstrauss lemma, n.: A surprising fact about high-dimensional data: a random projection can squash a large set of points into far fewer dimensions while distorting every distance between them only slightly., linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction., monosemanticitymonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several., orthogonalityorthogonality, n.: The mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map., parameter decompositionparameter decomposition, n.: An approach that tries to understand a neural network by splitting its weights — rather than its internal activity — into many simple pieces, each responsible for one mechanism., privileged basisprivileged basis, n.: A property of a layer in a neural network where the individual numbered slots, not just blends of them, carry special meaning, because something in the design treats each slot separately., toy modeltoy model, n.: A deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work., Transformer Circuits ThreadTransformer Circuits Thread, prop. n.: Anthropic's online publication for research on how language models work inside, presented as long interactive web articles rather than conventional papers.