linear representation hypothesis
In plain wordsThe idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.
The hypothesis that neural networks represent many featuresfeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity. as directions, or low-dimensional subspaces, in activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions., so that an activation is approximately a sum of feature directions each scaled by how strongly its feature is present. Two consequences give it practical force: features can be read out with a linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly. or a dot product, and they can be changed by adding or subtracting vectors, as in steeringsteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it. and concept erasureconcept erasure, n.: Removing a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it.. Supporting evidence includes word-vector analogies (king − man + woman ≈ queen), the success of probes and sparse autoencoderssparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept., and steering results. Kiho Park, Yo Joon Choe and Victor Veitch (2023) formalized versions of the hypothesis and argued that the appropriate inner product for measuring it is not always the Euclidean one. Counterexamples include circular representations of days of the week and months (Engels et al., 2024) and nonlinear 'onion' encodings in recurrent networks, so many researchers regard linearity as common rather than universal.
If the linear representation hypothesis holds here, one probe direction should track sentiment across every prompt.
Origin
Linear structure in word embeddings was popularized by Tomas Mikolov, Wen-tau Yih and Geoffrey Zweig's analogy results (2013). The phrase 'linear representation hypothesis' was in use in mechanistic interpretability by 2022, notably in Anthropic's 'Toy Models of Superposition', and was given formal treatment by Park, Choe and Veitch (2023; ICML 2024).
Usage note
Linear representation is not linear computation: networks compute features nonlinearly, but the hypothesis concerns how computed features are stored and read. Whether all important features, including those used in planning and reasoning, are linear remains open.
Related
- feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.
- superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
- linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly.
- steering vectorsteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it.
- word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text.
- activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions.
- concept erasureconcept erasure, n.: Removing a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it.
Sources
- Mikolov, T., Yih, W. & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. NAACL-HLT 2013.
- Park, K., Choe, Y. J. & Veitch, V. (2024). The Linear Representation Hypothesis and the Geometry of Large Language Models. ICML 2024.
- Engels, J., Michaud, E. J., Liao, I., Gurnee, W. & Tegmark, M. (2024). Not All Language Model Features Are Linear. arXiv preprint.
Mentioned in
vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points., activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model., feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity., linear algebralinear algebra, n.: The branch of mathematics about lists and grids of numbers and the even, straight-line transformations between them. It is the everyday language in which neural networks are written and computed., linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly., activation additionactivation addition, n.: A simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes., activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions., contrast-consistent searchcontrast-consistent search, n.: A method for finding where a language model internally tracks whether statements are true, using only the logical rule that a statement and its opposite cannot both be true., distributed alignment searchdistributed alignment search, n.: A method that searches, by training, for a hidden direction inside a neural network that plays the role of one variable in a human-readable explanation of the network's behavior., natural abstraction hypothesisnatural abstraction hypothesis, n.: The idea that the world naturally sorts into a limited set of useful concepts — like 'tree' or 'temperature' — that any capable mind, human or AI, would end up learning., Othello-GPTOthello-GPT, prop. n.: A small language model trained only on lists of moves from the board game Othello, which researchers found had built an internal picture of the board — evidence that such models can form world models., platonic representation hypothesisplatonic representation hypothesis, n.: The idea that bigger, more capable AI models, even ones trained on different things such as images and text, end up organizing information in increasingly similar ways, as if all were mapping the same underlying world., refusal directionrefusal direction, n.: A single direction inside a chat model's internal activity that largely decides whether it refuses a request. Erasing it makes the model comply with almost anything; adding it makes the model refuse harmless requests., steering vectorsteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it., testing with concept activation vectorstesting with concept activation vectors, phr.: A method for asking whether a human idea, such as 'wheels', matters to an image model's decision, such as 'car', by finding the direction for that idea inside the model and measuring its effect., Tomas MikolovTomas Mikolov, prop. n.: A Czech computer scientist who created word2vec, a fast way to turn words into lists of numbers that capture their meaning, and pioneered language models built from recurrent networks.