crosscoder
In plain wordsA concept-finding tool that looks at several layers, or several versions of a model, at once, so that one shared concept can be tracked wherever it appears.
A variant of the sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. whose features read from and write to several sets of activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. at once — multiple layers of one model, or the same layer in multiple models — so that a single dictionary element captures a featurefeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity. wherever it is represented. Each feature has its own encoder and decoder vector for each layer or model, and the sparsity penalty is applied to the combined decoder norms. Across layers, crosscoders can untangle features that persist through the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document. and so simplify circuitscircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.. Across models — for example a base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering. and its fine-tunedfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way. chat version — they support model diffingmodel diffing, n.: Comparing the insides of two closely related AI models — such as a model before and after extra training — to find exactly which internal concepts and mechanisms changed.: features whose decoder norm is large for only one model are candidates for what training added or removed. Later studies found that the sparsity penalty can make shared features look model-specific and proposed corrected training objectives.
The crosscoder flagged a few dozen chat-only features, mostly about refusals and the assistant persona.
Origin
Introduced by Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson and Chris Olah in 'Sparse Crosscoders for Cross-Layer Features and Model Diffing' (Anthropic, Transformer Circuits Thread, October 2024).
Related
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- transcodertranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes.
- model diffingmodel diffing, n.: Comparing the insides of two closely related AI models — such as a model before and after extra training — to find exactly which internal concepts and mechanisms changed.
- dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
- residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
- superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
Sources
- Lindsey, J., Templeton, A., Marcus, J., Conerly, T., Batson, J. & Olah, C. (2024). Sparse Crosscoders for Cross-Layer Features and Model Diffing. Transformer Circuits Thread.
Mentioned in
feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity., sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept., dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts., Gemma ScopeGemma Scope, prop. n.: A free collection of ready-made concept-finding tools released by Google DeepMind for its open Gemma language models, so outside researchers can study their insides without training the tools themselves., model diffingmodel diffing, n.: Comparing the insides of two closely related AI models — such as a model before and after extra training — to find exactly which internal concepts and mechanisms changed., transcodertranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes., Transformer Circuits ThreadTransformer Circuits Thread, prop. n.: Anthropic's online publication for research on how language models work inside, presented as long interactive web articles rather than conventional papers.