toy model
In plain wordsA deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work.
A deliberately small or simplified neural network, trained on synthetic data or a narrow algorithmic task, used to study a phenomenon where it can be fully analyzed and then to form hypotheses about larger models. Examples in interpretability include ReLU networks reconstructing sparse synthetic features (in 'Toy Models of Superposition'); one- and two-layer attention-only transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.; small transformers trained on modular addition, whose learned algorithm based on discrete Fourier transforms and trigonometric identities was fully reverse-engineered in the study of grokkinggrokking, n.: When a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right.; and Othello-GPTOthello-GPT, prop. n.: A small language model trained only on lists of moves from the board game Othello, which researchers found had built an internal picture of the board — evidence that such models can form world models.. Because ground truth is known, toy models are ideal for validating methods such as sparse autoencoderssparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. and automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand.. Their main risk is unrepresentativeness: a mechanism seen in a small idealized setting may be absent, or look different, in production models.
Before scaling up, test the new SAE variant on a toy model where we know the true features.
Origin
Borrowed from physics, where toy models are idealized systems that isolate one effect; the phrase has long been used informally in machine learning. In interpretability it was popularized by Nelson Elhage and colleagues' 'Toy Models of Superposition' (Anthropic, 2022) and Neel Nanda and colleagues' 'Progress Measures for Grokking via Mechanistic Interpretability' (2023).
Related
- superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
- grokkinggrokking, n.: When a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right.
- Othello-GPTOthello-GPT, prop. n.: A small language model trained only on lists of moves from the board game Othello, which researchers found had built an internal picture of the board — evidence that such models can form world models.
- mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
- universality hypothesisuniversality hypothesis, n.: The idea that different AI models trained on similar tasks tend to develop the same internal concepts and mechanisms, so what is learned about one model may carry over to others.
- model organism of misalignmentmodel organism of misalignment, n.: An AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes.
Sources
- Elhage, N. et al. (2022). Toy Models of Superposition. Transformer Circuits Thread. transformer-circuits.pub/2022/toy_model/index.html
- Nanda, N., Chan, L., Lieberum, T., Smith, J. & Steinhardt, J. (2023). Progress Measures for Grokking via Mechanistic Interpretability. ICLR 2023.
Mentioned in
superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read., Othello-GPTOthello-GPT, prop. n.: A small language model trained only on lists of moves from the board game Othello, which researchers found had built an internal picture of the board — evidence that such models can form world models., parameter decompositionparameter decomposition, n.: An approach that tries to understand a neural network by splitting its weights — rather than its internal activity — into many simple pieces, each responsible for one mechanism., universality hypothesisuniversality hypothesis, n.: The idea that different AI models trained on similar tasks tend to develop the same internal concepts and mechanisms, so what is learned about one model may carry over to others.