Skip to content
Last Dictionary
Plane
Interpretability & SteeringLanguage Models & TransformersAgents & Harnesses

automated interpretability

nounAlso: auto-interp, autointerp, automatic interpretability, interpretability agent

In plain wordsUsing AI models to help explain other AI models — for instance, having a language model read examples that excite an internal unit and write a description of what it detects.

  1. The use of machine-learning systems, usually language models, to carry out interpretability work at scale. The basic pipeline explains a unit — a neuron or sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. feature — by showing an explainer model the text excerpts that most strongly activate it and asking for a short natural-language description. The description is then scored by having a simulator model predict the unit's activations on new text from the description alone and correlating the predictions with the real activations; other scores test whether the description lets a model pick out activating examples or steer the feature. This made it practical to label millions of features. More recent work builds interpretability agents that design and run their own experiments — querying, ablating and steering units — and uses them to audit models for hidden objectives. Explanations often describe only a unit's strongest activations and can be confidently wrong.

    Auto-interp labeled the latent 'legal citations', but its simulation score was only 0.3.

Origin

Pioneered by Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu and William Saunders at OpenAI in 'Language models can explain neurons in language models' (May 2023), which used GPT-4 to explain all 307,200 neurons of GPT-2 XL. Earlier, MILAN (Evan Hernandez and colleagues, 2022) described vision neurons in words; interpretability agents that run their own experiments followed in Sarah Schwettmann and colleagues' FIND benchmark (2023) and MAIA (2024).

Related

See also
  • sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
  • feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.
  • NeuronpediaNeuronpedia, prop. n.: A free website where anyone can browse, search and test the internal concepts that researchers have found inside AI models, with examples of the text that switches each one on.
  • monosemanticitymonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several.
  • AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
  • LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.

Sources

  1. Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J. & Saunders, W. (2023). Language models can explain neurons in language models. OpenAI.
  2. Hernandez, E., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A. & Andreas, J. (2022). Natural Language Descriptions of Deep Visual Features. ICLR 2022.
  3. Schwettmann, S., Rott Shaham, T., Materzynska, J., Chowdhury, N., Li, S., Andreas, J., Bau, D. & Torralba, A. (2023). FIND: A Function Description Benchmark for Evaluating Interpretability Methods. arXiv:2309.03886. arxiv.org/abs/2309.03886

Mentioned in

feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity., dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts., Jan LeikeJan Leike, prop. n.: A German AI-alignment researcher who helped develop training from human feedback, co-led OpenAI's superalignment team, and now leads alignment research at Anthropic., monosemanticitymonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several., network dissectionnetwork dissection, n.: A method for finding out what individual units inside an image-recognition network respond to, such as dogs, windows, grass or stripes, by comparing where each unit lights up with pictures labeled by people., NeuronpediaNeuronpedia, prop. n.: A free website where anyone can browse, search and test the internal concepts that researchers have found inside AI models, with examples of the text that switches each one on., PatchscopesPatchscopes, prop. n.: A family of tricks for getting a language model to describe its own hidden states in words, by transplanting an internal state into a new prompt designed to make the model talk about it.