Skip to content
Last Dictionary
Plane
Interpretability & SteeringSafety & Alignment
contested

enumerative safety

nounAlso: enumerative interpretability

In plain wordsA proposed safety strategy: list every concept an AI model represents internally, then check the list for dangerous ones, such as deception, before trusting the model.

  1. A proposed approach to AI safety through interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., in which one would enumerate all the featuresfeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity. a model represents — for example with dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts. — and inspect the list for safety-relevant ones, such as features for deception, manipulation or dangerous knowledge, or verify that certain features are absent or never active in particular contexts. Its appeal is that it could support guarantees about behaviors no test happened to elicit, instead of relying on behavioral evaluations that may miss rare or concealed behavior. Obstacles include superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read., the uncertain completeness of any feature dictionary (rare features and unexplained reconstruction error), feature splittingfeature splitting, n.: What happens when a concept-finding tool is given more room: a broad concept it found before breaks into several narrower ones, much as 'animal' might split into 'dog', 'cat' and 'bird'., which makes 'all the features' ill-defined, and the gap between knowing a feature exists and knowing how it is used. Many researchers therefore treat full enumeration as an aspiration rather than a near-term method.

    Enumerative safety would mean checking every one of the model's millions of features, not just the ones we searched for.

Origin

The idea of enumerating all of a model's features to make safety claims was sketched in the safety-strategy section of Anthropic's 'Toy Models of Superposition' (September 2022) as a motivation for solving superposition. The name was in use by March 2023, when Neel Nanda wrote of interpretability plans 'aiming for enumerative safety'; Charbel-Raphaël's critique 'Against Almost Every Theory of Impact of Interpretability' (August 2023) spread it further.

Related

See also
  • superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
  • dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
  • sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
  • AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable.
  • mechanistic anomaly detectionmechanistic anomaly detection, n.: Flagging cases where an AI reaches its output by an unusual internal route — for instance because it is being tricked, or is tricking us — even when the output itself looks normal.
  • AI deceptionAI deception, n.: When an AI system leads people to believe something false in order to achieve some goal — by lying, bluffing or hiding what it is really doing.

Sources

  1. Elhage, N. et al. (2022). Toy Models of Superposition. Transformer Circuits Thread. transformer-circuits.pub/2022/toy_model/index.html
  2. Nanda, N. (2023). Othello-GPT: Future Work I Am Excited About. AI Alignment Forum, 29 March 2023. alignmentforum.org/posts/qgK7smTvJ4DB8rZ6h/othello-gpt-future-work-i-am-excited-about
  3. Charbel-Raphaël (2023). Against Almost Every Theory of Impact of Interpretability. AI Alignment Forum, 17 August 2023. alignmentforum.org/posts/LNA8mubrByG7SFacm/against-almost-every-theory-of-impact-of-interpretability-1