automated circuit discovery
In plain wordsSoftware methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand.
Algorithms that identify, with little human input, the circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name. in a network responsible for a specified behavior. The usual formulation treats the model as a computational graph of components — attention heads, MLPs or sparse features — and searches for a minimal subgraph whose behavior on a task dataset matches the full model's, measured by a metric such as KL divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. or logit difference when edges outside the subgraph are fed corrupted activations. ACDC (Automatic Circuit DisCovery) works backward from the output, pruning each edge whose removal changes the metric by less than a threshold; it recovered hand-found circuits such as indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model. and greater-than comparison in GPT-2 small. Gradient-based approximations — edge attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part. (EAP) and its integrated-gradients variant — score all edges at once and scale to larger models; learned masks and sparse-feature methods are alternatives.
On the greater-than task, ACDC kept 68 of 32,000 edges, all of which manual analysis had already found.
Origin
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim and Adrià Garriga-Alonso introduced ACDC in 'Towards Automated Circuit Discovery for Mechanistic Interpretability' (2023; NeurIPS 2023), automating the workflow of the IOI study. Aaquib Syed, Can Rager and Arthur Conmy proposed edge attribution patching later in 2023.
Related
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part.
- path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior.
- indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model.
- circuit tracingcircuit tracing, n.: Methods that map, step by step, which internal concepts an AI model used to produce a particular answer and how they fed into one another, like tracing wires on a circuit board.
Sources
- Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S. & Garriga-Alonso, A. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS 2023. arXiv:2304.14997. arxiv.org/abs/2304.14997
- Syed, A., Rager, C. & Conmy, A. (2023). Attribution Patching Outperforms Automated Circuit Discovery. arXiv preprint.
Mentioned in
circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name., attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part., indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model., path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior., sparse feature circuitsparse feature circuit, n.: A map of how a language model performs a task, drawn in terms of understandable concepts found by concept-finding tools, rather than hard-to-read neurons or whole layers., toy modeltoy model, n.: A deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work.