parameter decomposition

A family of interpretability methods that decompose a network's parameters, rather than its activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., into parameter-space components, often constrained to low rank, that add up to the original weights; the decomposition favors components that are individually simple and of which only a handful are needed for any one input. The aim is to identify the network's mechanisms directly, avoiding problems of activation-based dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts. such as reconstruction error, feature splittingfeature splitting, n.: What happens when a concept-finding tool is given more room: a broad concept it found before breaks into several narrower ones, much as 'animal' might split into 'dog', 'cat' and 'bird'. and the difficulty of saying how features are computed. Attribution-based parameter decomposition (APD) selects for each input the components with the highest gradient attributions and penalizes how many are needed, framed as minimizing the description length of the network's mechanisms. Stochastic parameter decomposition (SPD) replaced these attributions with learned estimates of causal importance and random masking, making the method more scalable and less sensitive to hyperparameters. Early demonstrations were on toy modelstoy model, n.: A deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work., such as recovering mechanisms of superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read. in small networks.

Parameter decomposition split the toy model's weights into one rank-one component per stored feature.