privileged basis

A basis of a representation that the architecture singles out, so that its coordinate directions are more likely than arbitrary directions to align with meaningful featuresfeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.. A representation has a privileged basis when some operation acts on it coordinate by coordinate — an elementwise activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer. such as ReLU, which treats each neuron separately, or optimizer effects that are not rotation-invariant. By contrast, the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document. of a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. is in principle non-privileged: rotating it, while applying compensating rotations to the weights that read and write it, yields an equivalent model, so its individual dimensions need have no meaning. The distinction tells researchers where neuron-level analysis can be expected to work, such as MLP hidden layers, and where they must look for directions instead. In practice the residual stream still shows some basis-aligned structure, including a few dimensions with very large activations, which Nelson Elhage, Robert Lasenby and Chris Olah (2023) traced largely to the Adam optimizer.

The MLP neurons have a privileged basis thanks to the GELU, so start the search there.