Skip to content
Last Dictionary
Plane
Interpretability & SteeringMathematics & StatisticsDeep Learning
emerging

local learning coefficient

nounAlso: LLC, estimated learning coefficient

In plain wordsA number measuring how complex a trained neural network effectively is around its current settings — usually far less than its raw parameter count suggests — used to spot changes as the network learns.

  1. A measure of the effective complexity of a trained neural network at a particular point in parameter space, derived from singular learning theorysingular learning theory, n.: A mathematical theory of learning for models like neural networks, whose many redundant settings break standard statistics; it explains how such models generalize and how their learning unfolds.. It localizes Sumio Watanabe's learning coefficient (the real log canonical threshold) to the neighborhood of a given solution: roughly, it describes how fast the volume of nearly-as-good parameters around that solution shrinks as the loss tolerance is tightened, so that flatter, more degenerate solutions have lower values. Unlike parameter count, it reflects redundancy among parameters. It is estimated by sampling from a tempered posterior concentrated around the trained parameters, typically with stochastic gradient Langevin dynamics, and comparing the samples' average loss with the loss at the solution. Tracking it across training checkpoints is the main tool of developmental interpretabilitydevelopmental interpretability, n.: The study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model., where rises and drops mark developmental stages; variants restricted to subsets of weights measure the complexity of individual components such as attention heads.

    The local learning coefficient plateaued and then jumped just as the model learned to use the context.

λ^(w∗)=nβ(Ew∣w∗β[Ln(w)]−Ln(w∗)),β=1log⁡n\hat{\lambda}(w^*) = n\beta \left( \mathbb{E}^{\beta}_{w \mid w^*}\left[ L_n(w) \right] - L_n(w^*) \right), \qquad \beta = \frac{1}{\log n}

Origin

Introduced by Edmund Lau, Zach Furman, George Wang, Daniel Murfet and Susan Wei in 'The Local Learning Coefficient: A Singularity-Aware Complexity Measure' (arXiv, August 2023), building on Watanabe's learning coefficient.

Related

See also
  • singular learning theorysingular learning theory, n.: A mathematical theory of learning for models like neural networks, whose many redundant settings break standard statistics; it explains how such models generalize and how their learning unfolds.
  • developmental interpretabilitydevelopmental interpretability, n.: The study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model.
  • loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.
  • Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.
  • Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability.

Sources

  1. Lau, E., Furman, Z., Wang, G., Murfet, D. & Wei, S. (2023). The Local Learning Coefficient: A Singularity-Aware Complexity Measure. arXiv:2308.12108. arxiv.org/abs/2308.12108

Mentioned in

developmental interpretabilitydevelopmental interpretability, n.: The study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model., singular learning theorysingular learning theory, n.: A mathematical theory of learning for models like neural networks, whose many redundant settings break standard statistics; it explains how such models generalize and how their learning unfolds.