edge of stability

nounAlso: EoS, progressive sharpening, self-stabilization

In plain wordsA surprising pattern in how neural networks train: the error landscape keeps getting steeper until it is just steep enough to make training wobble, then stays balanced at that edge while the error still falls.

A regime of neural-network training by gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down. in which curvature and step size lock together. Sharpness, the top eigenvalue of the HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass. of the training loss, climbs during training ('progressive sharpening') until it hits 2/η, the point at which classical analysis predicts that gradient descent with learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. η blows up; rather than diverging, training then remains pinned at or slightly above that value. The loss stops decreasing monotonically and zigzags along the sharpest direction from step to step, yet trends downward over longer stretches, which the textbook requirement that the step size stay below 2/sharpness says should not happen. The dynamics stabilize themselves, because moving along the unstable direction reduces sharpness. Adaptive optimizers show an analogous 'adaptive edge of stability' in which preconditioned sharpness hovers at a threshold set by the learning rate and momentum. The phenomenon ties the learning rate to the curvature of the solutions found, and so to sharp and flat minimasharp and flat minima, n.: Two kinds of low points a trained model can settle into: narrow pits where small changes spoil performance, and broad basins where they barely matter. Broad basins are thought to generalize better, though this is debated., warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing. and implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions..

\lambda_{\max}\!\left(\nabla^2 L(\theta_t)\right) \approx \frac{2}{\eta}

Origin

Named by Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter and Ameet Talwalkar in 'Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability' (arXiv February 2021; ICLR 2021). Related 'catapult' dynamics at large learning rates were described by Aitor Lewkowycz and colleagues (2020), and Alex Damian and colleagues explained the self-stabilization mechanism in 2022.

Sources

Cohen, J. M., Kaur, S., Li, Y., Kolter, J. Z. & Talwalkar, A. (2021). Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICLR 2021. arxiv.org/abs/2103.00065

Mentioned in

gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down., eigenvectoreigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies., learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls., learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing., loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys., HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass., sharp and flat minimasharp and flat minima, n.: Two kinds of low points a trained model can settle into: narrow pits where small changes spoil performance, and broad basins where they barely matter. Broad basins are thought to generalize better, though this is debated.