local minimum
In plain wordsA spot lower than everything immediately around it, like the bottom of a small valley, even though a deeper valley may exist somewhere else.
A point at which a function's value is no greater than at any nearby point, though it may be higher than the function's lowest value overall, the global minimum. For smooth functions a local minimum has zero gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill. and a positive-semidefinite HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.; other points with zero gradient are maxima or saddle pointssaddle point, n.: A spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass.. In convex problemsconvex optimization, n.: Finding the lowest point of a bowl-shaped landscape. Because a bowl has only one bottom, such problems can be solved reliably — unlike training a neural network, whose landscape is bumpy. every local minimum is global, but neural-network loss landscapesloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys. are not convex, and poor local minima were long feared as the main obstacle to training deep networks. Theory and experiments since the 2010s suggest that in large, overparameterizedoverparameterization, n.: Giving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps. networks most local minima have loss close to the global minimum, while saddle points and plateaus are the bigger practical hurdles; which minimum training reaches still affects generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers. (see sharp and flat minimasharp and flat minima, n.: Two kinds of low points a trained model can settle into: narrow pits where small changes spoil performance, and broad basins where they barely matter. Broad basins are thought to generalize better, though this is debated.).
A different random seed got us out of that local minimum, and the loss fell another 10 percent.
Origin
Pierre de Fermat's 1630s method for finding maxima and minima, which in effect sets the derivative to zero, is the classical ancestor. The local-versus-global distinction became central to numerical optimization in the 20th century and to neural networks in 1980s debates over the limits of backpropagation.
Related
- saddle pointsaddle point, n.: A spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass.
- loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.
- convex optimizationconvex optimization, n.: Finding the lowest point of a bowl-shaped landscape. Because a bowl has only one bottom, such problems can be solved reliably — unlike training a neural network, whose landscape is bumpy.
- sharp and flat minimasharp and flat minima, n.: Two kinds of low points a trained model can settle into: narrow pits where small changes spoil performance, and broad basins where they barely matter. Broad basins are thought to generalize better, though this is debated.
- gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.
- HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.
Mentioned in
gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down., optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible., convergenceconvergence, n.: The point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast., convex optimizationconvex optimization, n.: Finding the lowest point of a bowl-shaped landscape. Because a bowl has only one bottom, such problems can be solved reliably — unlike training a neural network, whose landscape is bumpy., hill climbinghill climbing, n.: A simple search method that keeps making whichever small change improves things, like climbing in fog by always stepping uphill — which can leave it stuck on a small hill instead of the highest peak., HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass., saddle pointsaddle point, n.: A spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass.