second-order optimization
In plain wordsTraining methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more.
A family of optimization methods that use second-derivative (curvature) information — the HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass. or an approximation to it — to choose both the direction and the scale of each update. Newton's method multiplies the gradient by the inverse Hessian, taking long steps along flat directions and short ones along steep ones, and converges quadratically near a minimum; but for n parameters the Hessian has n² entries, impossible to store or invert at modern scales, and it can be indefinite near saddle pointssaddle point, n.: A spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass.. Practical approximations include quasi-Newton methods such as L-BFGS (common for small full-batch problems), Hessian-free optimization using conjugate gradients and Hessian–vector products, Gauss–Newton and natural-gradient methods based on the Fisher information matrix, Kronecker-factored approximations (K-FAC), diagonal curvature estimates (AdaHessian, Sophia) and structured preconditioners such as ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed.. Adaptive methods such as AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been. are sometimes called crude diagonal second-order methods, though their statistics come from gradients alone.
Origin
Newton's method for root-finding (Isaac Newton, 1669; Joseph Raphson, 1690) minimizes a function by seeking a zero of its gradient. Quasi-Newton methods date from William Davidon's 1959 report and the 1970 BFGS update; Shun-ichi Amari's natural gradient (1998), James Martens's Hessian-free deep learning (2010) and Martens and Roger Grosse's K-FAC (2015) adapted the ideas to neural networks.
Usage note
The boundary is blurry: ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed. and MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method. use only gradients yet act as structured preconditioners, and are variously described as first-order, 'second-order-like' or non-diagonal methods.
Related
- HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.
- ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed.
- AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been.
- gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.
- optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment.
- loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.
- gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.
Sources
- Amari, S. (1998). Natural Gradient Works Efficiently in Learning. Neural Computation 10(2), 251–276.
- Martens, J. & Grosse, R. (2015). Optimizing Neural Networks with Kronecker-factored Approximate Curvature. ICML 2015.
Mentioned in
gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down., MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method., optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment., Fisher informationFisher information, n.: A number, or table of numbers, saying how sharply the probability of the data reacts when a model's settings are nudged; the sharper the reaction, the more precisely the data can pin those settings down., HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass., ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed., Shun-ichi AmariShun-ichi Amari, prop. n.: A Japanese mathematician who trained multilayer neural networks by gradient descent as early as 1967 and founded information geometry, the source of the 'natural gradient' method., Taylor seriesTaylor series, n.: A way of approximating a complicated curve near a chosen point by adding up simple pieces: a constant, a straight-line slope, a bend, and so on.