Skip to content
Last Dictionary
Plane
Deep LearningTraining & Post-trainingMathematics & Statistics

Adam

proper nounAlso: adaptive moment estimation, Adam optimizer, AdaMax, AMSGrad

In plain wordsThe most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been.

  1. An adaptive first-order optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment. that keeps two exponential moving averages per parameter — of the gradient (the first moment, a form of momentummomentum, n.: A training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps., decay β₁ ≈ 0.9) and of the squared gradient (the second moment, decay β₂ ≈ 0.999, or 0.95 for many language models) — corrects both for their bias toward zero early in training, and moves each parameter by the ratio of the first moment to the square root of the second, times the learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.. The resulting step is nearly invariant to the scale of the gradient and of similar size for every parameter, which makes Adam robust on poorly scaled problems. It combines ideas from RMSPropRMSProp, prop. n.: A training method that scales each setting's step by how large its recent changes have been, so jumpy settings move cautiously and quiet ones move faster. and AdaGradAdaGrad, prop. n.: A training method that gives each of a model's settings its own step size, shrinking steps for settings that have already had many large updates and keeping them larger for rarely updated ones. and became the default for transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., GANsgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people. and most deep learning outside image classification, where SGD with momentum often generalizes as well or better. Known weaknesses are its memory cost (two extra values per parameter), a gap in the original convergence proof (addressed by AMSGrad) and a poor interaction with L2 regularization, corrected by AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models..

    Swapping SGD for Adam got the transformer training on the first try, at the cost of two extra buffers per weight.

mt=β1mt−1+(1−β1) gt,vt=β2vt−1+(1−β2) gt2,θt=θt−1−η mt/(1−β1t)vt/(1−β2t)+ϵm_t = \beta_1 m_{t-1} + (1-\beta_1)\, g_t, \quad v_t = \beta_2 v_{t-1} + (1-\beta_2)\, g_t^2, \quad \theta_t = \theta_{t-1} - \eta \, \frac{m_t/(1-\beta_1^t)}{\sqrt{v_t/(1-\beta_2^t)} + \epsilon}

Origin

Introduced by Diederik P. Kingma and Jimmy Ba in 'Adam: A Method for Stochastic Optimization' (arXiv December 2014; ICLR 2015), which also proposed the AdaMax variant; the name derives from 'adaptive moment estimation'. It became one of the most cited papers in machine learning. Sashank Reddi, Satyen Kale and Sanjiv Kumar (2018) exposed and repaired a flaw in its convergence analysis.

Usage note

Written Adam, not ADAM. Because each step is roughly bounded by the learning rate whatever the gradient's scale, Adam learning rates (about 10⁻⁴ to 10⁻³) are not comparable with SGD ones (about 0.1).

Related

See also
  • AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models.
  • RMSPropRMSProp, prop. n.: A training method that scales each setting's step by how large its recent changes have been, so jumpy settings move cautiously and quiet ones move faster.
  • AdaGradAdaGrad, prop. n.: A training method that gives each of a model's settings its own step size, shrinking steps for settings that have already had many large updates and keeping them larger for rarely updated ones.
  • momentummomentum, n.: A training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps.
  • optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment.
  • MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method.
  • learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing.

Sources

  1. Kingma, D. P. & Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015. arxiv.org/abs/1412.6980
  2. Reddi, S. J., Kale, S. & Kumar, S. (2018). On the Convergence of Adam and Beyond. ICLR 2018.

Mentioned in

gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down., optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., AdaGradAdaGrad, prop. n.: A training method that gives each of a model's settings its own step size, shrinking steps for settings that have already had many large updates and keeping them larger for rarely updated ones., AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models., full fine-tuningfull fine-tuning, n.: Fine-tuning in which every one of a model's internal numbers is allowed to change — the most flexible option, but also the most expensive in memory and storage., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., ICLRICLR, prop. n.: A major deep-learning conference, held every spring since 2013, where the reviews of each submitted paper are posted online for anyone to read., learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls., learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing., momentummomentum, n.: A training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps., optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment., RMSPropRMSProp, prop. n.: A training method that scales each setting's step by how large its recent changes have been, so jumpy settings move cautiously and quiet ones move faster., stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset., variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves., AdafactorAdafactor, prop. n.: A memory-saving cousin of the popular Adam training method. Instead of keeping extra statistics for every single weight, it keeps summaries for each row and column of a weight table., Diederik KingmaDiederik Kingma, prop. n.: A Dutch computer scientist who co-invented the variational autoencoder and the Adam optimizer, two of the most widely used tools for training neural networks., exponential moving average of weightsexponential moving average of weights, n.: A smoothed copy of a model kept during training, which each update nudges only slightly toward the latest weights. The calmer copy often performs better than the jittery original., LionLion, prop. n.: A training method discovered by a computer search over possible algorithms. It moves every setting by the same small amount at each step, choosing only the direction, and needs less memory than the popular Adam method., second-order optimizationsecond-order optimization, n.: Training methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more., ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed., ZeROZeRO, prop. n.: A memory-saving technique for training big models on many chips: instead of every chip keeping its own complete copy of all the training bookkeeping, the chips divide it among themselves and share pieces only when needed.