Skip to content
Last Dictionary
Plane
Deep LearningMathematics & StatisticsMachine Learning

AdaGrad

proper nounAlso: adaptive gradient algorithm

In plain wordsA training method that gives each of a model's settings its own step size, shrinking steps for settings that have already had many large updates and keeping them larger for rarely updated ones.

  1. An adaptive optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment. that divides each parameter's learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. by the square root of the running sum of all its past squared gradients, plus a small constant for numerical safety. Parameters with large or frequent gradients get progressively smaller steps, while rarely active ones — weights for infrequent words or sparse features — keep larger steps, which made it effective for sparse problems such as click-through prediction and for early word embeddingsword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'. (GloVe was trained with it). It carries regret guarantees in online convex optimization and can be read as a diagonal approximation to a full-matrix preconditioner, an idea ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed. later revived. Because the accumulated sum only grows, the effective learning rate decays monotonically toward zero, which can stall the training of deep non-convex models; RMSPropRMSProp, prop. n.: A training method that scales each setting's step by how large its recent changes have been, so jumpy settings move cautiously and quiet ones move faster. and AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been. replace the sum with an exponential moving average to avoid this.

θt+1,i=θt,i−η∑s≤tgs,i2+ϵ gt,i\theta_{t+1,i} = \theta_{t,i} - \frac{\eta}{\sqrt{\sum_{s \le t} g_{s,i}^2} + \epsilon}\, g_{t,i}

Origin

Introduced by John Duchi, Elad Hazan and Yoram Singer at the COLT 2010 conference and in the Journal of Machine Learning Research (2011); H. Brendan McMahan and Matthew Streeter proposed closely related adaptive methods at the same conference. Google's DistBelief system (2012) used it for large-scale distributed training.

Related

See also
  • RMSPropRMSProp, prop. n.: A training method that scales each setting's step by how large its recent changes have been, so jumpy settings move cautiously and quiet ones move faster.
  • AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been.
  • optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment.
  • learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.
  • ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed.
  • stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.

Sources

  1. Duchi, J., Hazan, E. & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research 12, 2121–2159.

Mentioned in

AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been., optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment., RMSPropRMSProp, prop. n.: A training method that scales each setting's step by how large its recent changes have been, so jumpy settings move cautiously and quiet ones move faster., ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed.