Skip to content
Last Dictionary
Plane
Training & Post-trainingDeep LearningMathematics & Statistics

local SGD

nounAlso: local stochastic gradient descent, periodic averaging, model averaging, federated averaging, FedAvg

In plain wordsA way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed.

  1. A distributed optimization method in which each of K workers runs H steps of stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset. (or another optimizer) on its own data, starting from a shared model, after which the workers' parameters are averaged and the process repeats. Compared with synchronous data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical., which communicates every step, it cuts communication by a factor of about H, at the cost of some divergence between workers that can slow convergence, especially with heterogeneous data or large H. Theory shows that with suitable step sizes it can match mini-batch SGD's convergence rate while communicating far less. Federated averaging applies the same idea to data held on user devices, and DiLoCoDiLoCo, prop. n.: A training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual. adds an outer optimizer that treats the averaged change as a pseudo-gradient, which made the approach work for language-model pretraining across poorly connected clusters.

θ(r+1)=1K∑k=1Kθk(r, H)\theta^{(r+1)} = \frac{1}{K}\sum_{k=1}^{K} \theta_k^{(r,\,H)}

Origin

Parameter averaging for parallel SGD was studied by Martin Zinkevich and colleagues (2010) and used in speech recognition by Daniel Povey and colleagues (2014). H. Brendan McMahan and colleagues' federated averaging (2017) applied it to data held on user devices, and Sebastian Stich's 'Local SGD Converges Fast and Communicates Little' (2018) helped fix the name and its convergence theory; the first use of the name is not established.

Usage note

In the formula, θ_k^(r,H) are worker k's parameters after H local steps in round r.

Related

See also
  • DiLoCoDiLoCo, prop. n.: A training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual.
  • data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical.
  • federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates.
  • decentralized trainingdecentralized training, n.: Training an AI model on computers owned by many different people or companies around the world, linked over the ordinary internet rather than housed together in one data center.
  • stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.
  • all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.
Contrast with
  • data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical.

Sources

  1. Stich, S. U. (2019). Local SGD Converges Fast and Communicates Little. ICLR 2019.
  2. McMahan, H. B., Moore, E., Ramage, D., Hampson, S. & Agüera y Arcas, B. (2017). Communication-Efficient Learning of Deep Networks from Decentralized Data. AISTATS 2017.

Mentioned in

distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks., all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync., decentralized trainingdecentralized training, n.: Training an AI model on computers owned by many different people or companies around the world, linked over the ordinary internet rather than housed together in one data center., DiLoCoDiLoCo, prop. n.: A training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual., parameter serverparameter server, n.: An older design for training on many machines in which a few central computers hold the model and the others repeatedly fetch it, compute improvements on their share of data, and send them back.