parameter server
In plain wordsAn older design for training on many machines in which a few central computers hold the model and the others repeatedly fetch it, compute improvements on their share of data, and send them back.
A distributed-training architecture in which one or more server processes store the model parameters, sharded by key, while worker processes pull current parameters, compute gradients on their own data and push updates back for the servers to apply. Updates may be synchronous or asynchronous; in asynchronous SGD workers proceed without waiting for each other, tolerating slow machines at the cost of 'stale' gradients computed on outdated parameters. The design suited 2010s clusters of commodity CPUs and very large sparse models — click-through prediction, topic models and recommendation embeddings — in which each worker touches only a small fraction of the parameters. For dense deep networks on GPU clusters it was largely superseded after about 2017 by synchronous all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.-based data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical., which avoids server bandwidth bottlenecks and staleness, though parameter-server-style sharding persists for huge embedding tables in recommender systemsrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on..
Origin
Alexander Smola and Shravan Narayanamurthy described a parameter-server architecture for parallel topic models (VLDB 2010); Google's DistBelief (Jeffrey Dean and colleagues, 2012) used one for asynchronous 'Downpour SGD', and Mu Li and colleagues' OSDI 2014 paper generalized the design. Feng Niu and colleagues' lock-free Hogwild! (2011) analyzed asynchronous updates.
Related
- distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
- data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical.
- all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.
- stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.
- local SGDlocal SGD, n.: A way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed.
- recommender systemrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on.
- collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly.
Sources
- Smola, A. & Narayanamurthy, S. (2010). An Architecture for Parallel Topic Models. Proceedings of the VLDB Endowment 3(1).
- Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A., Josifovski, V., Long, J., Shekita, E. J. & Su, B.-Y. (2014). Scaling Distributed Machine Learning with the Parameter Server. OSDI 2014.
Mentioned in
distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks., collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly.