Skip to content
Last Dictionary
Plane
Training & Post-trainingHardware & Compute

all-reduce

nounAlso: allreduce, AllReduce, ring all-reduce, tree all-reduce, MPI_Allreduce

In plain wordsA group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.

  1. A collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly. operation that combines — most often sums or averages — an array held by each of p processes elementwise and leaves the full result on every process. It is the core primitive of synchronous data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical., used to average gradients after each backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change.. Ring all-reduce, a reduce-scatter followed by an all-gather around a logical ring, is bandwidth-optimal: each process sends and receives about 2(p − 1)/p times the array size whatever p is, though latency grows with p. Tree and hierarchical algorithms cut latency for small messages or very large groups, and in-network reduction offloads the arithmetic to switches. Because the whole model's gradients are reduced every step, all-reduce bandwidth often limits data-parallel scaling, motivating overlap with computation, gradient compression and low-communication methods such as local SGDlocal SGD, n.: A way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed. and DiLoCoDiLoCo, prop. n.: A training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual..

    The gradient all-reduce takes 40 milliseconds per step, mostly hidden behind the backward pass.

data sent per process (ring)=2(p−1)p n\text{data sent per process (ring)} = \frac{2(p-1)}{p}\,n

Origin

Standardized as MPI_Allreduce in MPI-1 (1994). Pitch Patarasuk and Xin Yuan proved the ring algorithm bandwidth-optimal (2009). Baidu Research's open-source ring all-reduce code and Uber's Horovod library, both 2017, popularized it for multi-node deep learning.

Related

See also
  • collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly.
  • data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical.
  • NCCLNCCL, prop. n.: NVIDIA's software library that lets its graphics chips exchange data with each other quickly — inside one computer and across networks — and that nearly all large-scale AI training on NVIDIA hardware relies on.
  • local SGDlocal SGD, n.: A way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed.
  • interconnectinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one.
  • distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.

Sources

  1. Patarasuk, P. & Yuan, X. (2009). Bandwidth optimal all-reduce algorithms for clusters of workstations. Journal of Parallel and Distributed Computing 69(2).
  2. Sergeev, A. & Del Balso, M. (2018). Horovod: fast and easy distributed deep learning in TensorFlow. arXiv:1802.05799. arxiv.org/abs/1802.05799

Mentioned in

backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical., distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks., interconnectinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one., collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly., DiLoCoDiLoCo, prop. n.: A training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual., InfiniBandInfiniBand, prop. n.: A very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay., local SGDlocal SGD, n.: A way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed., Megatron-LMMegatron-LM, prop. n.: NVIDIA's open-source software, first described in a 2019 research paper, for training very large language models across thousands of graphics chips by splitting each model in several ways at once., NCCLNCCL, prop. n.: NVIDIA's software library that lets its graphics chips exchange data with each other quickly — inside one computer and across networks — and that nearly all large-scale AI training on NVIDIA hardware relies on., parameter serverparameter server, n.: An older design for training on many machines in which a few central computers hold the model and the others repeatedly fetch it, compute improvements on their share of data, and send them back., RDMARDMA, abbr.: A way for one computer to read or write another computer's memory directly over the network, skipping the usual software middlemen, so data moves faster with less delay., silent data corruptionsilent data corruption, n.: When computer hardware quietly produces wrong numbers without crashing or raising any alarm. In huge AI training runs on thousands of chips, such hidden errors can damage a model before anyone notices., stragglerstraggler, n.: One slow computer or chip in a large group working in lockstep. Because everyone must wait for the slowest member at each step, a single straggler can slow down an entire AI training job., tensor parallelismtensor parallelism, n.: Splitting the individual giant tables of numbers inside each layer of a model across several chips, so that every chip does a slice of every calculation at the same time.