collective communication
In plain wordsThe standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly.
Communication operations in which every process in a group participates, as opposed to point-to-point sends between pairs. The standard set includes broadcast (one process's data to all), reduce (combine everyone's data, for example by summing, onto one process), all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync. (combine and give everyone the result), all-gather (everyone receives the concatenation of all pieces), reduce-scatter (combine, then give each process one shard of the result), all-to-all (each process sends a distinct piece to every other) and barrier (synchronization). Distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks. is built from them: data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical. uses all-reduce, ZeROZeRO, prop. n.: A memory-saving technique for training big models on many chips: instead of every chip keeping its own complete copy of all the training bookkeeping, the chips divide it among themselves and share pieces only when needed. and FSDPfully sharded data parallel, n.: A feature of the popular PyTorch software that lets many chips train a model too big for any one of them, by spreading the model and its training bookkeeping evenly across all the chips. use all-gather and reduce-scatter, and expert parallelismexpert parallelism, n.: A way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs. uses all-to-all. Libraries such as MPI, NCCLNCCL, prop. n.: NVIDIA's software library that lets its graphics chips exchange data with each other quickly — inside one computer and across networks — and that nearly all large-scale AI training on NVIDIA hardware relies on., AMD's RCCL and Gloo implement them with ring, tree or hierarchical algorithms chosen for message size and network topology; their bandwidth and latency often cap training efficiency.
Origin
Collective operations were standardized for parallel computing in the Message Passing Interface (MPI-1, 1994), building on earlier libraries for hypercube and cluster computers. Deep learning adopted them from high-performance computing in the mid-2010s, notably through NVIDIA's NCCL (2016) and ring all-reduce implementations.
Related
- all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.
- NCCLNCCL, prop. n.: NVIDIA's software library that lets its graphics chips exchange data with each other quickly — inside one computer and across networks — and that nearly all large-scale AI training on NVIDIA hardware relies on.
- distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
- data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical.
- interconnectinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one.
- expert parallelismexpert parallelism, n.: A way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs.
- parameter serverparameter server, n.: An older design for training on many machines in which a few central computers hold the model and the others repeatedly fetch it, compute improvements on their share of data, and send them back.
Sources
- Message Passing Interface Forum (1994). MPI: A Message-Passing Interface Standard, version 1.0.
Mentioned in
CUDACUDA, prop. n.: Nvidia's software system for programming its graphics chips to do general calculations. Most AI software is built on it, which is a big reason Nvidia chips dominate AI., distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks., interconnectinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one., all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync., context parallelismcontext parallelism, n.: A way of training or running a model on extremely long inputs — whole books or codebases — by splitting the text across many chips and having them swap the information each part needs to 'see' the others., expert parallelismexpert parallelism, n.: A way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs., fully sharded data parallelfully sharded data parallel, n.: A feature of the popular PyTorch software that lets many chips train a model too big for any one of them, by spreading the model and its training bookkeeping evenly across all the chips., NCCLNCCL, prop. n.: NVIDIA's software library that lets its graphics chips exchange data with each other quickly — inside one computer and across networks — and that nearly all large-scale AI training on NVIDIA hardware relies on., parameter serverparameter server, n.: An older design for training on many machines in which a few central computers hold the model and the others repeatedly fetch it, compute improvements on their share of data, and send them back., RDMARDMA, abbr.: A way for one computer to read or write another computer's memory directly over the network, skipping the usual software middlemen, so data moves faster with less delay., stragglerstraggler, n.: One slow computer or chip in a large group working in lockstep. Because everyone must wait for the slowest member at each step, a single straggler can slow down an entire AI training job.