straggler
In plain wordsOne slow computer or chip in a large group working in lockstep. Because everyone must wait for the slowest member at each step, a single straggler can slow down an entire AI training job.
A worker (machine, accelerator, process or network link) that runs markedly slower than its peers in a distributed computation, delaying every step that requires all workers to finish. In synchronous large-model training, each step ends with collective operations such as all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync. that cannot complete until the slowest rank arrives, and pipeline parallelismpipeline parallelism, n.: Splitting a model's layers into consecutive stages on different chips, like stations on an assembly line, and streaming small batches of data through so that every station stays busy. chains stages so that one slow stage stalls the rest; across thousands of accelerators, some straggler is likely at any moment. Causes include thermal or power throttling, degraded chips or memory, faulty cables and switches, congested links, interfering host processes, data-loading stalls and garbage-collection pauses. Because stragglers rarely crash a job, they show up as lost throughput, lowering goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough. and model FLOPs utilizationmodel FLOPs utilization, n.: The share of a computer's top calculating speed that actually goes into useful AI training work. Real systems often reach only around a third to a half.. Mitigations include per-rank timing telemetry to locate the culprit, swapping slow nodes for hot spares, rebalancing work, backup workers that let a step proceed without the slowest gradients, and asynchronous or partially synchronous training.
One GPU with a failing fan was throttling, and as a straggler it cut throughput for all 4,096 ranks by 30 percent.
Origin
A general parallel-computing term, popularized by Jeffrey Dean and Sanjay Ghemawat's 2004 MapReduce paper, which named stragglers as a main cause of slow jobs and countered them with speculative 'backup tasks'. Jianmin Chen and colleagues (2016) adapted the backup idea to synchronous stochastic gradient descent, and straggler detection became a standard topic in reports on clusters of tens of thousands of GPUs from 2024.
Usage note
A straggler is slow but correct; contrast silent data corruptionsilent data corruption, n.: When computer hardware quietly produces wrong numbers without crashing or raising any alarm. In huge AI training runs on thousands of chips, such hidden errors can damage a model before anyone notices., where hardware is wrong but often not slow, and fail-stop faults, which crash the job and are handled by checkpointing and restart. Related to the 'tail at scale' problem in serving, where rare slow responses dominate overall latency.
Related
- data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical.
- all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.
- collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly.
- pipeline parallelismpipeline parallelism, n.: Splitting a model's layers into consecutive stages on different chips, like stations on an assembly line, and streaming small batches of data through so that every station stays busy.
- goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough.
- silent data corruptionsilent data corruption, n.: When computer hardware quietly produces wrong numbers without crashing or raising any alarm. In huge AI training runs on thousands of chips, such hidden errors can damage a model before anyone notices.
- AI supercomputerAI supercomputer, n.: A huge network of AI chips — often tens or hundreds of thousands — wired together in a data center to train the largest AI models.
- silent data corruptionsilent data corruption, n.: When computer hardware quietly produces wrong numbers without crashing or raising any alarm. In huge AI training runs on thousands of chips, such hidden errors can damage a model before anyone notices.
Sources
- Dean, J. & Ghemawat, S. (2004). MapReduce: Simplified Data Processing on Large Clusters. Proc. OSDI 2004.
- Chen, J., Pan, X., Monga, R., Bengio, S. & Jozefowicz, R. (2016). Revisiting Distributed Synchronous SGD. arXiv preprint.
Mentioned in
silent data corruptionsilent data corruption, n.: When computer hardware quietly produces wrong numbers without crashing or raising any alarm. In huge AI training runs on thousands of chips, such hidden errors can damage a model before anyone notices.