DiLoCo
In plain wordsA training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual.
A low-communication distributed training algorithm ('Distributed Low-Communication') in which several workers — each possibly a whole cluster — train replicas of a model independently for many inner steps (hundreds) with AdamW, then communicate once to average their parameter changes. The averaged change is treated as an outer 'pseudo-gradient' and applied by an outer optimizer, Nesterov momentum, before the next round. Its authors framed it as a variant of federated averaging, a form of local SGDlocal SGD, n.: A way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed., adapted to large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. pretraining; in their tests eight workers on the C4 dataset reached the quality of fully synchronous training with about one five-hundredth of the communication, and the method coped with workers dropping out or joining mid-run and with workers whose data differed. Later variants synchronize parameter fragments in a staggered stream to overlap communication with computation and quantize the exchanged updates (Streaming DiLoCo, 2025); open implementations underpin decentralized trainingdecentralized training, n.: Training an AI model on computers owned by many different people or companies around the world, linked over the ordinary internet rather than housed together in one data center. runs such as Prime Intellect's INTELLECT-1.
With DiLoCo the two data centers synchronize only every 500 steps over an ordinary WAN link.
Origin
Introduced by Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc'Aurelio Ranzato, Arthur Szlam and Jiajun Shen at Google DeepMind (arXiv, November 2023). Prime Intellect's open reimplementation OpenDiLoCo (July 2024) and DeepMind's Streaming DiLoCo (January 2025) followed.
Related
- local SGDlocal SGD, n.: A way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed.
- decentralized trainingdecentralized training, n.: Training an AI model on computers owned by many different people or companies around the world, linked over the ordinary internet rather than housed together in one data center.
- data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical.
- all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.
- distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
- federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates.
Sources
- Douillard, A., Feng, Q., Rusu, A. A., Chhaparia, R., Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A. & Shen, J. (2023). DiLoCo: Distributed Low-Communication Training of Language Models. arXiv:2311.08105. arxiv.org/abs/2311.08105
- Jaghouar, S., Ong, J. M., Basra, M., Obeid, F. et al. (2024). INTELLECT-1 Technical Report. arXiv:2412.01152. arxiv.org/abs/2412.01152
Mentioned in
distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks., all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync., decentralized trainingdecentralized training, n.: Training an AI model on computers owned by many different people or companies around the world, linked over the ordinary internet rather than housed together in one data center., local SGDlocal SGD, n.: A way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed.