RDMA
In plain wordsA way for one computer to read or write another computer's memory directly over the network, skipping the usual software middlemen, so data moves faster with less delay.
A networking technique in which network adapters move data between the memories of two machines on their own, so that no CPU or operating system has to copy or handle the bytes in transit ('zero-copy', 'kernel bypass'); the payoff is very low latency, high throughput and little CPU overhead. It is native to InfiniBandInfiniBand, prop. n.: A very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay. and runs over Ethernet as RoCE (RDMA over Converged Ethernet) or iWARP. In AI clusters, GPUDirect RDMA lets network cards read and write GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. memory directly, so gradients and activations move between accelerators in different servers without staging through host memory; communication libraries such as Nvidia's NCCL build collectivescollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly. like all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync. on top of it. Ethernet RDMA needs careful congestion control at very large scale, a problem the Ultra Ethernet transport aims to address.
Origin
Grew out of 1990s research on user-level networking and the Virtual Interface Architecture (1997); standardized as a core feature of InfiniBand (2000), then carried to Ethernet via iWARP (IETF, 2007) and RoCE (2010; routable RoCEv2, 2014). Nvidia's GPUDirect RDMA appeared in 2012.
Related
- InfiniBandInfiniBand, prop. n.: A very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay.
- interconnectinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one.
- all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.
- collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly.
- scale-up and scale-outscale-up and scale-out, phr.: Two ways to get more computing power: make each machine bigger and more tightly linked (scale up), or add more machines connected by a network (scale out).
- distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
Mentioned in
interconnectinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one., disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them., InfiniBandInfiniBand, prop. n.: A very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay., scale-up and scale-outscale-up and scale-out, phr.: Two ways to get more computing power: make each machine bigger and more tightly linked (scale up), or add more machines connected by a network (scale out).