How GPU Cluster Networking Works, and How It Fails

GPU clusters use separate networks: a scale-up interconnect linking GPUs inside a server or rack, a scale-out back-end fabric linking servers for GPU-to-GPU traffic using RDMA over InfiniBand or lossless Ethernet, and a front-end network for storage, management, and users. Training synchronises all GPUs repeatedly, so the slowest link sets the pace for the whole job.

Why AI training traffic is different

A typical data center network carries many independent flows: web requests, database queries, service calls. A slow flow affects one request.

Distributed training is the opposite. Thousands of GPUs each compute on part of the problem, then exchange results in collective operations, most notably all-reduce, which combines gradients across every participant. Every GPU waits until the collective completes. The traffic is synchronised, bursty, and very high bandwidth, and the job moves at the speed of its slowest participant.

That single property explains most of how GPU fabrics are designed and how they fail.

The three networks

Large clusters run three physically separate networks, each sized for a different kind of traffic.

Scale-up

A very high bandwidth interconnect linking GPUs within a server, and increasingly across a rack, so a group of GPUs can behave almost like one large device. NVIDIA's NVLink and NVSwitch are the most widely deployed example. Traffic here never touches the data center network.

Scale-out back-end

The fabric connecting servers for GPU-to-GPU communication. Each GPU typically has its own dedicated network interface. This network carries the collective traffic and is built for bandwidth and low, predictable latency.

Front-end

An ordinary data center network for storage access, data loading, checkpoints, management, and user traffic. Keeping it separate stops checkpoint writes and data loading from interfering with collective operations.

RDMA, InfiniBand, and RoCE

Scale-out fabrics use RDMA, remote direct memory access, so data moves between GPU memory on different servers without passing through the CPU and operating system network stack. That removes overhead and latency.

Two transports dominate. InfiniBand was designed for this: credit-based flow control makes it lossless, and it has mature congestion management and tooling. RoCEv2 carries RDMA over Ethernet. It relies on the Ethernet network being configured to avoid loss, using priority flow control, explicit congestion notification, and a congestion control algorithm, and it benefits from Ethernet's broad ecosystem. Poorly tuned RoCE fabrics suffer from pause storms and head-of-line blocking, which are hard to diagnose.

Topology: rails and non-blocking Clos

Most back-end fabrics are built as non-blocking Clos, or fat-tree, networks, so any GPU can talk to any other at full bandwidth.

Many designs are rail-optimised: the first GPU's interface in every server connects to one set of leaf switches, the second GPU's to another, and so on. Collective libraries arrange communication so that most traffic stays within a rail, crossing fewer switch tiers. Placement matters too: the scheduler should place a job's servers to keep traffic local where possible.

How GPU fabrics fail

  • Link flaps and degraded links. A single link running at reduced speed or intermittently dropping slows every collective that crosses it, and therefore the entire job. Health checks must look for degradation, not just down links.
  • Congestion and pause storms. In lossless Ethernet, congestion can propagate backwards through pause frames and stall unrelated traffic.
  • Stragglers. One slow GPU, overheating node, or misbehaving interface holds everyone back. Throughput drops without any component being fully down.
  • Hardware failure mid-job. With enough components, something fails regularly. Jobs rely on checkpointing to restart, and the time lost is the gap since the last checkpoint plus restart overhead.
  • Misconfiguration. Wrong congestion settings, mismatched firmware, or incorrect cabling on a rail produce subtle, persistent slowdowns.

What this means for operations

Treat job throughput as the primary health signal, because most fabric problems appear as slowdowns rather than outages. Correlate slow training steps with per-link counters, error rates, and congestion signals, so a degraded link can be found and drained quickly. Automate removing unhealthy nodes from the scheduler. And record fabric-related slowdowns as incidents with contributing causes, because a pattern of degraded optics or firmware problems is only visible across many events.

The teams that run these clusters well treat the fabric as part of the training system, not as generic infrastructure. Network engineers see job metrics, and training engineers see link health, so a slowdown is investigated from both sides at once. Shared dashboards and a single incident channel for fabric problems remove most of the finger-pointing between the two groups.

Frequently asked questions

Why do AI clusters need a separate network?
Because training traffic is synchronised and extremely high bandwidth. Every GPU exchanges data in collective operations and waits for the slowest participant. Mixing that traffic with storage, management, and user traffic would add delay and variability that slows every step of training, so clusters use a dedicated back-end fabric.
InfiniBand or Ethernet for AI clusters?
Both are used at large scale. InfiniBand offers built-in lossless behaviour and mature congestion management for RDMA. Ethernet with RoCEv2 relies on careful configuration of flow control and congestion notification, but offers a broader vendor ecosystem and familiar operations. The choice often depends on scale, existing expertise, vendor strategy, and operational tooling.
What is a rail-optimised network?
It is a topology where the same-numbered GPU network interface in every server connects to the same group of leaf switches, forming a rail. Communication libraries route most collective traffic within rails, which reduces the number of switch hops and contention. It suits the regular communication patterns of distributed training.