What is Ultra Ethernet trying to fix, and does it change how you would choose a fabric today?

RoCE inherited a transport that assumes one ordered path per connection, which is why it needs a lossless network built by hand underneath it. What a redesigned transport changes about that, which of today's operational burdens it removes, and why the honest answer to a purchase decision is still about what ships.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

Congestion Control for AI FabricsCollective traffic is the worst case a network can see: hundreds of senders transmit to the same receiver at the same instant (incast), every flow is large and long-lived, and RDMA cannot tolerate a dropped packet. Congestion control is the set of mechanisms (PFC, ECN with DCQCN, adaptive routing, packet spraying) that keep queues from overflowing without stalling the fabric. On plain Ethernet a busy all-reduce can fall to about 60% of link rate; with a tuned control loop it holds above 90%. Reading the counters that show which one you have is the on-call skill.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.

Core

🔌 Networking & StorageSign in

RDMA, InfiniBand and RoCEv2Training across nodes moves hundreds of gigabytes per step, and a CPU-driven TCP stack cannot feed a 400 Gb/s link. RDMA lets a NIC write straight into a remote GPU's memory with no kernel and no copies, and it runs over two fabrics: InfiniBand, which is lossless by design, and RoCEv2, which is Ethernet made lossless by configuration. The choice is operational as much as technical, and the numbers that decide it are per-GPU bandwidth, the collective's volume, and who will debug a pause storm at 3 a.m.

Advanced

🔌 Networking & Storage🔒 Premium

Rail-Optimized and Fat-Tree FabricsA GPU cluster's network is built from two ideas: a fat tree (Clos) that gives every node a path to every other node with a chosen amount of oversubscription, and rail optimization, which wires GPU i of every node to the same leaf switch so the collectives that dominate training stay one hop away. Sizing one is arithmetic on port counts, and the interview question is usually that arithmetic: how many switches, what oversubscription, and where the NVLink domain ends and the fabric begins.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the complete network for a 1,024-GPU cluster: compute fabric, storage, management. What are the counts and where does the money go?Next in this trackDoes multi-node inference need the same fabric as training, or can it run on ordinary networking?Next in this trackOne fabric link goes down and comes back once an hour. What does that do to a 1,024-GPU training job, and how would you find it?

FEDITOR'S NOTE

Scored on identifying single-path ordered delivery as the constraint being removed, on naming the specific operational burdens that follow from it, and on separating a technology direction from a purchase decision.

DISCUSSION · 0

No comments yet — be the first to share your approach.