hardNewNVIDIACrusoeLambda

Design a non-blocking network fabric for 512 H100s. How many switches, how are they wired, and where does the NVLink domain end?

512 GPUs at 400 Gb/s each is 205 Tb/s of injection bandwidth a two-tier fabric must carry with no oversubscribed link. The port arithmetic that gives 16 leaves and 8 spines, the rail wiring that keeps NCCL traffic one hop away, and the line where 900 GB/s of NVLink becomes 50 GB/s of InfiniBand.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

Rail-Optimized and Fat-Tree FabricsA GPU cluster's network is built from two ideas: a fat tree (Clos) that gives every node a path to every other node with a chosen amount of oversubscription, and rail optimization, which wires GPU i of every node to the same leaf switch so the collectives that dominate training stay one hop away. Sizing one is arithmetic on port counts, and the interview question is usually that arithmetic: how many switches, what oversubscription, and where the NVLink domain ends and the fabric begins.

Foundational

🖧 Hardware & Cluster Build-Out

Scale-Out Fabric Choice: InfiniBand XDR vs Spectrum-XOutside the NVLink domain every GPU talks over a scale-out fabric, and as of September 2026 NVIDIA sells two at the same 800 Gb/s per port: Quantum-X800 InfiniBand and Spectrum-X Ethernet. They differ in congestion handling, operational familiarity and what happens when something misbehaves rather than in headline speed. The switch radix decides how many endpoints a two-tier fabric reaches, and that single number drives the switch count, the cable count and a large part of the budget.

Core

🔌 Networking & StorageSign in

RDMA, InfiniBand and RoCEv2Training across nodes moves hundreds of gigabytes per step, and a CPU-driven TCP stack cannot feed a 400 Gb/s link. RDMA lets a NIC write straight into a remote GPU's memory with no kernel and no copies, and it runs over two fabrics: InfiniBand, which is lossless by design, and RoCEv2, which is Ethernet made lossless by configuration. The choice is operational as much as technical, and the numbers that decide it are per-GPU bandwidth, the collective's volume, and who will debug a pause storm at 3 a.m.

Foundational

🖧 Hardware & Cluster Build-Out

The Bill of Materials for a Training ClusterA GPU cluster is not a pile of GPUs. A 512-GPU scalable unit built to NVIDIA's DGX SuperPOD B300 reference architecture needs 64 nodes, four separate networks, thousands of transceivers, storage that can absorb a checkpoint burst, a management plane, racks, power distribution and cooling equipment. Writing the list out in order is how a design becomes a purchase order, and the items people forget are the ones that hold up a deployment for weeks.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackExplain GPUDirect RDMA. What does it remove from the data path, and when does it change performance?Next in this trackAn Nsight summary reads DRAM 90%, SM 30%, occupancy 45%, L2 hit rate 20%. What is the bottleneck?Next in this trackWhy does the batch size change the TFLOPS a GEMM achieves? Draw the curve and explain each region.

FEDITOR'S NOTE

Scored on the port-counting derivation for a 1:1 Clos, understanding of rail-optimized wiring and why it matches NCCL's traffic, and a clear statement of the NVLink-to-network bandwidth cliff.

DISCUSSION · 0

No comments yet — be the first to share your approach.