hardNVIDIACrusoeLambda

Design a non-blocking network fabric for 512 H100s. How many switches, how are they wired, and where does the NVLink domain end?

Derive a 16-leaf, 8-spine fabric from 512 endpoints, trace local and cross-leaf routes, and check what rail placement and oversubscription actually guarantee.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Derive a 16-leaf, 8-spine fabric from 512 endpoints, trace local and cross-leaf routes, and check what rail placement and oversubscription actually guarantee.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🔌 Networking & Storage🔒 Premium

Rail-Optimized and Fat-Tree FabricsRail-optimized and Clos fabrics organize traffic between GPU nodes. Port counts, placement and oversubscription determine which paths are available to collectives.

Foundational

🖧 Hardware & Cluster Build-Out

Scale-Out Fabric Choice: InfiniBand XDR vs Spectrum-XInfiniBand and Spectrum-X connect GPUs across nodes. Compare congestion behavior, topology and operational requirements as well as port speed.

Core

🔌 Networking & StorageSign in

RDMA, InfiniBand and RoCEv2RDMA lets network adapters transfer data directly between registered memory regions. InfiniBand and RoCE provide different network and operating environments for those transfers.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceInterpret NCCL algbw and busbw, then isolate a slow collective with controlled tests. Distinguish rank delays, GPU-memory paths and shared-fabric limits without guessing from bandwidth ratios.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackExplain GPUDirect RDMA. What does it remove from the data path, and when does it change performance?Next in this trackAn Nsight summary reads DRAM 90%, SM 30%, occupancy 45%, L2 hit rate 20%. What is the bottleneck?Next in this trackWhy does the batch size change the TFLOPS a GEMM achieves? Draw the curve and explain each region.

FEDITOR'S NOTE

Count endpoints and ports before selecting switches. Keep rail locality separate from guaranteed traffic placement, and compare link bandwidth using the same direction convention.

DISCUSSION · 0

No comments yet — be the first to share your approach.