TL;DR: Start from what the cluster must train: a 350B dense model on 15T tokens is 3.2 × 10²⁵ FLOPs, which 10,240 H100s at 40% MFU finish in about 100 days of effective time. Nodes of 8 GPUs, pods of 1,024 (128 nodes) inside one non-blocking fabric tier, ten pods joined by a 2:1 oversubscribed core; tensor parallel on NVLink, pipeline inside a pod, data parallel hierarchical so the core carries the least. Storage sized from the checkpoint: 350B × 16 B = 5.6 TB, written in under a minute through local NVMe. Power about 14 MW at the feed. A stop every 5 hours means 3% warm spares, asynchronous checkpoints every 10 minutes and automated restart.
How to approach it
Ask what the cluster is for (one frontier run, or many jobs), which model and how many tokens, the deadline, the site's power, and whether it must be on one floor. Say the GPU count comes from the FLOPs and the deadline, then everything else serves that number. Work down the stack in order: nodes and pods, the fabric with port counts, storage from the checkpoint, power at the feed, failure statistics into spares and cadence, then the control plane. End with the table.
A strong answer
A typical situation: a lab has committed to 10,240 H100s and wants to know what it can train in a quarter, and how to lay it out. Training Cluster Design at 10k GPUs is the reference derivation; the answer below runs it in the order a candidate says it.
step 1: what 10,240 GPUs can train
sustained per GPU = 989 TFLOPS × 0.40 MFU ≈ 396 TFLOPS
100 days at 90% effective = 7.8e6 s → 10,240 × 3.96e14 × 7.8e6 ≈ 3.2e25 FLOPs
6 × N × D = 3.2e25 → N × D = 5.3e24 → a 350B model on 15T tokens
sanity: a 405B on 15.6T took about 16k GPUs and 54 days at similar MFU; 350B on 15T at 10k GPUs
in 100 days is the same order
Nodes and pods. A node is 8 GPUs on an NVLink switch at 900 GB/s per GPU, one 400 Gb/s NIC per GPU, several TB of NVMe, and 10 to 15 kW. 10,240 GPUs is 1,280 nodes. A pod is the set of nodes inside one non-blocking tier: with 64-port switches used as 32 down and 32 up, rail-optimized, that is 128 nodes (1,024 GPUs), so the cluster is ten pods. The parallelism plan decides what crosses what.
axis placement for the 350B run
TP 8: inside the node on NVLink; four activation collectives per layer, never on the NIC
PP 8: across 8 nodes in one pod; one activation tensor per micro-batch per boundary
DP 160: 10,240 ÷ 64 GPUs per replica; gradients once per step
per-GPU gradient shard = 350e9 × 2 B ÷ 64 = 10.9 GB; ring all-reduce ≈ 2 × 10.9 = 22 GB per step per GPU
hierarchical: reduce-scatter among the 16 replicas in a pod, then all-reduce the 1/16 shard across pods
cross-pod volume per GPU ≈ 22 ÷ 16 ≈ 1.4 GB per step, at 25 GB/s per NIC at 2:1 ≈ 55 ms, overlapped
sanity: the expensive tier carries the least traffic; that is the whole reason for pods
The fabric, with ports.
per pod: 1,024 endpoints; rail r's 128 NICs ÷ 32 down-ports = 4 leaves per rail × 8 rails = 32 leaves
spines: 32 leaves × 32 uplinks = 1,024 links ÷ 32 = 32 spines (non-blocking in-pod)
core at 2:1: 512 up-ports per pod × 10 = 5,120 ports ÷ 64 = 80 core switches
totals: 320 leaves + 320 spines + 80 core = 720 switches; about 25,000 cables, spine and core optical
Storage from the checkpoint. 350B × 16 B (bf16 weights and gradients, fp32 master and two Adam moments) = 5.6 TB per full checkpoint. Target under a minute so a 10-minute cadence costs under 10% even when synchronous: at least 95 GB/s aggregate. Asynchronous checkpoints hand state to host memory in seconds and let the host write at leisure, with local NVMe absorbing the burst (1,280 nodes × 7 GB/s = 9 TB/s of local write). A parallel filesystem at 100 to 200 GB/s holds the last few, and object storage keeps one per day. Data loading is 15T tokens over 100 days ≈ 1.7 M tokens/s, a few GB/s, trivial for the filesystem and non-trivial for the loader CPUs if tokenization happens online.
Power. 10,240 × 700 W = 7.2 MW at the chips; hosts, NICs and fans add about 40% → 10 MW; network, storage and management 1 MW; at a PUE of 1.25, about 14 MW at the feed. Air cooling is workable at Hopper density; the next generation at 120 kW racks is liquid.
Failures. At roughly 2 × 10⁻⁵ failures per GPU-hour, 10,240 GPUs stop about every 5 hours. Each stop costs T/2 + R; with asynchronous checkpoints every 10 minutes and a 5-minute automated restart that is 10 minutes, or 3.3% of the run, which is where the 90% effective time above came from. Spares: 1,280 nodes × 0.2 failures per node-day × 3-day repair ≈ 2% in repair, so keep 3% (about 40 nodes) warm and spread across pods. Failure domains are the pod (a spine), the rack (power) and the leaf (16 to 32 nodes); never place all of a pipeline stage's replicas on one leaf.
Control plane. A gang- and topology-aware scheduler, a fabric manager per node, an image and weight cache on every node, telemetry at about 80,000 GPU series per field, and a launcher that sets ranks to rails. A frontier run owns whole pods; smaller jobs share the remainder under quotas.
The trade-off to commit to is the 2:1 core. Non-blocking across pods costs roughly twice the core switches and optics and buys nothing if the DP traffic is hierarchical. The reversal condition: a parallelism plan that spans pods with pipeline or tensor traffic, or a multi-tenant cluster where many 2,048-GPU jobs straddle pods, and the core must go to 1:1. The Bill of Materials for a Training Cluster is this list priced, and nvidia-smi topo -m on the first node is where the design meets the hardware.
| Line | Quantity | Reason |
|---|---|---|
| GPUs | 10,240 H100 | 3.2e25 FLOPs in 100 days at 40% MFU |
| nodes, pods | 1,280 nodes, 10 pods of 1,024 | TP on NVLink, PP and most DP inside a pod |
| fabric | 320 leaves, 320 spines, 80 core, ~25k cables | non-blocking in-pod, 2:1 across |
| storage | NVMe burst, parallel FS 100 to 200 GB/s, object archive | 5.6 TB checkpoints in under a minute |
| power | ~14 MW at the feed | 700 W per GPU plus overheads and PUE |
| spares, cadence | ~40 warm nodes, async checkpoints every 10 min | a stop every 5 h, 3-day repair |
What interviewers probe next
- "Budget is halved; what do you cut?" Time, to 200 days; not MFU (unbuyable) and not the in-pod fabric (an oversubscribed pod costs more step time than it saves).
- "Why not one non-blocking fat tree?" Same guarantee at roughly double the optics; the pod structure exists because the parallelism plan makes it sufficient.
- "What changes on B200 nodes?" 180 GB and 1.8 TB/s NVLink shrink PP and grow DP; 800 Gb/s NICs halve the collective time; 120 kW racks force liquid cooling; the FLOP count halves the GPU count for the same run.
Common mistakes
- "10,000 GPUs on a fat tree with a Lustre filesystem" with no derivation for any number.
- Sizing storage from capacity instead of checkpoint write bandwidth.
- Treating failures as an operations problem rather than a design input.
- Placing pipeline stages across pods and then complaining about the core.
Key takeaways
- GPUs = FLOPs ÷ (peak × MFU × seconds); 10,240 H100s for 100 days trains about 350B on 15T.
- Pods of 1,024 non-blocking; TP on NVLink, PP in the pod, DP hierarchical; the core carries about 1.4 GB per GPU per step.
- Checkpoint = params × 16 B = 5.6 TB; under a minute means about 100 GB/s, via NVMe burst.
- A stop every 5 hours: 3% spares, async checkpoints every 10 minutes, automated restart, about 90% effective time.