The Bill of Materials for a Training Cluster
A GPU cluster is not a pile of GPUs. A 512-GPU scalable unit built to NVIDIA's DGX SuperPOD B300 reference architecture needs 64 nodes, four separate networks, thousands of transceivers, storage that can absorb a checkpoint burst, a management plane, racks, power distribution and cooling equipment. Writing the list out in order is how a design becomes a purchase order, and the items people forget are the ones that hold up a deployment for weeks.
TL;DR: Build the list network by network, because the four networks are what people under-count. A 512-GPU scalable unit following NVIDIA's DGX SuperPOD B300 reference architecture is 64 nodes across 16 racks, one 800 Gb/s compute fabric with 8 leaf switches, a separate storage fabric, an in-band management network, and an out-of-band network reaching every BMC and every PDU. That is roughly 1,024 compute-fabric links and about 2,048 transceivers before storage or management are counted. Add storage sized by checkpoint burst rather than by capacity, a management plane of head nodes and a provisioning path, racks with busway taps and CDUs, and spares. The items with the longest lead times are rarely the GPUs: power equipment, CDUs and the facility work usually gate the schedule, and ordering them after the compute is the classic way to have GPUs sitting in a warehouse.
The list, in the order it is assembled
| Layer | What a 512-GPU SU needs | The mistake |
|---|---|---|
| Compute | 64 nodes, 8 GPUs each; 16 racks at 4 nodes | Sizing racks by U rather than by kilowatts |
| Compute fabric | 8 leaf switches plus spines; 512 node-to-leaf and 512 leaf-to-spine links | Forgetting the leaf-to-spine half of the cabling |
| Optics and cables | about 2,048 modules; DAC inside racks | Reusing 400G cable lengths at 800G |
| Storage fabric | separate ports per node, sized by checkpoint burst | Sharing the compute fabric and discovering it at the first checkpoint |
| In-band management | one port per node for provisioning, telemetry, package installs | Assuming the compute fabric carries it |
| Out-of-band management | one port per BMC, per PDU, per switch, per CDU | Under-counting; there are more OOB endpoints than nodes |
| Storage | capacity for datasets plus burst bandwidth for checkpoints | Buying capacity and no bandwidth |
| Management plane | head or login nodes, a provisioning server, a metrics and logging tier | Running it on a compute node and losing it during an incident |
| Power | busway taps, PDUs, A and B feeds sized per rack | Reserving the average draw rather than the peak |
| Cooling | CDUs at 10 to 15 percent above rack load, manifolds, hoses, leak detection | Ordering after the compute and gating the schedule |
| Spares | GPUs or whole nodes, transceivers, cables, one of every switch type | Zero transceiver spares, which is a week of downtime for a $500 part |
Sizing storage from the checkpoint, not the dataset
checkpoint burst for a 70B model trained on this SU
training state at the corpus figure of 16 bytes per parameter:
70e9 x 16 = 1.12 TB per checkpoint
target write time so the job barely notices: 60 s
required write bandwidth = 1.12e12 / 60 = 18.7 GB/s sustained
dataset read, for comparison
8 nodes' worth of pre-tokenized text at the corpus figure of 64 KB/s per 8 GPUs:
64 nodes x 8 GPUs x 8 KB/s per GPU = about 4 MB/s
sanity: the checkpoint burst is four thousand times the steady read, so a storage tier
specified from the dataset rate is under-provisioned by three orders of magnitude for
the one operation that matters, which is why checkpoint bandwidth is the sizing number
Parallel Filesystems vs Object Storage covers which tier serves which; the bill-of-materials point is that "how many terabytes" is the wrong first question and "how many gigabytes per second during a burst" is the right one.
The four networks, drawn
The out-of-band network is the one that is always under-counted. Every node has a BMC, and so does every switch, every PDU and every CDU, so the endpoint count is larger than the node count and the switches for it are a separate purchase. It is also the network that has to work when nothing else does, which is exactly when someone discovers it was never finished.
Lead times decide the schedule
typical ordering, longest lead time first
1. facility work: power capacity, floor loading, water months to years
2. electrical distribution: busway, taps, upstream gear months
3. cooling: CDUs, manifolds, facility loop connections months
4. network switches weeks to months
5. compute nodes weeks to months
6. cables and transceivers weeks, and they are the
item most often wrong
7. spares order with the main purchase
the failure this ordering prevents
compute delivered into a hall that cannot power or cool it, which is the most expensive
idle asset in the industry
sanity: at $2.5 per GPU-hour, 512 GPUs sitting for one month of facility delay is
512 x 24 x 30 x 2.5 = about $921,000 of unrealized capacity, which is far more than
the cost of ordering the cooling early
What interviewers are listening for
Completeness and ordering. Naming four networks rather than one is the single clearest signal that a candidate has been near a real build, and naming the out-of-band network with its endpoint count is better still. The second signal is sizing storage from the checkpoint burst rather than the dataset. The third is lead time: an engineer who says "the GPUs are not the long pole, the power and cooling are, so those go first" has watched a deployment slip. If you can add the cost of the delay in GPU-hours, the point lands with whoever controls the budget.
Key takeaways
- Four networks, not one: compute fabric, storage fabric, in-band management, and out-of-band management reaching every BMC, PDU, switch and CDU.
- A 512-GPU scalable unit is 64 nodes in 16 racks with 8 leaf switches, roughly 1,024 compute-fabric links and about 2,048 transceivers.
- Size storage by checkpoint burst: 1.12 TB of training state for a 70B model in 60 seconds is 18.7 GB/s, thousands of times the steady dataset read.
- Order facility, electrical and cooling first; compute has a shorter lead time and arrives to a hall that must already be ready.
- A month of facility delay on 512 idle GPUs is about $921,000 at $2.5 per GPU-hour, which dwarfs the cost of ordering cooling early.