The Bill of Materials for a Training Cluster
A cluster bill of materials includes networking, storage, management, power, cooling and spares as well as GPUs. Size each from the workload and topology.
TL;DR: Start with the jobs the cluster must run, then count the equipment that supports them. Separate compute traffic, storage traffic and management needs, even if some share physical infrastructure. Check burst storage demand, cable counts, facility readiness and spares before treating the GPU delivery date as the deployment date.
A scalable unit (SU) is a repeatable building block of nodes, switches and racks that a cluster is assembled from. Rack space is counted in rack units (U), each 44.45 mm of vertical space.
The list, in the order it is assembled
| Layer | What a 512-GPU SU needs | The mistake |
|---|---|---|
| Compute | 64 nodes, 8 GPUs each; 16 racks at 4 nodes | Sizing racks by U rather than by kilowatts |
| Compute fabric | 8 leaf switches plus spines; 512 node-to-leaf and 512 leaf-to-spine links | Forgetting the leaf-to-spine half of the cabling |
| Optics and cables | count physical modules and copper links from the cable map | Reusing 400G cable lengths at 800G |
| Storage fabric | separate ports per node, sized by checkpoint burst | Sharing the compute fabric and discovering it at the first checkpoint |
| In-band management | one port per node for provisioning, telemetry, package installs | Assuming the compute fabric carries it |
| Out-of-band management | one port per BMC, per PDU, per switch, per CDU | Under-counting; there are more OOB endpoints than nodes |
| Storage | capacity for datasets plus burst bandwidth for checkpoints | Buying capacity and no bandwidth |
| Management plane | head or login nodes, a provisioning server, a metrics and logging tier | Running it on a compute node and losing it during an incident |
| Power | busway taps, PDUs, A and B feeds sized per rack | Reserving the average draw rather than the peak |
| Cooling | vendor-rated CDUs, manifolds, hoses, leak detection and required redundancy | Ordering after the compute and gating the schedule |
| Spares | GPUs or whole nodes, transceivers, cables, one of every switch type | Zero transceiver spares, which is a week of downtime for a $500 part |
Sizing storage from the checkpoint, not the dataset
Count the state the checkpoint actually saves; it may differ from the live training allocation. For an illustrative 70B recipe that saves 16 bytes per parameter:
saved payload = 70e9 x 16 bytes = 1.12 TB
write-time target = 60 seconds
required payload rate = 1.12e12 / 60 = 18.7 GB/s
sanity: this excludes metadata, retries, redundancy and simultaneous dataset traffic.
The input read rate depends on the data representation and training rate. Compare it with checkpoint bursts on any shared path. Parallel Filesystems vs Object Storage explains why capacity, access semantics and sustained bandwidth all belong in the storage decision.
The four networks, drawn
For the 64-node, 512-GPU example, count the endpoints for each job the network must do.
Compute fabric
- GPU collectives between nodes.
- Eight compute links per node give 512 node-to-leaf links in this example.
Storage fabric
- Dataset reads and checkpoint writes.
- Size the path for checkpoint bursts and any concurrent reads.
In-band management
- Provisioning, telemetry and package traffic.
- Count a management endpoint for each server.
Out-of-band management
- BMCs plus switches, power and cooling controllers.
- Keep an access path that can be tested when the production network is down.
Inventory management endpoints separately from compute nodes. Servers, switches, power equipment and cooling controllers may each need management connectivity. The out-of-band path is especially useful when the production network is unavailable, so test access under that failure condition.
Lead times decide the schedule
typical ordering, longest lead time first
1. facility work: power capacity, floor loading, water months to years
2. electrical distribution: busway, taps, upstream gear months
3. cooling: CDUs, manifolds, facility loop connections months
4. network switches weeks to months
5. compute nodes weeks to months
6. cables and transceivers weeks, and they are the
item most often wrong
7. spares order with the main purchase
the failure this ordering prevents
compute delivered into a hall that cannot power or cool it, which is the most expensive
idle asset in the industry
sanity: at $2.5 per GPU-hour, 512 GPUs sitting for one month of facility delay is
512 x 24 x 30 x 2.5 = about $921,000 of unrealized capacity, which is far more than
the cost of ordering the cooling early
What interviewers are listening for
Walk through the dependencies of a working cluster: compute, communication, storage, management, facility support and spares. Explain which traffic shares infrastructure and why. Use checkpoint size and the allowed save time to estimate storage bandwidth, then identify the actual long-lead items in the project.
Key takeaways
- Four networks, not one: compute fabric, storage fabric, in-band management, and out-of-band management reaching every BMC, PDU, switch and CDU.
- A 512-GPU scalable unit is 64 nodes in 16 racks with 8 leaf switches, roughly 1,024 compute-fabric links and a topology-dependent transceiver count.
- Size storage by checkpoint burst: 1.12 TB of training state for a 70B model in 60 seconds is 18.7 GB/s, which must be evaluated alongside the workload’s dataset reads.
- Check facility, electrical, cooling and hardware lead times together; order against the project’s actual critical path.
- A month of facility delay on 512 idle GPUs is about $921,000 at $2.5 per GPU-hour, which dwarfs the cost of ordering cooling early.