TL;DR: Start with the workload and the site's usable power, cooling and space. Choose a supported node and fabric design, then derive the bill of materials and acceptance tests; a budget for 1,000 GPUs does not establish that the site can run them.
How to approach it
Ask what the cluster must train or serve, when it must be ready, and what the facility can deliver. Select a repeatable unit from a reference architecture that matches the actual hardware. Count the dependent infrastructure and define successful acceptance before placing orders.
A strong answer
Suppose the budget is for roughly 1,000 GPUs in eight-GPU nodes. That is 125 nodes. Choosing 128 nodes gives 1,024 GPUs and may fit a repeatable unit more neatly, but it changes the purchase and operating budget. Explain the reason for rounding instead of silently changing the target.
A reference architecture is a starting point for supported configurations, not a substitute for site design. NVIDIA's B300 SuperPOD architecture describes scalable units and their supporting networks. Use the exact variant's port map, power design and bill of materials; do not mix counts from Ethernet, InfiniBand, AC and busbar variants.
For an independent illustrative budget, assume 128 nodes, four nodes per rack and 50 kW per populated compute rack. These are chosen planning inputs, not certified requirements.
128 nodes × 8 GPUs = 1,024 GPUs
128 / 4 = 32 compute racks
32 × 50 kW = 1.6 MW compute-rack IT load
Assumed other IT load = 0.2 MW
At an assumed PUE of 1.25: (1.6 + 0.2) × 1.25 = 2.25 MW facility load
PUE multiplies the total IT load; it does not replace a cooling design. Check both aggregate facility capacity and what can be delivered to each rack, including the required redundancy state. A hall offering 32 positions at 15 kW has only 480 kW of aggregate capacity at those positions. It cannot serve this 1.6 MW rack budget without a different arrangement. Do not infer a precise supported GPU count by dividing those totals: node size, cooling, floor loading and feed limits still constrain placement.
| Design area | Quantity to establish |
|---|---|
| Compute | Node configuration, usable memory, supported parallel layouts |
| Scale-out fabric | NIC ports, link rates, topology and concurrent traffic |
| Storage | Dataset capacity, reads, checkpoint writes and restore bursts |
| Management | Provisioning, telemetry and independent recovery access |
| Facility | Rack load, cooling interfaces, floor and service access |
Count each physical link and both of its switch endpoints where appropriate. For example, 128 nodes with eight compute NIC ports expose 1,024 host-facing links. That number alone does not determine spine count or optics: link speed, switch radix, breakout configuration, uplink ratio and cable type decide the rest. A copper cable does not require two separately purchased optical transceivers. The Bill of Materials for a Training Cluster explains how to keep those categories separate.
Storage needs both capacity and performance requirements. If a chosen checkpoint recipe produces 1.12 TB and must make it durable in 60 seconds, the payload rate is 18.67 GB/s. That excludes replication, metadata and competing jobs. Measure a representative checkpoint and restore while training runs. A low average dataset read rate cannot rule out bursty demand or CPU-side preparation stalls.
Write acceptance around observed behavior. Inventory installed versions and topology, check device and link health, run numerical workloads, then test collectives within a node, across a pair and across the intended training group. Compare like-for-like message sizes, placement and software. A two-node result near a single-node result does not prove GPUDirect RDMA; use transport evidence and targeted tests as well. Set tolerances from the platform's qualified baseline rather than an arbitrary percentage of the fleet median.
The delivery plan should show quoted lead times and dependencies for power, cooling, switching, storage and compute. Long-lead items vary by site and supplier. Stage acceptance so a missing facility dependency is found before the whole fleet arrives, and include spare capacity and replacement procedures.
Choose the simplest supported design that meets the measured workload. Reconsider a full-bandwidth training fabric when replicas operate independently, but first check whether prefill/decode transfers, distributed inference or shared storage still require substantial network capacity. GPU count alone cannot choose the network.
What interviewers probe next
- "What if rack power is lower?" Compare fewer nodes per rack, another node design or another site, including their cost and topology effects.
- "How much oversubscription is acceptable?" Test the intended concurrent workload against its completion target; a universal ratio is not enough.
- "What proves delivery?" The agreed inventory, correctness, sustained workload and recovery tests pass.
- "Which order goes first?" Use the actual dependency schedule and supplier commitments, not a blanket hardware lead-time rule.
Common mistakes
- Equating purchase budget with usable facility capacity.
- Combining switch counts from different reference designs.
- Sizing storage only from dataset size or only from one checkpoint.
- Accepting device visibility as proof of useful training performance.
Key takeaways
- Eight-GPU nodes turn 128 nodes into 1,024 GPUs.
- The illustrative facility budget is 2.25 MW after including other IT load and PUE.
- A 1.12 TB checkpoint in 60 seconds requires 18.67 GB/s of payload service.