TL;DR: Derive the fleet from model work and calendar time, then place parallel groups within explicit network boundaries. In the worked example, 350B dense parameters and 15T tokens require about 90 active days on 10,240 SXM GPUs at an assumed 40% active MFU, or 100 calendar days at 90% retained-progress availability. Checkpoint durability, fabric capacity, power and replacement capacity must support that schedule.
How to approach it
Ask whether this is one large run or a shared fleet, the model and sequence length, deadline, precision, facility limits and acceptable recovery time. Derive compute first. Draw logical placement before physical ports. Then size checkpoint traffic, power and failure recovery. State which numbers are assumptions to benchmark rather than purchased guarantees.
A strong answer
Ten thousand GPUs do not by themselves tell you when a training run will finish. Some hours advance the model; others go to repairs, replay or waiting. Define which time each efficiency factor covers so the estimate neither ignores lost progress nor charges for it twice.
Example work: C ≈ 6ND = 6 × 350e9 × 15e12 = 3.15e25 FLOPs
Assumed dense BF16 peak P = 989e12 FLOP/s per H100 SXM
Useful active rate = 10,240 × P × 0.40 = 4.051e18 FLOP/s
Active time = C / rate ≈ 90 days
Calendar time = 90 / 0.90 ≈ 100 days
Sanity: active efficiency and retained-progress availability have different denominators.
The 6ND estimate omits architecture-dependent attention and other costs. Calibrate it against the real model. NVIDIA's H100 specifications label the advertised BF16 tensor rate as sparse; the calculation uses the approximately 989 TFLOP/s dense rate. The 40% and 90% factors are planning assumptions, not measurements of this proposed cluster. Define active MFU over executing training intervals, and let the second factor account for lost progress and excluded downtime once.
One possible placement uses Tensor, Pipeline and Data Parallelism: TP 8 across the GPUs in a node, PP 8 across eight nodes, and DP 160 across the fleet. Each model replica occupies 64 GPUs. Group sixteen replicas into a pod and ten pods give 10,240 GPUs, with 128 nodes per pod. This geometry still needs to pass the model’s layer-divisibility, activation-memory and microbatch checks. Pipeline Bubbles can make a layout slow even when everything fits.
For an auditable local fabric example, consider one rail serving 128 node ports: four leaves, each with 32 host ports and 32 equal-speed uplinks; eight spines, each receiving four links from each leaf. That is 128 host links plus 128 leaf-spine links, with each spine using 16 ports. At equal link rates the leaf uplink sum matches its host-facing sum. Routing, switch fabric capacity and multiple paths still need validation. Eight independent rails repeat this local construction; inter-pod ports and links are an additional design, not hidden in the local count. NVLink within nodes, compute rails, storage and management networks have different jobs.
Do not declare a 2:1 inter-pod core adequate without a collective trace. In a balanced TP8/PP8 partition, BF16 gradients for 350B parameters are about 10.94 GB per model-parallel rank before data-parallel sharding details. A ring all-reduce over 160 replicas moves roughly 2 × 159/160 times its payload per participant. Distributed optimizers and hierarchical collectives change where traffic occurs; overlap changes exposed time. Benchmark the chosen algorithm across pod boundaries, including a failed link, before accepting oversubscription.
Checkpoint Storage follows saved state, not only weights. One explicit recipe is 2 bytes of BF16 weights, 4 bytes of FP32 master weights and 8 bytes of Adam moments per parameter: 4.9 TB for 350B. Saving gradients adds another 0.7 TB if the format requires them; RNG, loader and metadata add more. Writing 4.9 TB in 60 seconds needs about 81.7 GB/s aggregate payload throughput. Local staging can shorten the stall, but a local file is not yet protected from node loss. Bound the asynchronous backlog and publish a manifest only after required shards are durable. Megatron Core's checkpoint documentation also distinguishes formats and supported resharding; test restoration with the actual parallel layout.
A toy independent job-fatal hazard of 2e-5 per GPU-hour gives 10,240 × 2e-5 = 0.2048 events/hour, or 4.88 hours between interruptions. This assumption excludes correlated outages and recoverable errors. Separately, a node replacement rate of 0.002/day and three-day repair delay implies 1,280 × 0.002 × 3 = 7.68 nodes unavailable on average. A spare pool needs a chosen service percentile and rack/pod failure scenarios, not just eight rounded-up nodes. Checkpoint Cadence and restore drills determine retained progress.
For a power-sizing exercise, assume a measurement of 10 kW per eight-GPU node. The 1,280 nodes then use 12.8 MW of IT power. Add the network and storage load before applying facility overhead. GPU TDP alone cannot size the utility feed. Facility engineering must verify rack delivery, cooling and operation under failures, while spare capacity must match the topology of the jobs it will replace.
I would start with a homogeneous pod design, validated collective routing and durable checkpoint commits. I would relax fabric or spare capacity only after a representative training run and failure drill preserve the schedule. Watch p99 step time, exposed collective time, durable-checkpoint age, restart duration and available whole-job placements. Training Uptime and Interruption Statistics connects these signals to retained progress; Topology-Aware Scheduling explains why free GPU count alone is insufficient. High device utilization with little retained progress is an unsuccessful cluster.
What interviewers probe next
- What changes if a pod fails? Identify which DP participants disappear, whether the framework can resume with a smaller world size, and whether optimizer and loader state support that change. A spare GPU in another pod may not preserve the required topology.
- Can the core be oversubscribed? Replay the actual collective traffic and failure paths. Accept reduced capacity only when exposed communication and retained progress still meet the deadline.
Common mistakes
- Using sparse peak FLOPs for dense training, or applying downtime twice through overlapping efficiency factors.
- Calling local checkpoint staging durable, or using mean repair demand as a spare-capacity guarantee.
Key takeaways
- The example needs about 90 active days and 100 calendar days; those are different quantities.
- TP8 × PP8 × DP160 gives 10,240 GPUs, but memory and pipeline efficiency must validate the layout.
- A 4.9 TB state recipe needs 81.7 GB/s to finish writing in 60 seconds, before storage overhead.
- Spare capacity comes from repair and correlated-failure scenarios; an average is not a reliability guarantee.