GPU Cluster Design Practice: Size a 70B Training Run

To size a training cluster, derive the model’s work, estimate time at an explicit effective throughput, check per-rank memory and communication, then account for recovery and cost. The worked example below uses a hypothetical dense 70-billion-parameter model trained on two trillion tokens. It is a design rehearsal, not a hardware quote or a report of a company’s interview.

The training calculator lets you vary the inputs. Keep the same workload through the entire chain so that the memory, time and cost results describe one proposed run.

Compute gives a first schedule

Using the simplified 6ND dense-transformer approximation, the model work is 6 × 70e9 × 2e12 = 8.4e23 FLOPs. This approximation leaves out attention-specific and other work. The PaLM paper discusses model FLOPs utilization; use a consistent model-work convention when comparing estimates with reported utilization.

Assume an illustrative peak of 1,000 dense BF16 TFLOPs/s per GPU and an effective utilization of 40%. That gives 4e14 model FLOPs/s per GPU. It is an input to this exercise, not a claim about a well-tuned fleet.

GPU-seconds = 8.4e23 / 4e14 = 2.1e9
GPU-hours = 2.1e9 / 3,600 ≈ 583,333
time on 1,024 GPUs = 2.1e9 / 1,024 / 86,400 ≈ 23.74 days

If effective throughput falls to 30% of peak under the same convention, the estimate rises to about 31.65 days. Adding devices helps only if scaling and availability preserve the assumed useful rate. The napkin math walkthrough expands the units.

Memory constrains the parallelism plan

For an illustrative 16-byte-per-parameter mixed-precision Adam state layout, weights, gradients, master weights and moments total 1.12 TB before activations. Different optimizers and precision policies change this figure. Temporary buffers and the largest local layer matter too.

Aggregate capacity is insufficient evidence of fit. Draw what each rank owns under tensor, pipeline or sharded data parallelism, then add its local activations and peak temporary allocations. One full rank can fail even when the sum of free memory across the cluster looks adequate.

Use tensor parallelism where its frequent communication fits the topology, and pipeline parallelism where layer partitioning and the microbatch schedule make sense. Sharded data parallelism reduces replicated state at the cost of different communication. The parallelism comparison explains what each method shards; the training track develops the constraints.

Derive traffic for the chosen layout

As a separate communication baseline, consider full-gradient data parallelism with BF16 gradients for the entire 70B model. Its payload is 140 GB per rank. An ideal ring all-reduce transfers 2(N−1)/N times that payload per rank, approaching 280 GB at large N. NVIDIA’s nccl-tests documentation explains the normalized bandwidth convention.

A dedicated 400 Gb/s path provides at most 50 GB/s before overhead, giving an ideal bandwidth-only time near 5.6 seconds for that large-N traffic. This baseline is not the payload for every sharded design. Recompute bytes and collective types for the actual layout, gradient dtype and accumulation schedule.

Compare the communication exposed on the critical path with compute. Overlap can hide part of it; contention, late ranks and shared uplinks can expose more. Hierarchical algorithms can change where traffic flows, while additional sharding can introduce gathers. The all-reduce debugging guide separates these hypotheses.

Recovery belongs in the capacity estimate

Specify what the checkpoint contains, how long writing and restoring take, and how often the job saves state. Use the target fleet’s observed interruption behavior when available. A published failure count from another training run is a reference case, not the failure rate of this design.

On a suspected hardware problem, preserve the diagnostic evidence and keep unhealthy resources out of new placements. Define how a node passes acceptance before returning. For numerical anomalies, investigate data, optimizer behavior, software and hardware; a loss spike alone does not diagnose silent corruption.

The reliability track covers these decisions. The hardware deployment track adds power, cooling and physical acceptance, which GPU-count arithmetic alone cannot establish.

A checkpoint acceptance test should restore into a fresh process and verify the resumed step, optimizer state and input position. Record restore duration under the storage contention expected during recovery. A quiet single-node test can underestimate the delay when many workers restart together.

Close the estimate with a cost and a reversal condition

At a hypothetical $2 per GPU-hour, 583,333 GPU-hours costs about $1.17 million in GPU time. This excludes separately billed storage, networking and other services. It is not a current rental price. Check whether downtime is already included in the effective throughput before charging it again in the estimate.

If the deadline is 30 days, the 40% case leaves limited room for additional delays, while the 30% case already misses it. That sensitivity is a reason to measure a representative scale before committing. It may support more resources, fewer training tokens or a different schedule; explain which constraint the change addresses.

Use the systems design questions to rehearse the decision. Present the bound, the unverified assumptions and the experiment that would most change the plan.