The GPU Cluster Design Round: How to Size a Training Cluster Out Loud

If you are preparing for a GPU cluster design round, here is the thing to internalise first: it is a sizing exercise, not a box-drawing exercise. The interviewer is listening for a chain of arithmetic that goes from the model and the dataset to a number of GPUs, a wall-clock time, an interconnect requirement and a cost, and then for the failure modes you name without being asked. Candidates who draw a clean architecture and cannot say how long the run takes lose this round routinely. Candidates who say "this is roughly ten days of work on a thousand GPUs, so here is what I am protecting" almost never do.

The chain, worked end to end

Take a common prompt: design the infrastructure to train a 70 billion parameter dense model on 2 trillion tokens.

Step 1: total compute. For a dense transformer, training FLOPs are approximately 6 times parameters times tokens. That gives 6 × 7e10 × 2e12 ≈ 8.4e23 FLOPs. The factor of 6 is two for the forward pass multiply-accumulate and four for the backward pass; say that out loud, because it shows the number is derived rather than recalled.

Step 2: per-GPU throughput. Take a current data-centre GPU at roughly 1,000 dense BF16 TFLOPs of peak. Nobody achieves peak. Realistic model FLOPs utilisation for a well-tuned large training run is about 0.35 to 0.5. Use 0.4, so about 400 TFLOPs, or 4e14 FLOPs per second, per GPU.

Step 3: GPUs and time. 8.4e23 / 4e14 ≈ 2.1e9 GPU-seconds, which is about 583,000 GPU-hours. On 1,024 GPUs that is roughly 570 hours, or about 24 days. On 4,096 GPUs, about six days, if scaling holds, and whether it holds is the next question.

Step 4: memory. Compute tells you how long. Memory tells you whether it runs at all. With mixed precision and a standard optimiser, budget roughly 16 bytes per parameter for weights, gradients and optimiser state, plus activations. For 70B that is about 1.1 TB before activations, which is far past a single GPU and past a single 8-GPU node's aggregate memory once activations are counted. That is what forces your parallelism strategy, and saying so in that order (memory forces the strategy, not preference) is the right framing.

Step 5: cost. 583,000 GPU-hours at a couple of dollars per GPU-hour is roughly a million dollars of compute for one run. Say the number. Design rounds at this level are budget conversations, and the training calculator will check your arithmetic while you practise.

Our napkin math track drills each link of this chain separately, which is the fastest way to make it automatic.

Parallelism, chosen rather than listed

Once memory has forced your hand, name the strategy and say why each axis is there.

Tensor parallel splits individual matrix multiplies across GPUs and communicates within every layer, so it wants the fastest links you have. Keep it inside a node, on NVLink, and typically at a degree of 8 or below.

Pipeline parallel splits layers across groups and communicates activations at the boundaries, which is much less traffic. It costs you the bubble, which shrinks with more micro-batches.

Data parallel replicates and all-reduces the gradients each step. This is where your fabric earns its money.

Sharded data parallel (ZeRO-style, or FSDP) trades extra communication for a large memory saving by sharding optimiser state, gradients and optionally parameters.

The good answer says which axis solves which constraint. The weak answer lists all four as though naming them were the same as choosing. Our distributed training questions work each of these to the level a round demands.

The interconnect arithmetic that decides the design

This is the part most candidates skip, and it is the part that determines whether the design works.

Data parallel training all-reduces the full gradient every step. For 70B parameters in BF16 that is 140 GB of gradient. A ring all-reduce moves approximately 2 × (N-1)/N × the data volume per rank, which for large N is close to 2 × 140 GB ≈ 280 GB per rank of traffic across the step.

Now put that against your fabric. At 400 Gb/s per GPU, call it 50 GB/s, 280 GB takes about 5.6 seconds if you are bandwidth-limited and perfectly efficient. If your step's compute time is 2 seconds, you have just designed a cluster that spends most of its life waiting on the network. That single comparison, compute time per step against communication time per step, is the most valuable sentence you can say in this round.

The fixes are then obvious and you can name them in order: overlap communication with the backward pass so gradients reduce as they are produced, increase the micro-batch so there is more compute per step to hide behind, use hierarchical reduction so intra-node NVLink absorbs part of the work, shard so each rank moves less, or compress. Our networking and storage questions cover the fabric side, and debugging a slow all-reduce works the diagnostic version of the same material.

Do not draw the fabric as a cloud. Say rail-optimised or fat-tree, say the per-GPU bandwidth you are assuming, and say the oversubscription ratio at the spine.

The failure modes to name before you are asked

At a thousand GPUs running for weeks, failures are not exceptional events; they are part of the schedule. Naming them unprompted is one of the clearest seniority signals available in this round.

Node failure. With enough nodes and enough days, you will lose one. The question is what it costs. Checkpoint size for a 70B model with optimiser state is on the order of a terabyte. How long does that take to write, how often do you write it, and what is the expected lost work between checkpoints given your failure rate? That is a real optimisation with a real answer, not a shrug.

Stragglers. One slow rank sets the pace for every collective. Causes include thermal throttling, a degraded link, ECC retries, and a noisy neighbour on shared storage. Say how you would detect it: per-rank step times, not just the average.

Silent corruption. Rare per device, and not rare across a fleet over weeks. Loss spikes that do not recover are the symptom.

Power and thermal limits at rack level. Racks have a power budget and a cooling budget, and dense GPU nodes push both. This is where hardware and deployment knowledge separates candidates who have deployed a cluster from candidates who have used one.

Our reliability and observability track covers the detection and response side of each of these.

How to run the 45 minutes

1. Clarify for two minutes. Model size, token budget, deadline, budget, whether the hardware is fixed. Do not skip this and do not spend ten minutes on it.
2. State the ceiling. Total FLOPs, achievable throughput, the resulting time on a candidate cluster size. Now everyone knows what is being optimised.
3. Let memory pick the parallelism. Show the memory budget, then choose the axes.
4. Check communication against compute. The sentence that decides the design.
5. Name the failure modes and what each costs.
6. Close with cost and the one thing you would change first if the deadline halved or the budget did.

If you practise only one thing, practise steps 2 and 4 out loud. They are the two moments where the round is won.

Work through the AI systems design questions for the reference shapes, and the must-know set to find which link of the chain you are slowest on.