Napkin Math for AI Infrastructure Interviews: Six Worked Estimates

A useful infrastructure estimate carries its assumptions from the workload to the decision. The six examples below connect model size to memory, training time, communication and cost. They are simplified calculations for practice, not measured performance results or predictions of a particular interview.

Use the formula sheet for revision and the napkin math questions for variations. When the estimate is close to a memory limit or deadline, refine it; an arbitrary factor-of-two tolerance is not enough for every decision.

1. Count model state before choosing devices

For an illustrative dense model with 70 billion parameters, two-byte weights occupy 140 billion bytes, or 140 decimal GB. That cannot fit on an 80 GB device. Splitting it evenly across two devices gives 70 GB of weights per device before cache and runtime allocations; it does not prove that a particular engine layout will fit.

One-byte weights give a 70 GB raw payload. Quantization metadata, higher-precision tensors and runtime memory still need space. The worked Llama memory answer distinguishes an actual model configuration from this rounded example.

An illustrative mixed-precision Adam layout uses 16 bytes per parameter: two for weights, two for gradients, four for a master copy and eight for two moments. That is 1.12 TB for 70B parameters before activations. Optimizer and precision choices can change this layout; inspect the implementation before treating it as a requirement.

2. Derive KV cache with the correct head count

For full-attention MHA or GQA, raw retained state is:

bytes per token = 2 × layers × KV heads × head dimension × element bytes
                = 2 × 80 × 8 × 128 × 2
                = 327,680 bytes = 320 KiB

For an illustrative 4,000-token sequence, that is 1.31072 decimal GB across the replica. A 40 GB cache budget would hold floor(40 / 1.31072) = 30 such sequences before overhead, under the assumed placement. Uneven rank allocations can lower the limit.

Use KV heads, not query heads. The KV-cache walkthrough links the model configuration and explains paging, hybrid architectures and per-device checks. The calculator varies the arithmetic without turning a memory cap into a throughput claim.

3. Convert training work into time

The simplified dense-transformer estimate is 6ND FLOPs, with N parameters and D training tokens. It omits attention-specific work and other costs; long contexts can make those omissions significant. The PaLM paper’s utilization discussion is a useful reference for keeping model-work conventions consistent.

With N = 70B and D = 2T, work is 8.4 × 10²³ FLOPs. Assume, for this exercise, 1,000 dense BF16 TFLOPs/s peak and 40% utilization under the same work convention. Effective throughput is 4 × 10¹⁴ FLOPs/s per GPU.

GPU-seconds = 8.4e23 / 4e14 = 2.1e9
GPU-hours   = 2.1e9 / 3,600 ≈ 583,333
days on 1,024 GPUs = 2.1e9 / (1,024 × 86,400) ≈ 23.74

This assumes the effective rate holds at that scale. Recovery and idle periods need consistent treatment in the measurement. The training calculator makes these inputs explicit.

4. Compare arithmetic intensity with a hardware bound

An illustrative part with 1,000 TFLOPs/s compute and 3 TB/s memory bandwidth has a ridge point of about 333 FLOPs per byte. For a batch-one dense matrix-vector operation with BF16 weights, approximately 2P FLOPs over 2P weight bytes gives about 1 FLOP per byte, ignoring other traffic and reuse.

That places the operation below the model’s ridge point. It does not prove saturated bandwidth: small kernels, dependencies or launches may limit actual performance. Larger batches can reuse weights across more work. Prefill and decode behavior also depends on shapes and context, so neither is universally one bottleneck. The roofline calculator checks the bound.

5. Keep collective payload and path together

For N ranks and a payload S per rank, an ideal ring all-reduce transfers 2(N−1)/N × S per rank. NVIDIA’s nccl-tests performance explanation distinguishes algorithm bandwidth from normalized bus bandwidth.

A 400 Gb/s link is 50 GB/s before overhead. At large N, a 140 GB payload gives about 280 GB of ideal ring traffic and a 5.6-second bandwidth-only bound on a dedicated path. This is a full-gradient data-parallel example; model sharding, precision and collective choice change S. Compare exposed communication with compute after accounting for overlap and contention.

6. Price useful output

Assume a hypothetical $2 per GPU-hour and 2,000 accepted output tokens per second per GPU over the billed interval. That gives 7.2 million tokens per hour and about $0.278 per million tokens in GPU cost. The price is illustrative, not a current quote.

If the rate already averages idle time, do not divide by utilization again. If it is a saturated benchmark, incorporate the actual duty cycle separately. Add the other billed resources and include prefill work in the workload being measured. The cost calculator helps keep the numerator and denominator consistent.