Napkin Math for AI Infrastructure Interviews: The Numbers You Must Be Able to Derive
Almost every AI infra round has arithmetic hiding in it. Here are the six chains worth memorising, the constants you need, and the derivations, so you can size a KV cache, a training run or a serving fleet without a calculator.
BY BRANDON SULLIVAN · AIINFRAINTERVIEWS EDITORIAL · UPDATED SEPTEMBER 6, 2026 · 11 MIN READ
PRACTICE THIS:Napkin math questions ·The formula sheet ·KV cache calculator ·Cost per token calculator
Nearly every AI infrastructure interview round has arithmetic hidden inside it. The design round asks how many GPUs. The serving round asks how many concurrent requests fit. The debugging round asks whether 8 seconds is slow. The cost round asks what a million tokens costs. In every case the candidates who do well are not the ones with better intuition, they are the ones who have six specific chains at their fingertips and can narrate them out loud without a calculator.
Here are the six, with the derivations.
1. Model memory from parameters
Bytes per parameter by precision: FP32 is 4, BF16 and FP16 are 2, FP8 is 1, INT4 is 0.5.
For inference, weights alone: parameters × bytes per parameter. A 70B model in BF16 is about 140 GB, which does not fit on one 80 GB GPU, which is why it is served across at least two. In FP8 it is about 70 GB, which just about does, and that single fact drives a lot of quantisation decisions.
For training with mixed precision and a standard optimiser, budget roughly 16 bytes per parameter: 2 for BF16 weights, 2 for BF16 gradients, 4 for the FP32 master copy, and 8 for the two optimiser moments. So 70B needs about 1.1 TB before activations. This is the number that forces your parallelism strategy.
2. KV cache bytes per token
This is the most useful single formula in serving interviews, because the KV cache, not the weights, is what limits concurrency.
Bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_element
The 2 is for K and V. Note kv_heads, not attention heads: with grouped-query attention, many query heads share one key-value head, and the ratio is often 4 to 8, which cuts the cache proportionally. Candidates who use the attention head count overestimate by that factor, and it is the most common error in the whole topic.
Worked: 80 layers, 8 kv heads, head dimension 128, BF16 at 2 bytes. That is 2 × 80 × 8 × 128 × 2 = 327,680 bytes, so about 320 KB per token. A 4,000-token conversation costs about 1.3 GB. If you have 40 GB left after weights on a GPU, you fit roughly 30 such conversations, and that is your concurrency limit.
The KV cache calculator will check your working, and the inference and serving questions cover what to do when the number is too small.
3. Training FLOPs
Total training FLOPs ≈ 6 × parameters × tokens for a dense transformer. The 6 is 2 for the forward multiply-accumulate and 4 for the backward pass, which computes gradients with respect to both inputs and weights.
Per-GPU achievable throughput = peak FLOPs × model FLOPs utilisation, where a well-tuned large run lands around 0.35 to 0.5. Use 0.4 and say you are using it.
Worked: 70B parameters on 2T tokens is 6 × 7e10 × 2e12 = 8.4e23 FLOPs. On a GPU with 1,000 peak BF16 TFLOPs at 40 percent, that is 4e14 FLOPs per second, so 2.1e9 GPU-seconds, or about 583,000 GPU-hours. On 1,024 GPUs, about 24 days. The training calculator checks the chain, and the GPU cluster design round works it into a full design answer.
4. Arithmetic intensity and the ridge point
Arithmetic intensity is FLOPs per byte moved. The hardware ridge point is peak FLOPs divided by memory bandwidth. Below it you are bandwidth-bound; above it, compute-bound. One division tells you which optimisation is worth doing.
For a part with 1,000 TFLOPs of peak and 3 TB/s of bandwidth, the ridge point is about 333 FLOPs per byte. That is a high bar, and it explains the central fact of LLM serving: decoding one token at batch size 1 reads the entire weight matrix to do a single matrix-vector product, so its intensity is around 2, which is a hundred times below the ridge. Decode is bandwidth-bound, and batching is the fix because it amortises the same weight read across more work. Prefill processes many tokens at once, is compute-bound, and needs different attention entirely.
This is the reasoning behind continuous batching, speculative decoding and disaggregated prefill and decode, all covered in the GPU architecture track.
5. Collective communication time
Gradient bytes = parameters × bytes per element. A ring all-reduce moves about 2 × (N-1)/N × that volume per rank, which for large N is close to twice the gradient size.
Fabric bandwidth is quoted in bits. Divide by 8. 400 Gb/s is 50 GB/s. This is where a lot of otherwise correct chains go wrong by a factor of 8.
Worked: 70B in BF16 is 140 GB of gradient, so about 280 GB per rank of traffic, so at 50 GB/s about 5.6 seconds if the fabric is the limit. Compare that against your compute time per step, and you know immediately whether the design is compute-bound or communication-bound. That comparison is the most valuable sentence in a cluster design round, and it is also step zero in debugging a slow all-reduce.
6. Cost per million tokens
Tokens per GPU-hour = output tokens per second per GPU × 3,600. Dollars per million tokens = (GPU hourly rate ÷ tokens per GPU-hour) × 1,000,000.
Worked: a GPU at 2 dollars an hour producing 2,000 output tokens per second yields 7.2 million tokens per hour, so about 28 cents per million output tokens at full utilisation. Real fleets do not run at full utilisation, so divide by your actual utilisation, and remember that prefill costs compute too even though it produces no output tokens.
The cost per token calculator is the fast way to sanity check these, and the napkin math questions walk variations.
The constants worth carrying
- Bytes per parameter: 4 (FP32), 2 (BF16), 1 (FP8), 0.5 (INT4)
- Training memory: about 16 bytes per parameter with a standard optimiser, before activations
- Training FLOPs: 6 × parameters × tokens
- Realistic model FLOPs utilisation: 0.35 to 0.5 for large runs
- Ring all-reduce traffic: about 2× the gradient volume per rank
- Network is bits, memory is bytes: divide by 8
- Seconds per day: 86,400. Hours per month: about 730.
How to say it out loud
The arithmetic is only half the skill. The other half is narrating it without losing your place, because these rounds are verbal.
State each assumption as you use it: "call it a thousand teraflops peak, forty percent utilisation, so four times ten to the fourteen per GPU per second." Round aggressively and say which way you rounded. Keep the powers of ten written down where you can see them, because that is where mistakes actually happen, not in the formulas.
And say the answer's meaning, not just the number. "Twenty-four days on a thousand GPUs, so if the deadline is two weeks we need to talk about either more GPUs or fewer tokens" is a design conversation. "2.1e9" is a number.
The formula sheet has all of these in one place for revision, and the must-know questions will show you which chain you are slowest on.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
Six chains cover most of it: model memory from parameter count and precision, KV cache bytes per token, training FLOPs as roughly 6 times parameters times tokens, arithmetic intensity against the hardware ridge point, all-reduce time from gradient size and fabric bandwidth, and cost per million tokens from throughput and GPU hourly rate. Everything else is a variation on one of these.
Discussion (5)
The thing that actually gets people is not the formulas, it is the unit hygiene. Bits versus bytes on the network, GB versus GiB in memory, and per-GPU versus per-node bandwidth. I have seen a candidate produce a beautiful chain that was off by 8x because 400 Gb/s quietly became 400 GB/s halfway through.
The network one is universal. Fabric vendors quote bits, memory vendors quote bytes, and the factor of 8 sits right in the middle of every design round.
For serving rounds, learn the KV chain cold because it is the capacity limit in practice, not the weights. Weights are a fixed cost you pay once. KV is per concurrent request and it is what decides how many users fit on the box.
Add the ridge point to the memorise list. Peak FLOPs divided by memory bandwidth gives you the arithmetic intensity where a kernel flips from bandwidth-bound to compute-bound. It is one division and it explains why decode is slow and prefill is not.
Practice out loud, not on paper. The round is verbal and there is a real skill in narrating 'call it 4e14, so about 2e9 GPU-seconds, which is roughly 580 thousand GPU-hours' without losing your place. Silent arithmetic reads as uncertainty even when it is correct.