TL;DR: Training compute ≈ 6 × N × D = 6 × 70.6e9 × 15e12 ≈ 6.4e24 FLOPs. The 6 is 2 FLOPs per parameter per token in the forward pass (one multiply, one add) plus 4 in the backward pass (gradients with respect to activations and to weights). Attention over the sequence adds a few percent at 8k context and is left out of the napkin figure.
How to approach it
Confirm the two inputs, parameter count and token count, and ask whether the interviewer wants the model FLOPs or the hardware FLOPs after utilization (the first is 6ND; the second divides by MFU and is a different question). State the formula and explain the 6 before substituting, because the explanation is the part that shows understanding. Then do the exponent arithmetic slowly and out loud, and check the order of magnitude against a known reference: Llama 3.1 405B's compute budget is in the same decade.
A strong answer
A typical situation: a planning meeting needs a GPU count and a date, and every number in the conversation hangs off one figure nobody has computed yet. It is one multiplication, and getting the 6 wrong scales everything downstream.
The 6ND rule for training FLOPs says compute = 6 × parameters × tokens. It comes from counting what a dense matrix multiply costs per parameter per token:
- Forward: every parameter participates in one multiply-accumulate per token, which is 2 FLOPs (a multiply and an add). So 2N per token.
- Backward: two matrix multiplies of the same size, one for the gradient with respect to the layer's input (needed to keep propagating) and one for the gradient with respect to the weights. That is 2 × 2N = 4N per token.
- Total: 6N per token, and 6ND over the run.
inputs: N = 70.6e9 parameters
D = 15e12 tokens
C = 6 × N × D
= 6 × 70.6e9 × 15e12
= 6 × 70.6 × 15 × 1e21
= 6,354 × 1e21
≈ 6.35e24 FLOPs
sanity: Llama 3.1 405B on 15.6T tokens is 6 × 405e9 × 15.6e12 ≈ 3.8e25, about six times this,
which matches the ratio of parameter counts (405 ÷ 70.6 ≈ 5.7). Both sit in the 1e24 to 1e25
decade that frontier-scale open models have occupied.
The rule ignores attention's own arithmetic, the score and context matrix multiplies that scale with sequence length rather than parameter count. Per token per layer they cost roughly 4 × sequence × hidden FLOPs forward; at 8,192 context on this model that is about 4 × 8,192 × 8,192 ≈ 2.7e8 per layer, or 2.1e10 across 80 layers, against 2N = 1.4e11 for the parameter term. So attention adds about 15% forward at 8k, less at shorter training sequence lengths, and it is standard to leave it out of the estimate and say so. At 128k context the same term would dominate, which is why long-context training is expensive out of proportion to token count.
What makes the number useful is what it converts into. Divide by a fleet's effective throughput and you have time to train; divide by a deadline and you have a GPU count; multiply by a price per FLOP and you have a bill. All three are separate questions in this bank, and all three start from 6.35e24. The 6 is also the reason inference is cheap by comparison: a generated token costs 2N FLOPs, a third of a training token, and a training run's 15T tokens equal the compute of 45T generated tokens.
The reversal condition, which is the mixture-of-experts variant of the rule: for a MoE model, N in 6ND is the active parameter count, not the total, because inactive experts do no arithmetic. DeepSeek-V3 trains at 6 × 37e9 per token, not 6 × 671e9, which is how a 671B model can be trained for less compute than a dense 70B on the same tokens.
Training FLOPs: 6ND and GPU-Hours and Time to Train are the two pages this figure feeds, and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE on a live run is how the assumed MFU gets checked against the real one.
What interviewers probe next
- "Where does the 6 come from?" Two forward (multiply, add) plus four backward (two same-sized matmuls: gradient to input, gradient to weights).
- "Does that include attention?" No; the attention term scales with sequence length squared and adds about 15% at 8k for this model. Say it is excluded.
- "What is the FLOPs figure for one inference token?" 2N, about 1.4e11 for the 70B; prefill of an 8k prompt is 2N × 8,192 ≈ 1.2e15.
- "Convert it to H100 time." At 989 TFLOPS dense bf16 and 40% MFU, one card does 3.96e14 per second, so 6.35e24 ÷ 3.96e14 ≈ 1.6e10 GPU-seconds, about 4.5 million GPU-hours.
Common mistakes
- Using 2ND (forgetting the backward pass) or 8ND (adding an imaginary optimizer term; the optimizer update is order N per step, negligible against 6N per token).
- Getting the exponent wrong by a factor of a thousand from mixing 1e9 and 1e12 carelessly; write the mantissa and the exponent on separate lines.
- Using total parameters for a MoE model.
- Confusing model FLOPs with hardware FLOPs and quoting 6ND as if it were what the cluster executed at 100% utilization.
Key takeaways
- C = 6ND; 6 = 2 forward + 4 backward. For a 70B on 15T tokens, 6.35e24 FLOPs.
- Attention is excluded and adds about 15% at 8k context; say so.
- Inference is 2N per generated token, one third of training per token.
- For MoE, N is active parameters; memory follows total, compute follows active.