TL;DR: With an approximate N = 70.6B and D = 15T processed tokens, the dense training estimate is 6ND = 6.354 × 10²⁴ FLOPs. Six comes from forward multiplication plus two backward matrix products, each counted at two FLOPs per weight per token. This is a modeled work budget, excluding attention and recomputation unless explicitly added.
How to approach it
First define the parameter count and whether D counts processed token presentations. Derive the six from one trainable matrix, then substitute the values and check the exponent. State omitted work and the attention convention before using the result in a time estimate. Keep utilization out of the work count.
A strong answer
A FLOP is one floating-point operation, a single multiply or add. Two teams estimate the same run. One multiplies by two for the forward pass; the other multiplies by six. They have counted different jobs: ordinary inference and full training. The quickest way to resolve the disagreement is to follow one matrix through backward.
Training FLOPs: 6ND derives this using Y = XW. Forward costs about 2 × tokens × input_width × output_width FLOPs. Backward computes both the input gradient and the weight gradient, each with the same leading-order count. Their total is three products, hence six FLOPs per weight per token.
N = 70.6e9 approximate parameters
D = 15e12 processed training tokens
C = 6 × N × D
= 6 × 70.6 × 15 × 10²¹
= 6,354 × 10²¹
= 6.354 × 10²⁴ FLOPs
sanity: with a round 70B count, C = 6.3e24;
using 70.6B raises the estimate by about 0.86%.
A small matrix makes the factor of six visible. Use the conventional leading count of two FLOPs per multiply-add; this exercise omits biases and scalar operations. The product order below explains dependencies, not a mandatory serial kernel schedule.
The input count is approximate because not every stored parameter participates in the same dense operation. A lookup table and a vocabulary projection have different work. For detailed budgeting, sum actual layer operations and document shared or tied weights. The rough number should retain its assumptions rather than pretend to be an exact hardware instruction count.
| Counted component | FLOPs per token |
|---|---|
| Projection forward | Approximately 2N |
| Projection backward | Approximately 4N |
| Base training estimate | Approximately 6N |
| Attention and recomputation | Count separately |
The comparison table keeps the meaning of the six visible without adding an unrelated hardware diagram. No GPU specification is needed to derive this work budget.
Attention performs work between token positions. For an illustrative L = 80 layers, query width h = 8,192 and S = 8,192 positions, PaLM's full-square accounting convention adds 12LhS FLOPs per training token:
base = 6 × 70.6e9 = 423.6e9 FLOPs/token
full-square attention = 12 × 80 × 8,192 × 8,192
= 64.4245e9 FLOPs/token
relative addition = 64.4245 / 423.6 ≈ 15.21%
A causal kernel can skip masked token pairs. An ideal triangular pair count gives roughly half this correction, about 7.6%; tile boundaries and implementation determine actual work. Softmax and scalar operations remain separate. Therefore “attention adds a few percent” needs a sequence length and a convention. At longer contexts, budget it explicitly rather than relying on the short-context approximation.
For a run with multiple sequence lengths, calculate a token-weighted sum across phases. Fifteen trillion tokens all at 8k and fifteen trillion with a late 128k phase need different attention budgets, even though their 6ND term is identical.
Recomputation adds work when training rebuilds activations instead of retaining them. Repeating every projection forward during backward adds roughly 2ND to the projection count, taking that component from 6ND to 8ND. Selective checkpointing rebuilds only some operations, so eight is not a universal training factor. The optimizer also does work once per update; count it separately when material rather than hiding it in another 2ND term.
To turn work into time, GPU-Hours and Time to Train divides by GPU count times the declared dense peak times an MFU consistent with the same useful-work model. At an assumed 40% MFU and 989 TFLOP/s per GPU, the base work gives about 4.462 million GPU-hours. That is an assumed effective-throughput calculation, not the number of FLOPs the hardware executed. A tensor-active counter is not itself MFU.
I would use 6ND for an initial dense-model budget, then refine the largest omitted term before making a schedule commitment. If attention, adapter training or expert routing changes the operation mix materially, switch to a layer- or phase-specific count.
What interviewers probe next
- “Double context but keep token count fixed?” The base 6ND is unchanged; the full-square attention total roughly doubles because its per-token cost grows with context.
- “What if the dataset is used twice?” Count both token presentations in D. Unique dataset size alone undercounts work.
- “Can I divide by MFU to get hardware FLOPs?” No. MFU relates modeled useful throughput to peak. It does not measure instructions executed during stalls or recomputation.
- “What changes for MoE?” Count participating experts, shared layers and routing. Total resident parameters overcount a token's selected expert computation, but a single advertised active count can still omit padding and imbalance.
Common mistakes
- Counting only forward work for full training.
- Treating attention's percentage as independent of context and causal work skipping.
- Counting unique data instead of actual training presentations.
- Calling C/MFU a measured executed-FLOP total.
Key takeaways
- 6 × 70.6e9 × 15e12 = 6.354e24 modeled FLOPs.
- The six is two forward plus four backward.
- The stated 8k full-square attention correction is 15.21%; a triangular pair count differs.
- Hardware time requires a matching throughput model and additional schedule accounting.