Training FLOPs: 6ND
Derive the six from one forward and two backward matrix products. Count training tokens, attention and recomputation consistently before turning model work into a schedule.
TL;DR: For a dense trainable matrix, forward multiplication and its two backward matrix products each cost about two FLOPs per weight per token. That gives the approximation C = 6ND for N participating parameters and D processed training tokens. Attention, recomputation and other operations need separate accounting. A utilization factor converts a stated work budget into time; it does not turn that budget into a measurement of executed FLOPs.
Rebuild the six from one layer
A FLOP is one floating-point operation, such as a single multiply or a single add. Counting them gives a hardware-independent size for a training run, which you can then divide by how many operations per second your GPUs sustain.
Suppose you know how to multiply an input X by weights W to produce Y. Training must also work backward from the loss: how should W change, and how should the input to this layer change? Those are two separate matrix products.
Take a toy layer with T = 10 token vectors, input width 4 and output width 3. It has 12 weights. Let G be the gradient arriving at Y. The three products are:
| Product | Shapes and modeled FLOPs |
|---|---|
| Forward Y = XW | (10 × 4)(4 × 3): 240 |
| Input gradient GWᵀ | (10 × 3)(3 × 4): 240 |
| Weight gradient XᵀG | (4 × 10)(10 × 3): 240 |
The conventional count treats a multiply-add as two FLOPs, so each product costs 2 × 10 × 4 × 3 = 240. Their sum is 720, equal to 6 × 12 weights × 10 tokens. The weight gradient is an input to the optimizer; calculating it is not itself the optimizer update.
Scale the count to a run
When most work comes from dense trainable matrices, add their counts and approximate the result as 6N FLOPs per token. Define N consistently: an embedding lookup is not a dense multiplication over its entire table, while a vocabulary output projection can be substantial. A model-family parameter label is useful for rough planning, but exact accounting follows the architecture and tensor operations.
illustrative dense run:
N = 70.6 × 10⁹ parameters
D = 15 × 10¹² processed tokens
C = 6ND = 6 × 70.6 × 15 × 10²¹
= 6.354 × 10²⁴ FLOPs
sanity: doubling D doubles this modeled work at fixed N.
To turn work into time, divide by the delivered rate: number of GPUs × peak FLOPs per second per GPU × MFU (the fraction of peak the run actually sustains, defined below). For an illustrative 16,384 GPUs at a BF16 dense peak of 989 × 10¹² FLOP/s and 40% MFU:
delivered rate = 16,384 × 989 × 10¹² × 0.40 = 6.48 × 10¹⁸ FLOP/s
time = 6.354 × 10²⁴ ÷ 6.48 × 10¹⁸ ≈ 980,000 s ≈ 11.3 days
sanity: halving MFU to 20% doubles the time to about 22.7 days
D counts token presentations used by the run. If the same dataset is processed twice, those repeated presentations count twice. Padding, masked losses and repeated steps require care: a token can incur computation even when it does not contribute a loss. Keep useful-data accounting and executed-operation accounting distinct.
Attention depends on sequence length
The query/key score product and probability/value product involve token pairs rather than learned weight matrices. For L layers with total query width h and sequence length S, a full-square attention count for the two forward products is about 4LS²h per sequence. Including their backward products gives about 12LSh per token. PaLM's compute appendix uses this correction alongside 6N.
For the illustrative dimensions L = 80 and h = 8,192, the full-square correction is 64.42 billion FLOPs per token at S = 8,192. The 6N term is 423.6 billion, so the correction is 15.21%. At S = 131,072 it is about 1.03 trillion FLOPs per token, larger than the parameter term.
Causal attention permits only S(S+1)/2 token pairs. A kernel that skips masked work can approach half the full-square pair count; blocked implementations need not achieve the exact triangular minimum. Under that pair-count convention, the 8k correction is about 32.22 billion FLOPs per token, or 7.61%. Label the convention instead of claiming a universal “few percent at 8k.” Softmax and other operations are still outside these matrix-product estimates.
Long-context training often changes sequence length during the run. Sum each phase's token count times its chosen FLOPs-per-token model, rather than applying the final context length to every training token.
Required work and executed work differ
Activation Checkpointing discards some intermediates and recomputes them during backward. If every counted projection forward is repeated once, that projection component rises from 6N to about 8N FLOPs per token. Selective recomputation produces another amount. This can make training faster overall when the saved memory enables a better configuration, even though more arithmetic is executed.
Model FLOPs utilization, MFU, counts a declared useful-work model per unit time relative to hardware peak. Hardware FLOPs utilization can include recomputation. Dividing C by MFU gives neither an instruction count nor proof that the missing fraction was spent on extra arithmetic. The PaLM appendix reports the two accounting choices separately.
Where the shortcut needs another model
For a mixture of experts, use the expected participating expert computation plus shared layers and routing work, rather than total resident parameters. Token imbalance, padding and capacity policies can make actual work differ from a simple active-parameter figure.
For adapter training, frozen base weights do not need weight gradients, but input gradients may still be needed to reach earlier trainable adapters. Full-model 6ND therefore does not directly describe the run. Model Memory Footprint explains the corresponding storage distinction.
Inference has only forward computation, commonly approximated by 2N per processed token plus attention. Its latency also depends on traffic and execution layout; this FLOP count cannot establish single-stream speed. See Bandwidth-Bound Decode Throughput for that separate estimate.
Check understanding
An interviewer doubles the training sequence length while holding the total token count fixed. The 6ND term stays fixed. The full-square attention correction per token doubles, so its run total doubles. This follows from having half as many sequences, each with roughly four times as many token pairs.
What to remember
- Six comes from three matrix products at two FLOPs per multiply-add.
- The worked run requires 6.354 × 10²⁴ modeled projection FLOPs, about 11.3 days on 16,384 GPUs at 989 TFLOP/s and 40% MFU.
- Declare whether attention counts full-square or skipped causal work.
- Recomputed work and useful model work use different numerators.
- State parameter and token-count conventions before converting FLOPs into a schedule.