TL;DR: MFU is useful model FLOPs divided by the fleet's declared peak work over the same interval. With 70.6B parameters, four million global tokens, 512 H100s, a 989 TFLOP/s dense BF16 peak and 6.2 seconds, the 6N convention gives 53.97%. That percentage is a calculation to validate and compare with a matching workload, not a universal grade for the system.
How to approach it
First define what a token count includes and whether it is global or per data-parallel replica. Name the useful FLOPs convention, GPU peak and timing boundaries. Compute the ratio with units, then compare raw throughput with a like-for-like reference. Use a trace to investigate bottlenecks rather than diagnosing them from the percentage alone.
A strong answer
Model FLOPs utilization (MFU) asks what fraction of the GPUs' peak arithmetic went into the model's useful work. The dashboard says 100% GPU activity, but useful training throughput fell. That can happen while kernels stay active: the run may spend more time waiting on memory, repeating work or communicating. MFU and HFU separates useful progress from counted hardware arithmetic.
Take a hypothetical full training-step interval with these inputs. The token total counts the global data batch once, after choosing a policy for padding and replayed steps; it is not the sum of duplicate tensor-parallel counters.
N = 70.6e9 approximate dense parameters
T = 4.0e6 global processed tokens
G = 512 GPUs
P = 989e12 FLOP/s per GPU
seconds = 6.2
useful modeled work = 6NT = 1.6944e18 FLOPs
peak work in interval = GPt = 3.1394816e18 FLOPs
MFU = 1.6944e18 / 3.1394816e18 = 0.53971 = 53.97%
sanity: tokens/s = 4e6 / 6.2; multiplying this by 6N
and dividing by GP gives the same ratio.
The peak is the rounded H100 SXM dense BF16 value. Using another precision or a structured-sparse figure changes the denominator. Check that the actual operation mix makes that peak a useful reference.
Training FLOPs: 6ND explains the numerator. Here it excludes attention and recomputation. Adding attention changes reported MFU even if the token rate does not move. At long context, compare a more complete useful-work count, with causal versus full-square attention assumptions stated. A high number may reveal a counting error, but it is not evidence of one merely because it exceeds a familiar published band.
| Check before comparison | Why it matters |
|---|---|
| Same model-work convention | Attention may be included in only one report |
| Same token-count scope | TP ranks can duplicate a global batch |
| Same elapsed interval | Daily time includes gaps a step timer may omit |
| Same peak convention | Precision and sparsity change the denominator |
For a larger window, suppose the same useful work takes eight seconds once separately excluded input delays are included. The ratio becomes 41.83%. Both values can be correctly calculated; 53.97% describes the 6.2-second interval and 41.83% describes the eight-second interval. Label them so a schedule uses the right one.
HFU counts modeled work that the hardware executes, including recomputation. Suppose the chosen projection-work model counts useful forward and backward work as 6N FLOPs per token. Repeating that projection forward once adds 2N, making executed work 8N. For this same scope and elapsed interval, HFU would therefore be 8/6, or 4/3, of MFU. That factor changes with selective recomputation or a different operation mix. PaLM's appendix distinguishes useful work from the extra work of rebuilding activations.
A “good” result is evidence that the chosen configuration makes useful progress efficiently for this workload. I would compare a controlled baseline before and after one change, keeping model and useful data fixed. If throughput falls, profile before assigning a cause. A new architecture or longer context may need another reference rather than a lower universal pass mark.
NVIDIA's DCGM definitions describe tensor activity as pipeline activity over sampled time. DCGM_FI_PROF_PIPE_TENSOR_ACTIVE is helpful context but does not equal useful FLOPs divided by peak. Inspect critical-path host gaps, collective arrival skew, pipeline idle intervals and memory-heavy kernels. These contributions can overlap; they cannot be read as independent slices of 100% minus MFU.
What interviewers probe next
- “Can recomputation improve MFU?” Yes, if saved memory enables enough useful throughput to reduce time per update. It does not improve MFU merely by executing extra FLOPs.
- “What if I count tokens on every TP rank?” You can multiply the numerator by the tensor-parallel degree even though those ranks processed the same global batch. Deduplicate that count.
- “How do I compare a MoE run?” Use its participating computation, including shared layers and routing assumptions, and compare a matching setup. Total resident parameters are not a token's operation count.
- “What is the next optimization?” Choose it from measured critical-path evidence. A single utilization percentage cannot establish whether fusion, fewer stages or better overlap will help.
Common mistakes
- Confusing sampled activity with useful model throughput.
- Mixing token, FLOP or time definitions between reports.
- Multiplying every MFU by 4/3 without checking recomputation scope.
- Treating 35% or 50% as a universal system-quality threshold.
Key takeaways
- The stated hypothetical step gives 53.97% MFU under the 6N convention.
- Widening its interval from 6.2 to eight seconds gives 41.83%.
- More recomputed arithmetic alone does not increase useful progress.
- Compare raw throughput and aligned definitions before judging the percentage.