Core

MFU and HFU

Model FLOPs utilization is the fraction of a GPU's peak that goes into the model's own forward and backward math, computed from 6ND and the step time; hardware FLOPs utilization also counts recomputation. Production LLM training lands at 35 to 45% MFU, and knowing where the other 55% goes is the job.

a free account unlocks the core curriculum tier · no card

RELATED CONCEPTS

Training FLOPs: 6ND Activation Checkpointing Pipeline Parallelism and the Bubble GPU-Hours and Time to Train Profiling with Nsight Tensor Cores and Matrix Units

LESSONS THAT TEACH THIS

Training at ScaleA training step is four costs and only one of them is the model13 min→Training at ScaleMFU is the only throughput number that cannot be gamed15 min→Training at ScaleCapstone: plan a large run end to end and defend the schedule300 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismWhat is MFU, how do you compute it from a running job, and what counts as a good number?→Distributed Training & ParallelismYour training run is at 60% of the step time you projected. How do you find out whether it is compute, memory, network or I/O?→Napkin Math, Cost & CapacityEstimate how many H100-hours it took to train Llama 3.1 405B, then check it against the paper→Napkin Math, Cost & CapacityWhat does it cost per million output tokens to serve a 70B model on eight H100s?→CUDA, Triton & Kernel EngineeringA training step runs at 20 percent model FLOPs utilization. Profile it and find where the missing time goes.→Napkin Math, Cost & CapacityHow many FLOPs does it take to train a 70B model on 15 trillion tokens?→