Core

MFU and HFU

Measure useful model work, separate recomputation, and align token counts, timing and hardware peaks. Learn what activity counters can and cannot tell you.

This concept is included with a free account. Sign in to continue reading.

free account · no card needed

RELATED CONCEPTS

Training FLOPs: 6ND Activation Checkpointing Pipeline Parallelism and the Bubble GPU-Hours and Time to Train Profiling with Nsight Tensor Cores and Matrix Units

LESSONS THAT TEACH THIS

Training at ScaleA training step follows a critical path through compute, exchange and update18 min→Training at ScaleMFU connects token throughput to an explicit compute model20 min→Training at ScaleCapstone: plan a large run end to end and defend the schedule300 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismWhat is MFU, how do you compute it from a running job, and what counts as a good number?→Distributed Training & ParallelismYour training run delivers only 60% of the throughput you projected. How do you find out whether it is compute, memory, network or I/O?→Napkin Math, Cost & CapacityEstimate how many H100-hours it took to train Llama 3.1 405B, then check it against the paper→Napkin Math, Cost & CapacityWhat does it cost per million output tokens to serve a 70B model on eight H100s?→CUDA, Triton & Kernel EngineeringA training step runs at 20 percent model FLOPs utilization. Profile it and find where the missing time goes.→Napkin Math, Cost & CapacityHow many FLOPs does it take to train a 70B model on 15 trillion tokens?→