AI Infra Interviews logo
← Napkin Math, Cost & Capacity / 12
mediumMeta

Estimate how many H100-hours it took to train Llama 3.1 405B, then check it against the paper

Derive GPU-hours from an explicit FLOP convention, compare Meta’s 30.84 million reported hours, and avoid mistaking the residual for a measured MFU.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Derive GPU-hours from an explicit FLOP convention, compare Meta’s 30.84 million reported hours, and avoid mistaking the residual for a measured MFU.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Core
🧮 Napkin Math & CapacitySign in
GPU-Hours and Time to TrainConvert a stated work budget into elapsed time and GPU-hours. Vary utilization, invert for a deadline, and separate productive time from calendar delays without counting them twice.
Core
🕸️ Distributed TrainingSign in
MFU and HFUMeasure useful model work, separate recomputation, and align token counts, timing and hardware peaks. Learn what activity counters can and cannot tell you.
Foundational
🧮 Napkin Math & Capacity
Training FLOPs: 6NDDerive the six from one forward and two backward matrix products. Count training tokens, attention and recomputation consistently before turning model work into a schedule.
Core
🚀 Inference & ServingSign in
Attention Variants: MHA, GQA, MQA and MLACompare MHA, GQA, MQA and compressed MLA by the state each retains. Derive cache payload in consistent units, distinguish storage capacity from serving throughput, and account for hybrid layers and rank replication.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackSize an inference fleet for a 70B model serving 1,000 concurrent usersNext in this trackA MoE model and a dense model reach the same quality. How do their GPU counts compare for serving and for training?Next in this trackEstimate how long it takes to write a checkpoint for a 405B training run
FEDITOR'S NOTE

Trace the example, state its assumptions, and distinguish the estimate from measured behavior.

DISCUSSION · 0

No comments yet — be the first to share your approach.