TL;DR: With C = 6.354e24 modeled FLOPs, 16,384 H100s, a 989 TFLOP/s dense BF16 peak and an assumed 40% model FLOPs utilization (MFU), duration is 11.346 days, or 4.462 million GPU-hours. At 30% MFU the same calculation gives 15.129 days. A calendar estimate must state which delays the MFU interval already includes before adding any more.
How to approach it
First carry forward the work convention from Training FLOPs: 6ND. Specify the arithmetic peak, fleet size and utilization interval. Compute the useful fleet rate, divide the work, then convert seconds to days and GPU-hours. Finish with a sensitivity case and the conditions that would change the calendar schedule.
A strong answer
A plan says “11 days.” Later, a checkpoint restore takes hours, and the team cannot tell whether those hours were included. The missing input is the scope of the throughput estimate, not another decimal place in the division.
GPU-hours are the number of GPUs multiplied by the hours they run, the unit cloud capacity is usually billed in. GPU-Hours and Time to Train defines the conversion. For this example, use an approximate 70.6B dense model and 15T processed tokens, giving 6.354e24 modeled projection FLOPs. Attention and recomputation are excluded from that useful-work convention. The MFU must use the same convention.
G = 16,384
P = 989e12 FLOP/s per GPU, dense BF16 H100 SXM
u = 0.40, assumed
R = G × P × u = 6.4815104e18 useful FLOP/s
t = C / R = 6.354e24 / 6.4815104e18
≈ 980,327 seconds
≈ 11.346 days
GPU-hours = G × t / 3,600 ≈ 4,461,577
sanity: C / (P × u × 3,600) produces the same GPU-hours.
The hardware peak is a rounded dense BF16 value from the H100 specification. A structured-sparse peak counts a different arithmetic capability. Changing that number without changing the workload would make the estimate look faster without doing any less work.
| Assumed MFU | Modeled duration |
|---|---|
| 30% | 15.129 days |
| 40% | 11.346 days |
| 50% | 9.077 days |
The 40%-to-30% change raises duration by one third, about 3.782 days. It also raises device-hours by about 1.487 million if all GPUs remain allocated. A dollar estimate needs an explicit rate and billing scope; there is no universal “four million dollars” attached to those days.
MFU is computed from useful tokens per second times the declared model work per token, divided by the fleet's chosen peak. In an illustrative one-second interval with 16 million training tokens:
MFU = (16e6 × 6 × 70.6e9) / (16,384 × 989e12)
= 0.41827 ≈ 41.83%
That is a numerical example, not an observed step. A tensor-pipeline-active counter measures activity over its own sampling definition and cannot replace this throughput calculation. PaLM's compute appendix gives the distinction between model and hardware FLOPs utilization, including rematerialization.
If 40% was measured during productive steps only, separately budget excluded evaluation, compilation, recovery and other intervals. If those intervals are already inside the measurement window, charging them again overstates the delay. Likewise, learning-rate warm-up tokens already included in D are not extra training work just because their learning rate differs.
The same run can have two valid utilization numbers with different denominators. The toy ten-hour clock shows exactly where the availability factor belongs.
For a transparent calendar scenario, suppose the fleet is productive for 90% of elapsed time and sustains the assumed 40% MFU during that productive time. Then calendar duration is 11.346/0.90 = 12.607 days. Both percentages are assumptions for this scenario. A fixed overhead allowance cannot establish a promised delivery date without workload and recovery evidence.
I would replace the MFU assumption with a representative throughput measurement as soon as a feasible layout is available. Record model version, sequence mix, batch, precision and parallel configuration. If the model's FLOPs convention changes, recompute both C and MFU; otherwise a revised formula can appear to improve efficiency without changing throughput.
What interviewers probe next
- “What if the fleet has 8,192 GPUs?” At the same assumed MFU, duration doubles to 22.693 days and device-hours stay constant. Whether utilization stays constant requires a scaling test.
- “Does FP8 halve the time?” Only if the applicable peak doubles and useful utilization stays the same, with an acceptable training result. Format conversion and the changed operation mix need measurement.
- “Is 40% a safe default?” It is an explicit scenario value here. A new model, context mix or parallel layout needs evidence and a sensitivity range.
- “What if 40% covers the whole run already?” Use it directly for that interval's work budget; do not divide by a second availability factor for the same downtime.
Common mistakes
- Dropping the 10¹² when converting TFLOP/s.
- Mixing attention-inclusive MFU with an attention-exclusive work budget.
- Treating a tensor activity counter as model FLOPs utilization.
- Adding overhead already included in the measured interval.
Key takeaways
- The stated assumptions give 11.346 days and 4.462M GPU-hours.
- At 30% MFU, the modeled duration rises to 15.129 days.
- A 90%-productive calendar scenario gives 12.607 days, provided downtime was excluded from MFU.
- Fleet scaling and precision changes require a new useful-throughput measurement.