GPUs and time-to-train calculator
Estimate dense-model training FLOPs, time, GPU rental cost and a memory lower bound. Change the assumed MFU, fleet size and deadline; check the formula and its limits.
Run
Fleet
Scaling assumes the same MFU at every fleet size. Replace the starting GPU-hour rate with your quote; it is not a live price.
This assumed Adam layout uses 16 bytes per parameter: bf16 weights and gradients, fp32 master weights and two fp32 moments. For 70B that is 1.1 TB, which fully sharded (ZeRO-3 / FSDP) needs at least 18 × H100 80GB SXM with an assumed 80% memory allowance. Optimizer and gradient precision can change this layout. Activations, temporary parameter gathers and communication buffers still need to fit; this aggregate check does not establish a feasible parallelism plan.
- FLOPs
- 6 × 70e9 × 15e12 = 6.30e+24 FLOPs
- Effective rate
- 4,096 × 989 TFLOPS × 40% = 1620.38P FLOP/s
- Time
- FLOPs ÷ effective rate = 45.0 days
- GPUs for deadline
- ceil[FLOPs ÷ (989e12 FLOP/s × 40% × 30 days × 86,400 s/day)] = 6,144
- Static memory
- 70e9 × 16 B = 1.1 TB
This dense-model estimate omits the context-dependent attention term. The chosen MFU must use the same FLOPs convention. Communication and recomputation already reduce measured MFU; do not deduct them twice. Add downtime outside the measurement interval, storage, networking and other charges separately when planning a schedule or budget.
Use the result to make a decision
Start with a dense-model compute estimate, then test how time and cost change with utilization. The default MFU is illustrative. A measured MFU already reflects communication and recomputation in its timing interval; add only overhead outside that interval. Check the memory layout and parallelism before treating a GPU count as deployable.
Questions people ask
For dense parameter matrix multiplications, the approximation counts two FLOPs per parameter per token in forward and four in backward. The PaLM paper derives this convention and a separate attention term. This calculator omits that context-dependent term. Sparse models, long-context workloads and other architectures need a workload-specific FLOPs estimate.
Use a measurement for the same model, precision, sequence length, batch, parallelism and fleet size. Its model-FLOPs convention must match this calculator’s 6ND approximation. The 40% default is a teaching input, not an industry benchmark. At fixed hardware, reducing MFU from 40% to 20% doubles estimated time and GPU rental cost.
The assumed layout is 2 bytes for weights, 2 for gradients, 4 for master weights and 8 for Adam’s two moments: 16 bytes per parameter. Other implementations and precisions differ. The calculator assumes full sharding and an 80% memory allowance. Activations, parameter gathers and communication buffers still require a per-device fit check.