The napkin math every AI infra loop expects you to know cold

weights_bytes = params × bytes_per_param

Variables

bf16/fp16 = 2 B, fp8/int8 = 1 B, int4 = 0.5 B, fp32 = 4 B.

Worked

Llama 3.1 70B in bf16: 70.6e9 × 2 = 141 GB. It does not fit on one 80 GB card before a single token of cache. In int4: 35 GB.

Trap

Saying "16 GB" for a 7B model in fp16. The 16 is bits, not gigabytes; the answer is 14 GB.

kv_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes

Variables

The 2 is K and V. kv_heads is the number of key-value heads (8 for Llama 3 GQA, not the 64 query heads). For MLA: layers × (latent + rope) × bytes, no factor of 2.

Worked

Llama 3.1 70B, bf16: 2 × 80 × 8 × 128 × 2 = 320 KB per token. At 128k context that is 42 GB for one sequence. DeepSeek-V3 with MLA: 61 × 576 × 2 = 70 KB per token.

Trap

Using the query-head count. With 64 heads the 70B figure becomes 2.6 MB per token, eight times too high, and every downstream capacity number is wrong.

kv_total = kv_bytes_per_token × context × concurrent_sequences

Variables

context is the tokens each sequence holds (prompt plus generated so far). Sequences are what the engine has in flight, not requests per second.

Worked

70B, 8k context, 32 concurrent: 327,680 B × 8,192 × 32 = 85.9 GB (80 GiB). On 2 × H100 after 141 GB of weights there is nowhere to put it; on 4 × H100 (320 GB) it fits with room.

Trap

Forgetting that the cache is per sequence. "It fits for one user" and "it fits for the launch" are different questions by a factor of the batch.

train_flops = 6 × N × D

Variables

N parameters, D training tokens. 2 FLOPs per parameter per token forward, 4 backward.

Worked

70B on 15T tokens: 6 × 70e9 × 15e12 = 6.3e24 FLOPs.

Trap

Quoting 2ND. That is inference; the backward pass is the other two thirds of training.

seconds = train_flops ÷ (gpus × peak_flops × MFU)

Variables

peak is the dense bf16 figure for the part; MFU is the fraction of it the run actually sustains.

Worked

6.3e24 FLOPs on 16,384 H100 (989 TFLOPS) at 40% MFU: 6.3e24 ÷ (16,384 × 989e12 × 0.4) = 9.7e5 s, about 11 days.

Trap

Using the sparse peak from the marketing sheet (1,979 TFLOPS for H100 bf16). It halves the answer and no dense training run reaches it.

MFU = (6 × N × tokens_per_step ÷ step_seconds) ÷ (gpus × peak_flops)

Variables

Observed model throughput over theoretical. HFU (hardware FLOPs utilization) also counts recomputation from activation checkpointing, so it is always higher.

Worked

70B, a 4M-token step in 2.07 s on 2,048 H100s: needed 1.68e18 FLOPs, available 4.2e18, MFU ≈ 40%.

Trap

Reporting HFU as MFU. Checkpointing makes the GPUs look busier than the model is.

static_bytes = params × 16   (bf16 weights 2 + grads 2 + fp32 master 4 + Adam m 4 + Adam v 4)

Variables

Activations are extra and scale with batch × sequence × hidden × layers, reduced by checkpointing.

Worked

70B: 1.12 TB of static state. Fully sharded across 80 GB cards at 80% usable, that is 18 GPUs before a single activation.

Trap

Assuming 2 bytes per parameter because that is what inference costs. Training is 8x that before activations.

intensity = FLOPs ÷ bytes_moved     ridge = peak_flops ÷ memory_bandwidth

Variables

Below the ridge a kernel is memory-bound and attains intensity × bandwidth; above it, the peak.

Worked

H100: 989e12 ÷ 3.35e12 ≈ 295 FLOP/byte. Decode at batch 1 (bf16) has intensity ≈ 1 and attains about 3.4 TFLOPS, under 0.4% of peak.

Trap

Treating a low-utilization kernel as "badly written" when it is memory-bound by construction. The fix is fewer bytes (fusion, quantization) or more FLOPs per byte (batching), not more threads.

tokens_per_second ≈ batch × bandwidth ÷ (active_weight_bytes + batch × kv_bytes_per_sequence)

Variables

Every decode step reads all active weights once plus each sequence's KV cache, and emits one token per sequence.

Worked

70B bf16 on one H100, batch 1, no cache: 3.35e12 ÷ 1.41e11 ≈ 24 tokens/s. That is the physics of single-stream decode on a 70B model.

Trap

Expecting tensor cores to help. At batch 1 they are idle 99% of the time; the only lever is bytes per token.

usd_per_M = (fleet_usd_per_hour ÷ 3600) ÷ (tokens_per_second × utilization) × 1e6

Variables

fleet price is per replica; tokens per second is per replica; utilization is the fraction of the hour the replica is actually generating.

Worked

8 × H100 at $2.50/h serving 2,000 tokens/s at 100% utilization: $20 ÷ 3600 ÷ 2000 × 1e6 ≈ $2.78 per million output tokens. At 40% utilization it is $6.94.

Trap

Quoting a number without utilization. Bursty traffic doubles or triples the real cost, and the interviewer knows it.

bytes_per_rank = 2 × (n − 1) ÷ n × buffer_bytes     time ≈ bytes_per_rank ÷ link_bandwidth

Variables

A reduce-scatter then an all-gather, each moving (n−1)/n of the buffer per rank. Approaches 2 × buffer as n grows, independent of n.

Worked

Gradients of a 70B model in bf16 (141 GB) across 8 ranks: 2 × 7/8 × 141 GB = 247 GB sent per rank, and 247 GB received. NVLink's 900 GB/s is bidirectional, so either divide 247 GB by the 450 GB/s one direction gets, or divide the full 494 GB by 900: both give about 0.55 s per step, which is why you overlap it with the backward pass.

Trap

Two traps. Believing the all-reduce gets cheaper per rank with more ranks: the per-rank volume is flat, while the latency term and the slowest link grow. And dividing one-way bytes by a bidirectional link rating, which halves the answer.

flops_per_token ≈ 2 × active_params     prefill_flops ≈ 2 × active_params × prompt_tokens

Variables

active_params is what a forward pass touches: total for dense models, the routed subset for MoE (37B of DeepSeek-V3's 671B).

Worked

Prefilling an 8k prompt through a 70B dense model: 2 × 70e9 × 8,192 ≈ 1.15e15 FLOPs, about 1.2 s at 100% of one H100 and 2 to 3 s in practice. That is your floor for time to first token before batching.

Trap

Using total parameters for an MoE model. DeepSeek-V3 decode moves like a 37B model for compute, and like a 671B model for memory.