napkin math

The first number every serving interview asks for, derived from parameter count and bytes per parameter, in three precisions, with the part people forget to add.

Four numbers from the config file, one formula, and a per-sequence result that is a third of the model's own weights. Plus the mistake that makes the answer eight times too big.

The 6ND rule, where the 6 comes from, and the 6.3e24 that every training-time and GPU-count question in the loop is built on.

The fleet equation applied to 6.35e24 FLOPs: 11 days, and how the answer swings from 9 to 15 with the one parameter the interviewer wants you to state.

The fleet equation solved for GPU count: about 6,200 H100s at 40% MFU, why it rounds up to a power of two, and the reasons a good answer adds 15% before naming a number.

Why decode speed is a division of bandwidth by weight bytes, why the answer is about 24 tokens a second regardless of how fast the tensor cores are, and the two ways to double it.

Fleet dollars per second over tokens per second, times a million: a worked $2.78, the utilization term that turns it into $6.94, and why batch is the lever that moves it by 50x.

Sixteen bytes per parameter is 1.1 TB of static state before a single activation; LoRA keeps the 141 GB of frozen weights and shrinks the rest to a few gigabytes. The derivation, the GPU counts, and the activations that both still pay.

Two divisions decide it: the kernel's FLOPs per byte against the card's ridge point. The worked case, the profiler counters that give the inputs, and what to change once you know which wall you hit.

Gradient bytes per step over step time, with the ring all-reduce factor that nearly doubles it: the arithmetic for a 7B and a 70B, the link speeds that cover it, and the step-time threshold where overlap stops hiding it.

Prefill is the compute-bound half of serving: 2 × N × tokens FLOPs over the effective TFLOPS of the replica. The chain that gives a 0.24 s floor on a node, the single-card version, and what it means for time to first token.

6ND on 15.6 trillion tokens gives 3.8e25 FLOPs; converting to GPU-hours needs an MFU, and Meta's published 30.8 million hours lets you solve for it. The chain, the check, and what the implied MFU tells you.

Two constraints per replica, memory and throughput, each derived from the model and the traffic; whichever binds sets the users per replica, and the replica count follows with headroom. The full chain from 1,000 users to a number of nodes.

Memory follows total parameters and compute follows active ones, so the two models pull the GPU count in opposite directions. The chain for a 235B-total, 22B-active MoE against a dense 70B, for serving and then for training.

Sixteen bytes per parameter is 6.5 TB per checkpoint; over a 100 GB/s filesystem that is about a minute, which at a 30-minute cadence is 3% of the run. The chain, the storage tiers that meet it, and why the write is made asynchronous.

700 W per GPU is the number everyone knows and it is only 40% of the answer. The chain from GPU TDP to node, to IT load, to facility power with PUE: about 1.8 MW, and what that means in racks and dollars.

Capex plus two years of power and operations, minus what the cards are worth at the end, against a rental rate at a utilization. The chain that produces a break-even utilization, and the two inputs that swing it from 60% to over 100%.

fp8 halves the bytes per decode step and doubles the compute peak, but the all-reduces and launch overhead do not shrink, so the measured gain is 1.5x at a fixed batch and about 2x once the freed memory raises the batch. The chain, with the step-time table.

The ring all-reduce formula, 2(n−1)/n times the buffer over the link bandwidth, applied to 141 GB of bf16 gradients on an NVLink node: 0.27 s by the formula, and the two reasons the measured number is different.

8 TB/s over the bytes per step: 57 tokens per second single-stream in bf16, 113 in fp8, 227 in fp4, and the batch curve on a single card that now holds the whole model. Where the curve bends and what caps it.

The token stream is tens of megabytes a second and the checkpoints are tens of gigabytes a second; the two workloads differ by a thousand and want different tiers. The chain from tokens per second to bytes, the checkpoint burst, and the restore that sets the real spec.

Peak sets the fleet, average sets the bill, and the ratio between them is idle money. The chain from a 3x diurnal peak to a replica count, the utilization it implies, what autoscaling can and cannot recover given model load times, and what to do with the trough.

Dollars per FLOP from the GPU price and the MFU, times 6N: a 70B training token costs about three quarters of a microdollar, and the whole 15T-token run follows in one multiplication. The chain, the comparison to an inference token, and why the training token is cheaper.

Each card reads an eighth of the weights in 5.3 ms, then the step pays 160 latency-bound all-reduces and hundreds of kernel launches that do not shrink with sharding. The chain to a 9 to 12 ms step, the communication floor, and why TP8 gives 4x rather than 8x at batch one.

For bf16 weights the decode intensity equals the batch, so the ridge is at batch 295 on an H100. Then the KV cache enters and the answer changes shape: the projections go compute-bound at 300 while the attention reads never do, and at 8k context the step as a whole stays memory-bound at any batch.

6ND to a compute budget, the fleet equation to a GPU count, the rental rate to a bill: 64 H100s, about 19 days, roughly $75k plus a buffer. The chain, the reasons the answer is 64 and not 59, and the memory check that says the fleet is compute-sized, not memory-sized.

Activations are the term the 16-bytes-per-parameter rule leaves out, and at 8k context they are bigger than the weights: about 34 × sequence × hidden bytes per layer, 2.3 GB, 182 GB per micro-batch across 80 layers. The chain, what FlashAttention removes, and what checkpointing does to the number and the compute.

Concurrency equals arrival rate times residence time, and residence time on an LLM replica grows with concurrency. The chain from 50 requests a second to 1,000 in flight, the replica count with a stated utilization, and the feedback loop that turns 90% utilization into a queue that does not drain.

Decode is bandwidth-bound, so cost per token tracks dollars per terabyte per second, and on spec the MI300X wins by 2x. The table, the memory-capacity term that changes the replica shape, and the software-efficiency discount that decides whether the spec ranking survives a benchmark.

Two bounds, bandwidth and compute, and five questions about what the number counts. For a 70B on an H100 the claim exceeds the bf16 compute peak and is impossible; for an 8B at high batch it is routine; for input tokens it is easy. The chain that tells the cases apart.