memory

Decode is bandwidth-bound, so the FP8 FLOPS number that dominates the marketing does not move it. Where the B300 pays is capacity, and capacity converts into throughput through batch size rather than directly. The arithmetic that decides it, and the case where the B200 wins.

Sixteen bytes per parameter is the bill for mixed-precision Adam. ZeRO pays it in three installments: optimizer state, then gradients, then the weights themselves. The per-rank bytes for a 70B at 8, 16, 32 and 64 GPUs, and what each stage adds to the wire.

Sixteen bytes per parameter says 1.6 TB of state against a 640 GB node, so the model spans nodes before the first token. The arithmetic that sizes the fleet, the layout that puts each axis on the right link, and the two numbers that decide FSDP against pipeline across nodes.

Mathematically identical for a mean loss and a stateless model, and different in three ways that matter to infrastructure: activation memory, communication frequency, and any layer that looks at the batch. With the byte counts and the DDP call that makes it work.

Full recomputation costs a third more compute to cut activation memory by an order of magnitude. The per-layer arithmetic that says whether you need it at all, the selective variant that costs almost nothing, and the point at which a smaller micro-batch is the cheaper way out.

Every token a model has seen leaves a key and a value in every layer. Multiply that out for a 70B model and you will see why memory, not compute, caps how many users a GPU can hold.

Before vLLM, engines reserved the maximum context for every request up front and wasted most of it. The fix borrowed virtual memory from the operating system, and the numbers show how much capacity it returned.

Same model size, three attention layouts, a 30x spread in cache per token. The formulas are short, and they explain why DeepSeek can hold four times the context of a dense 70B on the same cards.

Three hundred fine-tunes as three hundred deployments would need three hundred GPUs. Batched adapters put them on one base, and the arithmetic says what each extra adapter costs in memory and in step time.

Failing at startup and failing under load are different problems with different fixes, and the engine tells you which in its own log. Where the memory actually goes, the four flags that move it, and the arithmetic that says whether it can fit at all.

Halving the weights halves the GPU count and moves the quality evaluation onto you. What the format actually buys in memory and in speed, why those are different questions, and the evaluation that has to run before it ships.

A full cache pool with idle streaming multiprocessors is a specific signature with a specific set of remedies. Six of them, ranked by what they cost, and the one that is free because the flag was set from the model rather than the product.

Four things break in order and only one of them is the attention math everyone expects. What a single million-token sequence costs in memory, why the first token takes minutes, and what the model's own architecture already did about the parts that would otherwise be impossible.

Three published numbers turn tuning from guesswork into a loop with a stopping rule. Which flag moves each, the order to adjust them in, and the two out-of-memory failures that have different fixes despite looking the same.

Fifty full copies is fifty deployments and is unaffordable. Fifty low-rank adapters over one shared base is one deployment, because an adapter is a tiny fraction of the weights and adapters from different tenants can share a batch. The arithmetic that makes it work and the two limits that decide how far it goes.

The first number every serving interview asks for, derived from parameter count and bytes per parameter, in three precisions, with the part people forget to add.

Four numbers from the config file, one formula, and a per-sequence result that is a third of the model's own weights. Plus the mistake that makes the answer eight times too big.

Sixteen bytes per parameter is 1.1 TB of static state before a single activation; LoRA keeps the 141 GB of frozen weights and shrinks the rest to a few gigabytes. The derivation, the GPU counts, and the activations that both still pay.

Memory follows total parameters and compute follows active ones, so the two models pull the GPU count in opposite directions. The chain for a 235B-total, 22B-active MoE against a dense 70B, for serving and then for training.

Activations are the term the 16-bytes-per-parameter rule leaves out, and at 8k context they are bigger than the weights: about 34 × sequence × hidden bytes per layer, 2.3 GB, 182 GB per micro-batch across 80 layers. The chain, what FlashAttention removes, and what checkpointing does to the number and the compute.

A replica per variant is 100 copies of a 140 GB base and 800 GPUs mostly idle; multi-LoRA holds the base once and hundreds of adapters in a few gigabytes. The memory arithmetic, the throughput cost of mixed batches, the routing that keeps hot adapters resident, and the two cases where a variant earns its own replica.

A video request is not a stream of tokens: it is 40 denoising passes over a latent the size of a small film, and the user waits a minute for the whole thing. The memory and FLOP arithmetic per request, why batching works differently, the queue and progress design for minute-long jobs, and where the cost goes.