napkin math

Compare raw weight sizes at 16, 8 and 4 bits, then budget KV cache and runtime memory for Llama 3.1 70B. Follow the worked per-GPU example.

Calculate Llama 3.1 70B KV cache memory at 128k context using eight KV heads. Worked BF16 and FP8 payloads, GB versus GiB, and per-GPU capacity checks.

The 6ND rule, where the 6 comes from, and the 6.3e24 that every training-time and GPU-count question in the loop is built on.

Work from training FLOPs to fleet throughput, days and GPU-hours. Then see how utilization and downtime change the estimate.

Solve for useful throughput, round to a supported parallel layout, and account for calendar time once. The right GPU count need not be a power of two.

Check model placement first, then derive a TP8 traffic floor and add context. Separate a single stream’s pace from an ideal bandwidth ratio.

Compute cost from a matched bill and token count. Show when an activity factor belongs in the denominator and when it would double-count idle time.

Separate frozen weights from trainable state, count rank-16 adapters from matrix shapes, and check peak memory before choosing a fine-tuning layout.

Turn FLOPs and HBM bytes into resource bounds, then compare them with elapsed time. A tighter memory bound suggests an investigation rather than proving the measured bottleneck.

Count gradient traffic with the actual dtype, convert one-direction rates, and expose the bucket-readiness constraint that an average bandwidth division misses.

Work through projection and attention FLOPs for an 8,192-token prompt. Separate a peak-based lower bound, an assumed-rate estimate and observed TTFT.

Derive GPU-hours from an explicit FLOP convention, compare Meta’s 30.84 million reported hours, and avoid mistaking the residual for a measured MFU.

Translate active streams into replicas using growth reservations and matched latency tests. Show peak and failure headroom separately, with no invented benchmark.

Separate resident parameters, active work and the union of experts touched by a batch. Compare toy dense and MoE budgets without promising a fixed speedup.

Inventory a resumable checkpoint instead of rounding its bytes per parameter. Trace snapshot stall, durable drain and publication, then calculate the storage floor.

Build a power budget from complete servers to the facility meter, then check rack limits and distinguish a design maximum from an annual electricity bill.

Compare ownership with the actual rental contract, using explicit cash flows, resale assumptions and the hours that can really be avoided.

Calculate FP8 savings by separating weights, KV cache and other step costs. Compare the same batch first, then check whether larger batches meet latency and quality targets.

Derive ring all-reduce traffic, match directional bytes to bandwidth, and check memory feasibility before estimating the communication exposed on a training step.

Count weight and cache traffic on a specified B200 configuration, check finite capacity, and separate aggregate throughput from per-user pace. Worked bounds guide a benchmark rather than replace it.

Convert useful tokens into input traffic, then calculate separate checkpoint and restore budgets. Learn why an asynchronous writer can keep up with the cadence and still miss the recovery deadline.

Size for a tested peak and a stated failure, then price an explicit daily scaling schedule. Keep request-slot occupancy separate from GPU utilization.

Derive training cost from work per token, achieved model FLOPs utilization and a declared hourly price. Check the total through GPU-hours and keep project costs separate.

Estimate a TP8 decode step from weights, cache reads and explicitly assumed collective times, then replace the serial worksheet with a measured critical path.

Derive the batch-295 shortcut, count the tensors it omits, and test whether the resulting kernel estimate is even inside the replica’s memory budget.

Calculate the fleet, duration and rental budget for a 7B training run. Check the deadline, actual sharding scheme and paid reservation period before placing the order.

Estimate activation memory from saved tensor shapes and lifetimes. Use a published reference formula carefully, then check checkpointing and parallel ownership against the real peak.

Use Little’s law to distinguish waiting and active requests, then test saturation with a stated service model. Avoid turning a mean occupancy identity into a universal latency rule.

Use bandwidth per dollar as a first comparison, then check feasible memory, supported execution and complete service cost before ranking accelerators.

Define what a throughput claim counts, then compare its required arithmetic and memory traffic with a feasible deployment and matched latency targets.