Baseten Napkin Math, Cost & Capacity interview questions

Calculate Llama 3.1 70B KV cache memory at 128k context using eight KV heads. Worked BF16 and FP8 payloads, GB versus GiB, and per-GPU capacity checks.

Check model placement first, then derive a TP8 traffic floor and add context. Separate a single stream’s pace from an ideal bandwidth ratio.

Compute cost from a matched bill and token count. Show when an activity factor belongs in the denominator and when it would double-count idle time.

Translate active streams into replicas using growth reservations and matched latency tests. Show peak and failure headroom separately, with no invented benchmark.

Derive the batch-295 shortcut, count the tensors it omits, and test whether the resulting kernel estimate is even inside the replica’s memory budget.

Use Little’s law to distinguish waiting and active requests, then test saturation with a stated service model. Avoid turning a mean occupancy identity into a universal latency rule.

Compare raw weight sizes at 16, 8 and 4 bits, then budget KV cache and runtime memory for Llama 3.1 70B. Follow the worked per-GPU example.

Work from training FLOPs to fleet throughput, days and GPU-hours. Then see how utilization and downtime change the estimate.

Solve for useful throughput, round to a supported parallel layout, and account for calendar time once. The right GPU count need not be a power of two.

Separate frozen weights from trainable state, count rank-16 adapters from matrix shapes, and check peak memory before choosing a fine-tuning layout.

The 6ND rule, where the 6 comes from, and the 6.3e24 that every training-time and GPU-count question in the loop is built on.

Count gradient traffic with the actual dtype, convert one-direction rates, and expose the bucket-readiness constraint that an average bandwidth division misses.

Turn FLOPs and HBM bytes into resource bounds, then compare them with elapsed time. A tighter memory bound suggests an investigation rather than proving the measured bottleneck.

Separate resident parameters, active work and the union of experts touched by a batch. Compare toy dense and MoE budgets without promising a fixed speedup.

Calculate the fleet, duration and rental budget for a 7B training run. Check the deadline, actual sharding scheme and paid reservation period before placing the order.

Use bandwidth per dollar as a first comparison, then check feasible memory, supported execution and complete service cost before ranking accelerators.

Work through projection and attention FLOPs for an 8,192-token prompt. Separate a peak-based lower bound, an assumed-rate estimate and observed TTFT.

Derive GPU-hours from an explicit FLOP convention, compare Meta’s 30.84 million reported hours, and avoid mistaking the residual for a measured MFU.

Inventory a resumable checkpoint instead of rounding its bytes per parameter. Trace snapshot stall, durable drain and publication, then calculate the storage floor.

Build a power budget from complete servers to the facility meter, then check rack limits and distinguish a design maximum from an annual electricity bill.

Compare ownership with the actual rental contract, using explicit cash flows, resale assumptions and the hours that can really be avoided.