← Napkin Math, Cost & Capacity / 30

expert★ EssentialFireworksTogether AIGroq

A vendor claims 10,000 tokens per second per GPU. Sanity-check it.

Define what a throughput claim counts, then compare its required arithmetic and memory traffic with a feasible deployment and matched latency targets.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Define what a throughput claim counts, then compare its required arithmetic and memory traffic with a feasible deployment and matched latency targets.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputEstimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

🧮 Napkin Math & CapacitySign in

Arithmetic Intensity by OperationCount the FLOPs and memory transfers of a projection, attention and a residual add. Learn where the batch-size shortcut works, where it breaks, and which hardware roof applies.

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodeSeparate prefill and decode to control phase interference, then budget the KV handoff. A short isolated transfer can still overload a shared path; cancellation stops generation before it necessarily releases transfer buffers.

🚀 Inference & Serving

Prefill vs DecodeFollow one prompt through its first prediction and later decode steps, then separate GPU work from the wait a user sees. Learn why batch size and context change the likely bottleneck.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Closest to this topicHow much does moving from bf16 to fp8 save in serving cost?Closest to this topicHow long does prefill take for an 8k-token prompt on a 70B model?Closest to this topicHow many tokens per second can a 70B model generate for a single user on H100s?

FEDITOR'S NOTE

Use bounds to test a stated configuration. A result below a bound is only possible in the model, not a benchmark or a service guarantee.

DISCUSSION · 0

No comments yet — be the first to share your approach.