← Napkin Math, Cost & Capacity / 28

hardBasetenAnyscale

Model our serving request queue with Little's law. What happens as we approach saturation?

Use Little’s law to distinguish waiting and active requests, then test saturation with a stated service model. Avoid turning a mean occupancy identity into a universal latency rule.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Use Little’s law to distinguish waiting and active requests, then test saturation with a stated service model. Avoid turning a mean occupancy identity into a universal latency rule.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🧮 Napkin Math & Capacity

KV Cache SizingDerive cache payload from the stored attention state, then account for mixed sequence lengths, block allocation and per-device limits. Worked examples distinguish exact bytes from deployable capacity.

🧮 Napkin Math & Capacity🔒 Premium

Capacity Planning and UtilizationCapacity planning connects demand, useful throughput and acceptable waiting time. Serving and training need different models of bursts, job shape and spare capacity.

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency budget follows a request through networking, queueing and model execution. Use measurements to find which stage causes slow responses under load.

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputEstimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackRank the H100, MI300X and Trainium2 by cost per token for decode Next in this trackA vendor claims 10,000 tokens per second per GPU. Sanity-check it.Closest to this topicWhy does object storage behave badly for random reads, and what does a training pipeline do about it?

FEDITOR'S NOTE

Require consistent populations and means; distinguish an M/M/1 example from continuous batching and test whether a backlog actually drains.

DISCUSSION · 0

No comments yet — be the first to share your approach.