hardFireworks

Design the fastest serving path for a 70B model with a 200 ms TTFT target at production scale.

Build a measured first-token budget and test the smallest serving design that meets it, including cold prompts, queueing and delivery.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Build a measured first-token budget and test the smallest serving design that meets it, including cold prompts, queueing and delivery.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational

🧮 Open Weights & Serving Engines

vLLM Server Arguments That MatterSeparate executor memory, allocated cache pools and logical occupancy. Map context, sequence and token-batch limits to the constraints they control, then tune a pinned deployment against per-rank memory and latency.

Advanced

🚀 Inference & Serving🔒 Premium

Chunked PrefillFollow a long prompt through a shared token budget, including its final partial chunk. Learn why chunking can reduce stream pauses without guaranteeing a fixed token gap or a universal throughput gain.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodeSeparate prefill and decode to control phase interference, then budget the KV handoff. A short isolated transfer can still overload a shared path; cancellation stops generation before it necessarily releases transfer buffers.

Core

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency budget follows a request through networking, queueing and model execution. Use measurements to find which stage causes slow responses under load.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this tracknvidia-smi shows 30 percent utilization on our serving fleet. Is that a problem, and what would you look at instead?Next in this trackWhat changes when you serve a vision-language model instead of a text-only one?Next in this trackWhen does offloading the KV cache to CPU memory or NVMe beat recomputing it?

FEDITOR'S NOTE

Define which requests must meet 200 ms and where the clock starts. Use a complete request trace to justify each optimization and test overload behavior before adding a separate pool.

DISCUSSION · 0

No comments yet — be the first to share your approach.