hardFireworksAnthropic

Design an LLM service for a 200 ms time-to-first-token SLO at p99. Decompose the budget and say what you would give up.

Work a 200 ms request budget, distinguish stage allowances from percentiles, and price the capacity needed to meet the measured latency contract.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Work a 200 ms request budget, distinguish stage allowances from percentiles, and price the capacity needed to meet the measured latency contract.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Core

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency budget follows a request through networking, queueing and model execution. Use measurements to find which stage causes slow responses under load.

Advanced

📐 AI Systems Design🔒 Premium

Capacity and BackpressureBackpressure limits accepted work when a service cannot keep up. Bounded queues and clear retry rules prevent overload from spreading through the request path.

Advanced

🚀 Inference & Serving🔒 Premium

Chunked PrefillFollow a long prompt through a shared token budget, including its final partial chunk. Learn why chunking can reduce stream pauses without guaranteeing a fixed token gap or a universal throughput gain.

Foundational

🚀 Inference & Serving

Prefill vs DecodeFollow one prompt through its first prediction and later decode steps, then separate GPU work from the wait a user sees. Learn why batch size and context change the likely bottleneck.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the runtime for long-running agents: sessions, tool sandboxes, KV that lives for hours, and checkpoints.Next in this trackDesign serving for a video generation model: diffusion steps, batching, memory, and a latency profile unlike an LLM's.Next in this trackDesign speculative decoding into a production serving fleet: draft placement, acceptance monitoring, and the batch regime where it pays.

FEDITOR'S NOTE

Assess a complete client-visible latency budget, workload-specific capacity, honest percentile reasoning and the treatment of rejected work.

DISCUSSION · 0

No comments yet — be the first to share your approach.