Your p99 time to first token is four times p50. Find out why.

A fourfold ratio is a symptom, not a diagnosis. Trace individual requests, separate waiting from execution and delivery, and test workload, scheduling, cache and capacity hypotheses against the service objective.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🚀 Inference & Serving🔒 Premium

Chunked PrefillFollow a long prompt through a shared token budget, including its final partial chunk. Learn why chunking can reduce stream pauses without guaranteeing a fixed token gap or a universal throughput gain.

Core

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency budget follows a request through networking, queueing and model execution. Use measurements to find which stage causes slow responses under load.

Foundational

🧮 Open Weights & Serving Engines

vLLM Server Arguments That MatterSeparate executor memory, allocated cache pools and logical occupancy. Map context, sequence and token-batch limits to the constraints they control, then tune a pinned deployment against per-rank memory and latency.

Foundational

🧮 Open Weights & Serving Engines

Serving Benchmarks That Do Not LieDesign reproducible serving experiments with realistic arrivals, cache state and failure accounting. Interpret latency-qualified throughput and conditional per-rank bounds without turning optimistic kernel estimates into production capacity.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYou have one week to get a newly released trillion-parameter model serving production traffic. Plan it.Next in this trackSelf-host a 753B open-weights model or call a hosted API? Work the crossover.Next in this trackOne replica in a serving fleet returns tokens slower than its siblings. Find out why.

FEDITOR'S NOTE

Scored on timestamp boundaries, request-level decomposition, valid percentile aggregation and measured remedies that include admission, scheduling, workload and non-GPU delays.

DISCUSSION · 0

No comments yet — be the first to share your approach.