AI Infra Interviews logo
← LLM Inference & Serving / 14
hard★ EssentialBasetenAnthropicOpenAI

Your p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.

Trace the slow requests through queue, engine and gateway clocks. A flat median narrows the investigation, but it does not establish a cause.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Trace the slow requests through queue, engine and gateway clocks. A flat median narrows the investigation, but it does not establish a cause.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Core
📐 AI Systems DesignSign in
Designing for Latency SLOsA latency budget follows a request through networking, queueing and model execution. Use measurements to find which stage causes slow responses under load.
Advanced
📐 AI Systems Design🔒 Premium
Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.
Advanced
🩺 Fleet Reliability & Observability🔒 Premium
Tracing Inference RequestsConnect gateway, queue, model execution and streaming delivery without confusing overlapping spans or GPU activity with user-visible latency.
Advanced
📐 AI Systems Design🔒 Premium
Capacity and BackpressureBackpressure limits accepted work when a service cannot keep up. Bounded queues and clear retry rules prevent overload from spreading through the request path.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackvLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?Next in this trackHow would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?Next in this trackDesign an autoscaler for GPU inference replicas that reacts to load without thrashing.
FEDITOR'S NOTE

Look for matched request cohorts and per-request phase clocks before a proposed mitigation. A flat median is evidence about the distribution, not a diagnosis.

DISCUSSION · 0

No comments yet — be the first to share your approach.