← LLM Inference & Serving / 32

hardNew

Would request hedging improve an LLM endpoint whose p99 TTFT is too high?

Price duplicate work before racing replicas. Separate independent stragglers from shared overload, and define what happens when a winner starts streaming.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Price duplicate work before racing replicas. Separate independent stragglers from shared overload, and define what happens when a winner starts streaming.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 303 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency budget follows a request through networking, queueing and model execution. Use measurements to find which stage causes slow responses under load.

🚀 Inference & Serving

Latency Metrics: TTFT, TPOT and GoodputRead a stream from its timestamps: distinguish first-token wait, individual gaps and average generation pace. Compute goodput with explicit deadlines, denominators and failure accounting.

💻 Coding for Infra🔒 Premium

Producer-Consumer PipelinesProducer-consumer pipelines connect stages that run at different rates. Bounded buffers absorb temporary variation, while the slowest sustained stage limits throughput.

🩺 Fleet Reliability & Observability

Alert Design and On-Call LoadUseful alerts identify a condition that needs a timely response. Connect pages to user impact or imminent risk, and reduce duplicate noise without hiding failures.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign moderation in a streaming inference path without checking text after users have already seen it Next in this trackDesign real-time spelling or code suggestions while the user keeps typing Closest to this topicWhat is the difference between static and continuous batching, and why did it change LLM serving?

DISCUSSION · 0

No comments yet — be the first to share your approach.