← LLM Inference & Serving / 32

hard

Would request hedging improve an LLM endpoint whose p99 TTFT is too high?

Price duplicate work before racing replicas. Separate independent stragglers from shared overload, and define what happens when a winner starts streaming.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Price duplicate work before racing replicas. Separate independent stragglers from shared overload, and define what happens when a winner starts streaming.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 318 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency budget follows a request through networking, queueing and model execution. Use measurements to find which stage causes slow responses under load.

💻 Coding for Infra🔒 Premium

Asynchronous GPU Buffer OwnershipA host function can return while a GPU still uses its pointers. Tie buffer reuse to device completion, including cancellation, cross-stream consumers and foreign-function boundaries.

🚀 Inference & Serving

Latency Metrics: TTFT, TPOT and GoodputRead a stream from its timestamps: distinguish first-token wait, individual gaps and average generation pace. Compute goodput with explicit deadlines, denominators and failure accounting.

📐 AI Systems Design🔒 Premium

Offline Batch Inference vs Online ServingOnline serving promises each request a latency; a batch job promises a deadline for a whole set and a checked result for every item. That changes what to optimize, when an engine's startup cost dominates, and what delivery means.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign moderation in a streaming inference path without checking text after users have already seen it Next in this trackDesign real-time spelling or code suggestions while the user keeps typing Next in this trackFirst-token latency is fast, but the reasoning model's answer takes seconds. What do you check?

DISCUSSION · 0

No comments yet — be the first to share your approach.