← LLM Inference & Serving / 34

hardNew

Design real-time spelling or code suggestions while the user keeps typing

Trace document revisions, cancellation and deadlines so a late correct answer cannot overwrite newer text. Size useful suggestions rather than every keystroke.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Trace document revisions, cancellation and deadlines so a late correct answer cannot overwrite newer text. Size useful suggestions rather than every keystroke.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 303 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

💻 Coding for Infra🔒 Premium

Producer-Consumer PipelinesProducer-consumer pipelines connect stages that run at different rates. Bounded buffers absorb temporary variation, while the slowest sustained stage limits throughput.

🚀 Inference & Serving🔒 Premium

Reranking and Retrieval BudgetsA reranker spends extra computation on retrieved candidates. Count query-document pairs, protect permission boundaries and measure whether better evidence justifies its added queueing and latency.

🚀 Inference & Serving🔒 Premium

Speculative DecodingA draft proposes tokens; the target checks them with a sampling rule that preserves its distribution. Work through the correction after rejection, count accepted drafts separately from emitted tokens, and decide whether the measured round cost earns a speedup.

🚀 Inference & Serving

Latency Metrics: TTFT, TPOT and GoodputRead a stream from its timestamps: distinguish first-token wait, individual gaps and average generation pace. Compute goodput with explicit deadlines, denominators and failure accounting.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Closest to this topicWould request hedging improve an LLM endpoint whose p99 TTFT is too high?Closest to this topicDesign the API for a GPU job scheduler in 45 minutes. What are the resources, states and semantics?Closest to this topicImplement a key-value store where a read can ask for the value as of an earlier version. What is the data structure?

DISCUSSION · 0

No comments yet — be the first to share your approach.