← LLM Inference & Serving / 33

hard

Design moderation in a streaming inference path without checking text after users have already seen it

Choose the release boundary, account for buffering delay, and keep policy versions attached to decisions. Extend the design to queued review and appeals.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Choose the release boundary, account for buffering delay, and keep policy versions attached to decisions. Extend the design to queued review and appeals.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 318 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🚀 Inference & Serving🔒 Premium

Voice Turns, Barge-In and Playback BuffersVoice interruption must stop unwanted generation and invalidate queued playback. Track turn identity, bound audio in milliseconds, and distinguish generated, submitted and actually played output.

🩺 Fleet Reliability & Observability🔒 Premium

Tracing Inference RequestsConnect gateway, queue, model execution and streaming delivery without confusing overlapping spans or GPU activity with user-visible latency.

🚀 Inference & Serving🔒 Premium

Reasoning Budgets and Answer LatencyReasoning tokens consume generation time and retained state before an answer becomes usable. Separate the first generated token from the first answer token, reserve output capacity, and evaluate reasoning limits against task success.

🚀 Inference & Serving🔒 Premium

Reranking and Retrieval BudgetsA reranker spends extra computation on retrieved candidates. Count query-document pairs, protect permission boundaries and measure whether better evidence justifies its added queueing and latency.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign real-time spelling or code suggestions while the user keeps typing Next in this trackFirst-token latency is fast, but the reasoning model's answer takes seconds. What do you check?Closest to this topicYour speech model runs faster than real time. Why do users still wait or hear gaps?

DISCUSSION · 0

No comments yet — be the first to share your approach.