← LLM Inference & Serving / 33

hardNew

Design moderation in a streaming inference path without checking text after users have already seen it

Choose the release boundary, account for buffering delay, and keep policy versions attached to decisions. Extend the design to queued review and appeals.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Choose the release boundary, account for buffering delay, and keep policy versions attached to decisions. Extend the design to queued review and appeals.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 303 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🚀 Inference & Serving🔒 Premium

Reranking and Retrieval BudgetsA reranker spends extra computation on retrieved candidates. Count query-document pairs, protect permission boundaries and measure whether better evidence justifies its added queueing and latency.

💻 Coding for Infra🔒 Premium

Parsing Kernel Traces and LogsTrace analysis combines parsing, classification and aggregation. Stream large inputs and define the timing metric before deciding which operations dominate.

🚀 Inference & Serving🔒 Premium

Speculative DecodingA draft proposes tokens; the target checks them with a sampling rule that preserves its distribution. Work through the correction after rejection, count accepted drafts separately from emitted tokens, and decide whether the measured round cost earns a speedup.

🚀 Inference & Serving

Latency Metrics: TTFT, TPOT and GoodputRead a stream from its timestamps: distinguish first-token wait, individual gaps and average generation pace. Compute goodput with explicit deadlines, denominators and failure accounting.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign real-time spelling or code suggestions while the user keeps typing Closest to this topicYour speech model runs faster than real time. Why do users still wait or hear gaps?Closest to this topicDesign an LLM batching system end to end: the queue, the batch, the KV cache and streaming. Give me numbers.

DISCUSSION · 0

No comments yet — be the first to share your approach.