One serving replica has a per-token latency 40 percent worse than its peers. Find out why.

Compare equivalent requests, inspect the replica’s software and device state, and reduce new traffic only after checking where that load will go.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Compare equivalent requests, inspect the replica’s software and device state, and reduce new traffic only after checking where that load will go.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational

🚀 Inference & Serving

Prefill vs DecodeFollow one prompt through its first prediction and later decode steps, then separate GPU work from the wait a user sees. Learn why batch size and context change the likely bottleneck.

Foundational

🧮 Open Weights & Serving Engines

Serving Benchmarks That Do Not LieDesign reproducible serving experiments with realistic arrivals, cache state and failure accounting. Interpret latency-qualified throughput and conditional per-rank bounds without turning optimistic kernel estimates into production capacity.

Core

🩺 Fleet Reliability & ObservabilitySign in

SLOs for AI SystemsSLOs define the service outcomes users or training teams need. Choose measurable latency, availability and progress targets, then use their error budgets to guide decisions.

Advanced

🚀 Inference & Serving🔒 Premium

Chunked PrefillFollow a long prompt through a shared token budget, including its final partial chunk. Learn why chunking can reduce stream pauses without guaranteeing a fixed token gap or a universal throughput gain.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWrite the policy for when a GPU is replaced rather than returned to service. What are the triggers and what do they cost?Next in this trackHow much spare capacity does a 16,000-GPU fleet need, and what are you actually reserving it for?Next in this trackHow would you build a postmortem practice that people take seriously and that actually reduces incidents?

FEDITOR'S NOTE

Look for workload-normalized latency, evidence across routing and engine layers, and mitigation that accounts for spare capacity and existing streams.

DISCUSSION · 0

No comments yet — be the first to share your approach.