Error rate on an inference fleet tripled ten minutes after a deploy. What do you do first, and what should have caught it?

Stop rollout expansion, preserve evidence and use the safest available mitigation. Compare versions under representative load, and size canary exposure from the regression you need to detect rather than a fixed traffic percentage.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Core

🩺 Fleet Reliability & ObservabilitySign in

SLOs for AI SystemsSLOs define the service outcomes users or training teams need. Choose measurable latency, availability and progress targets, then use their error budgets to guide decisions.

Advanced

🚀 Inference & Serving🔒 Premium

Hybrid State Checkpoints and RollbackA hybrid model must restore attention history and recurrent state to the same token boundary. Learn why a cache hit can require replay, how rejected draft tokens corrupt an in-place state, and what saved checkpoints cost.

Advanced

🧭 Ownership & Judgment🔒 Premium

Migrations and DeprecationsA migration changes both a system and the workflows that depend on it. Explain why the move is worthwhile, how each rollout stage limits risk, what makes rollback possible, and what users need before the old path can be retired.

Foundational

🩺 Fleet Reliability & Observability

GPU Failure Modes and XID ErrorsXID messages report GPU errors that need interpretation alongside logs and workload behavior. Use the code, recovery guidance and recurrence pattern to choose the next check.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow do you measure effective training time, and where does the missing ten percent of a well-run cluster actually go?Next in this trackWalk me through reading a collective flight-recorder dump. What is in it, and how do you find the rank that caused a hang?Next in this trackThe facility asks you to cap GPU power by 30 percent for the summer. What does that cost a training run, and how would you do it?

FEDITOR'S NOTE

Do not promise that rollback is instant, harmless or conclusive. Distinguish expected error counts from statistical evidence, and keep the error-budget arithmetic consistent.

DISCUSSION · 0

No comments yet — be the first to share your approach.