← GPU Fleet Reliability & Observability / 28

hardMetaMicrosoft

How much spare capacity does a 16,000-GPU fleet need, and what are you actually reserving it for?

Size a spare pool by separating nodes in repair from ready replacements. Work through a 2,048-node example, then check placement, recovery time and shared failure domains.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Size a spare pool by separating nodes in repair from ready replacements. Work through a 2,048-node example, then check placement, recovery time and shared failure domains.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🧮 Napkin Math & Capacity🔒 Premium

Capacity Planning and UtilizationCapacity planning connects demand, useful throughput and acceptable waiting time. Serving and training need different models of bursts, job shape and spare capacity.

🩺 Fleet Reliability & Observability

GPU Failure Modes and XID ErrorsXID messages report GPU errors that need interpretation alongside logs and workload behavior. Use the code, recovery guidance and recurrence pattern to choose the next check.

🩺 Fleet Reliability & ObservabilitySign in

DCGM and GPU TelemetryDCGM provides GPU telemetry and diagnostics. Combine activity, errors, environmental readings and application progress to understand a fleet’s health.

🩺 Fleet Reliability & Observability🔒 Premium

ECC, Row Remapping and Memory ErrorsECC and row remapping detect or manage some GPU memory faults. Recovery depends on error containment, remapping state and the device’s supported procedure.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow would you build a postmortem practice that people take seriously and that actually reduces incidents?Next in this trackYour fleet is replacing one percent of its GPUs every month. Is that normal, and how would you find out?Closest to this topicWrite the policy for when a GPU is replaced rather than returned to service. What are the triggers and what do they cost?

FEDITOR'S NOTE

Distinguish mean repair occupancy from a ready-capacity guarantee, and show when two reserve budgets may share the same nodes.

DISCUSSION · 0

No comments yet — be the first to share your approach.