Design the controller that decides whether a GPU node is healthy. What are its states, and how do you tune its false-positive rate?

Give every health-state transition a reason, owner and completion condition. Calibrate intervention against missed-fault risk and disruption cost, and prevent a bad signal source from removing the whole fleet.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Node Health Checks and Burn-InHealth checks look for faults before and during GPU workloads. Burn-in, pre-flight checks and continuous monitoring cover different conditions and need explicit acceptance criteria.

Foundational

📐 AI Systems Design

Control Plane and API Design for GPU PlatformsA control plane records intended work and coordinates its execution. Clear retry, cancellation and listing semantics keep its API useful when requests or workers fail.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Incident Response for GPU FleetsGPU incident response restores useful service while preserving enough evidence to diagnose the failure. Mitigation, repair and return-to-service checks have different purposes.

Foundational

🗂️ Scheduling & Orchestration

Node Lifecycle: Drain, Upgrade and ReturnFleet maintenance moves nodes through drain, repair and validation before returning them to service. Deadlines and concurrency limits keep that work within the available capacity.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackError rate on an inference fleet tripled ten minutes after a deploy. What do you do first, and what should have caught it?Next in this trackHow do you measure effective training time, and where does the missing ten percent of a well-run cluster actually go?Next in this trackWalk me through reading a collective flight-recorder dump. What is in it, and how do you find the rank that caused a hang?

FEDITOR'S NOTE

Distinguish restricting admission from stopping work. Use expected false-positive counts without calling them prevented damage, and keep suspect clearance consistent with repair acceptance.

DISCUSSION · 0

No comments yet — be the first to share your approach.