Meta GPU Fleet Reliability & Observability interview questions

Use the Llama 3 interruption record without turning job interruptions into device lifetime, predictable failures or a universal spare percentage.

Build observability around durable useful progress and aligned phase timing, showing why full-step timers can hide the late rank.

Investigate rising corrected-memory errors without promising a failure prediction, and follow recovery flags and vendor repair criteria.

Find where useful progress stopped by comparing per-rank traces, host stacks and device events. Distinguish late arrival, mismatched collectives and stalled execution before choosing a recovery action.

Derive a checkpoint interval from measured job interruptions and exposed save cost. Trace durability, lost work and recovery before treating an estimate as goodput.

Promise measurable job outcomes: time to useful work, recoverable progress and successful restoration. Define the population and exclusions before choosing targets, and give each error budget an agreed operating policy.

Read link counters with their generation-specific units and traffic exposure. Correlate changes with actual communication performance, then choose monitoring, maintenance or immediate recovery from the supported fault state.

Use one consistent timeline to separate discarded progress, recovery delay and detection. Explain the contributing conditions, then assign corrective work with owners, dates and tests that demonstrate the intended behavior.

Give every health-state transition a reason, owner and completion condition. Calibrate intervention against missed-fault risk and disruption cost, and prevent a bad signal source from removing the whole fleet.

Define which work survived, count the resources it occupied, and explain the remaining time without counting the same delay twice.

Compare the same collective across the ranks that should participate. Missing records, mismatched arguments, and unfinished operations lead to different next checks.

A successful operation can still produce a wrong value. Explain what comparisons can detect, what they miss, and how to investigate without mistaking ordinary numerical variation for corruption.

Separate removing a suspect GPU from production from proving it qualifies for replacement. Use vendor recovery requirements, device history, targeted tests, and the cost of recurrence.

Size a spare pool by separating nodes in repair from ready replacements. Work through a 2,048-node example, then check placement, recovery time and shared failure domains.

Determine what the replacement count measures, compare equivalent device exposure and investigate differences by cohort, location, age, software and policy. Job-interruption statistics cannot supply a hardware replacement baseline.

Handle an inaccessible GPU with bounded recovery and preserved evidence, without treating a reboot or repeat count as proof of root cause.

Use DCGM fields with their actual semantics, combine device health with workload progress, and calibrate alerts on correlated fleet data.

Interpret XID codes with device and software context, separating immediate recovery from evidence-based root-cause investigation.

Find the critical-path delay using local phase timing and controlled comparisons, rather than diagnosing a node from utilization or clock ratios.

Define request success, first-output latency and streaming continuity for each service class. Count bad outcomes against an explicit budget, then use paired long and short windows to detect significant ongoing degradation.

Build an acceptance suite around the node’s intended workload and supported hardware limits. Combine configuration, correctness, communication and sustained-load evidence, then record a baseline and an explicit return-to-service decision.