GPU Fleet Reliability & Observability Interview Questions

Use the Llama 3 interruption record without turning job interruptions into device lifetime, predictable failures or a universal spare percentage.

Interpret XID codes with device and software context, separating immediate recovery from evidence-based root-cause investigation.

Use DCGM fields with their actual semantics, combine device health with workload progress, and calibrate alerts on correlated fleet data.

Build observability around durable useful progress and aligned phase timing, showing why full-step timers can hide the late rank.

Investigate rising corrected-memory errors without promising a failure prediction, and follow recovery flags and vendor repair criteria.

Handle an inaccessible GPU with bounded recovery and preserved evidence, without treating a reboot or repeat count as proof of root cause.

Find where useful progress stopped by comparing per-rank traces, host stacks and device events. Distinguish late arrival, mismatched collectives and stalled execution before choosing a recovery action.

Find the critical-path delay using local phase timing and controlled comparisons, rather than diagnosing a node from utilization or clock ratios.

Derive a checkpoint interval from measured job interruptions and exposed save cost. Trace durability, lost work and recovery before treating an estimate as goodput.

Promise measurable job outcomes: time to useful work, recoverable progress and successful restoration. Define the population and exclusions before choosing targets, and give each error budget an agreed operating policy.

Define request success, first-output latency and streaming continuity for each service class. Count bad outcomes against an explicit budget, then use paired long and short windows to detect significant ongoing degradation.

Build an acceptance suite around the node’s intended workload and supported hardware limits. Combine configuration, correctness, communication and sustained-load evidence, then record a baseline and an explicit return-to-service decision.

Correlate clock-event reasons, temperature, configured limits and workload timing. Use the scope of the slowdown to choose the next investigation, and distinguish intentional power limits from unexpected thermal constraints.

Read link counters with their generation-specific units and traffic exposure. Correlate changes with actual communication performance, then choose monitoring, maintenance or immediate recovery from the supported fault state.

Use one consistent timeline to separate discarded progress, recovery delay and detection. Explain the contributing conditions, then assign corrective work with owners, dates and tests that demonstrate the intended behavior.

Page when material impact or risk needs timely human action. Verify that automation actually recovered the workload, group related events, and use measured responder load to improve the policy.

Give every health-state transition a reason, owner and completion condition. Calibrate intervention against missed-fault risk and disruption cost, and prevent a bad signal source from removing the whole fleet.

Stop rollout expansion, preserve evidence and use the safest available mitigation. Compare versions under representative load, and size canary exposure from the regression you need to detect rather than a fixed traffic percentage.

Define which work survived, count the resources it occupied, and explain the remaining time without counting the same delay twice.

Compare the same collective across the ranks that should participate. Missing records, mismatched arguments, and unfinished operations lead to different next checks.

Measure throughput at several power limits, translate the facility target into a GPU budget, and account for slower ranks before changing the fleet.

A successful operation can still produce a wrong value. Explain what comparisons can detect, what they miss, and how to investigate without mistaking ordinary numerical variation for corruption.

Page when timely human action matters. Use budget burn, careful grouping, bounded maintenance rules, and regular testing to keep those pages useful.

Loss of cooling can outpace human response. Use a simple heat calculation to explain why protection must be automatic, then distinguish that illustration from the installed system’s validated behavior.

Follow a request from the client's connection pool to the first text on screen. Work the timing budget, locate a buffering delay and choose what to trace without losing the slow requests.

Compare equivalent requests, inspect the replica’s software and device state, and reduce new traffic only after checking where that load will go.

Separate removing a suspect GPU from production from proving it qualifies for replacement. Use vendor recovery requirements, device history, targeted tests, and the cost of recurrence.

Size a spare pool by separating nodes in repair from ready replacements. Work through a 2,048-node example, then check placement, recovery time and shared failure domains.

Build an incident-review process that leads to verified changes. Set review thresholds, budget the work and interpret recurrence without mistaking a metric for proof.

Determine what the replacement count measures, compare equivalent device exposure and investigate differences by cohort, location, age, software and policy. Job-interruption statistics cannot supply a hardware replacement baseline.