Together AI GPU Fleet Reliability & Observability interview questions

Unlike a training job, a slow serving replica holds nobody else back, so it survives unnoticed while degrading a slice of users. The five causes and the metric that distinguishes each, why routing can create the symptom with no hardware fault, and the fix that limits damage while you investigate.

A published run gives the numbers directly: 419 unexpected interruptions in 54 days on 16,384 GPUs, and about three quarters of them hardware. What that implies per GPU-hour, which components dominate, and why a fleet above a few thousand GPUs must treat failure as routine rather than exceptional.

The driver logs a numbered code when something goes wrong on a GPU, and the number tells you whether to retry the job, drain the node or file a hardware return. The codes worth memorizing in three groups, the action each implies, and the automation that turns a log line into a drained node.

The exporter offers hundreds of fields and about a dozen of them change decisions. The health fields that predict failure, the performance fields that tell you whether work is happening, the one everybody alerts on that means nothing, and the diagnostic levels with the time each takes.

A training run has one number that matters and a handful that explain it. Goodput as the top-level metric, the per-rank timing that finds a straggler among a thousand, the hardware layer beneath it, and the three dashboards that serve three different people asking three different questions.

Correctable errors are corrected, so nothing is wrong yet, and that is exactly why they are the most useful signal a fleet has. What rising rates predict, the remapping budget that decides whether a device can heal itself, and the policy that converts a trend into a scheduled drain instead of a failed training run.

The device has stopped answering on PCIe, so the driver cannot reset it and no software fix exists: the host has to re-enumerate the bus. What causes it, why the node is out until it reboots, the three-strike policy that separates a transient from a dying card, and the automation that handles it without a human.

A hang is a collective that one rank never entered, and finding it means asking which rank is missing rather than what is broken. The flight recorder that answers that in seconds, what to do when it is not enabled, and the four causes that a missing rank turns out to be.

Five causes produce the same symptom and each has a check that takes under a minute. The order to run them in, what each one eliminates, and the arithmetic showing that one slow node in a thousand costs the whole job its speed rather than a thousandth of it.

Three quantities decide it: how often the job is interrupted, how long a checkpoint takes, and how long a restart takes. The formula that turns them into an optimal interval, the goodput it leaves, and the measured result showing that making checkpoints faster is worth more than checkpointing more often.

Availability is the wrong promise for a platform whose jobs run for weeks. Four objectives that a researcher can actually plan against, how each is measured from data you already collect, the error budget that makes them enforceable, and the one number that should never be an objective.

Four objectives, each measured at a percentile because averages hide the experience you are promising. The error budget in minutes per month, the burn-rate arithmetic that catches a fast failure in an hour and a slow one in a day, and why two of the four need separate targets per traffic class.

Infant mortality is real and diagnostics that run for a minute do not find it. The five-stage acceptance suite with what each stage catches and how long it takes, the pass criteria as numbers rather than judgments, and the sustained-load stage that finds the faults everything else misses.

Throttling is the hardware protecting itself, so it produces no error and no failure, only a job that is quietly slower. The two telemetry fields that name it, the arithmetic linking clock to throughput, and why the response differs completely depending on whether the cause is one node or the room.

A link that corrupts occasionally does not fail; it retransmits, and each retransmission costs bandwidth that shows up as a slow job with no error anywhere. The counters that reveal it, the arithmetic turning a replay rate into lost bandwidth, and the threshold where watching becomes draining.

The timeline is the easy part and the detection gap is the valuable part: not what broke, but how long it was broken before anyone knew and why. A worked example with its five sections, the distinction between the trigger and the cause, and the property that separates action items that ship from ones that do not.