TL;DR: The top-level metric is goodput: useful training steps completed over wall-clock time, which captures restarts, checkpoint pauses, stragglers and idle time in one number a researcher and an executive both understand. Everything else exists to explain a drop in it. Below it, collect per-rank step timing rather than an aggregate, because the aggregate hides the one rank in a thousand that is slow and the whole job runs at that rank's speed. Below that, collective timing from the library's own instrumentation, which distinguishes a slow rank from a slow network. Below that, the hardware layer from the device telemetry. Three dashboards serve three audiences: a researcher wants their run's goodput, step time and loss; an on-call engineer wants which jobs are degraded and which nodes are implicated; a capacity planner wants fleet utilization and the breakdown of where goodput was lost. Page on goodput dropping below a threshold for a run, on a job that has stopped progressing, and on fatal hardware, and on nothing else.
How to approach it
Name the top-level metric first and derive the layers from what would explain a drop in it, because a list of signals with no hierarchy is what produces dashboards nobody reads. Say why per-rank rather than aggregate, since that is the specific design decision that makes straggler detection possible. Then the three audiences, because the same data serves them differently. Close with the alerting policy, which should be short.
A strong answer
A typical situation: a run's throughput drops 15% and stays there. The cluster dashboard shows every node healthy, every GPU at 100% utilization, and no errors anywhere. One rank is running 15% slower than the rest, and because every collective is a barrier the whole job runs at its pace, but the metrics are all aggregates and an aggregate of 1,024 ranks moves 0.015% when one rank slows by 15%.
The metric hierarchy, each layer explaining the one above:
level 0 goodput = useful training steps x tokens per step / wall-clock time
or as a fraction: time spent making forward progress / total time
a well-run large job lands near 90%; the published Llama 3 run reported about that
this is the number in the weekly report and the one a researcher asks about
level 1 where the missing time went, as a breakdown that sums to the gap:
restarts and their recovery (from job lifecycle events)
checkpoint pauses (from the checkpoint instrumentation)
straggler waiting (from per-rank step timing)
data-loader stalls (GPU idle at step boundaries)
scheduled maintenance (from the drain events)
the property that matters: it sums. If the breakdown does not account for the gap,
the instrumentation has a hole rather than the cluster having a mystery
level 2 per-rank step timing: every rank reports its own step duration, and the dashboard
shows the distribution rather than the mean
the metric that finds a straggler: max minus median across ranks, per step
a healthy job: under 5% spread. A straggler: one rank persistently above
this is the level that would have found the scenario above in one glance
level 3 collective timing from the library's instrumentation: how long each collective took
and which rank arrived last
distinguishes "rank 847 computes slowly" from "the network between 847 and its
neighbor is slow", which have different fixes
level 4 hardware telemetry per device: the health and performance fields, which explain
why rank 847 is slow: thermal throttling, a degraded link, memory errors
MFU and HFU covers the efficiency side of level 0; Stragglers and Hangs covers levels 2 and 3; DCGM and GPU Telemetry covers level 4.
Why per-rank and not aggregate, with the arithmetic that makes it obvious:
1,024 ranks, healthy step time 400 ms, one rank at 460 ms (15% slow)
aggregate mean step time = (1,023 x 400 + 460) / 1,024 = 400.06 ms
observed change in the mean: 0.015%, indistinguishable from noise
actual job step time = 460 ms, because every collective waits for the slowest rank
observed change in job throughput: 15%
so the mean of per-rank timings is useless and the maximum is the job's actual speed
collect per rank, display max, median and the spread, alert on the spread
sanity: this is why a cluster can show every component healthy while a job runs 15% slow.
The information was averaged away before anyone looked at it
Three dashboards for three audiences:
| Audience | Question | What the dashboard shows |
|---|---|---|
| Researcher | Is my run healthy and how fast is it going? | Goodput, step time over time, loss curve, tokens consumed, estimated completion |
| On-call engineer | What is broken and which nodes are implicated? | Jobs below their goodput threshold, per-rank spread per job, nodes with fatal or recoverable faults, drains in progress |
| Capacity planner | Where did the fleet's time go this week? | Fleet utilization, goodput breakdown aggregated across runs, failure counts by cause, spare pool depth |
The alerting policy, deliberately short:
page a run's goodput below its threshold for longer than a window (say 80% for 30 minutes)
a run with no step progress for longer than a step-time multiple (a hang)
a fatal hardware fault on any node
ticket a recoverable hardware event, a node quarantined, a rising straggler spread that has
not yet crossed the goodput threshold
never component-level metrics with no job impact, and anything derived from GPU utilization
sanity: the alerting is on the researcher's experience, not on component health, because a
component can be degraded without affecting a job and a job can be degraded with every
component nominally healthy. The scenario above is the second case
SLOs for AI Systems covers turning those thresholds into an error budget.
The reversal condition: this hierarchy assumes a small number of large, long-running jobs, which is the training case. A cluster running many short jobs inverts it: per-rank timing is noise across thousands of short-lived processes, and the top-level metric becomes queue wait and job success rate rather than goodput. The signals below stay the same and the aggregation changes, so a platform serving both needs two views over one collection rather than two collection systems.
What interviewers probe next
- "How do you get per-rank step timing without adding overhead?" A timestamp per rank per step, emitted asynchronously, is a few bytes and no synchronization. The cost is in the collection path, not in the job.
- "What if the straggler moves between ranks?" Then it is not a bad device but something systematic: a data distribution effect, a shared resource, or a collective pattern. That distinction is why the dashboard shows which rank as well as how much.
- "How do you attribute a restart to a cause?" Correlate the job lifecycle event with the hardware faults on its nodes within a window. Most restarts have one, and the ones that do not are the interesting minority.
- "What is a reasonable goodput target?" About 90% for a large well-run job. Below 80% something structural is wrong, and the breakdown says which of the five contributors it is.
Common mistakes
- Reporting aggregate step time, which moves 0.015% when one rank of 1,024 slows by 15%.
- Alerting on component health rather than on job impact, which pages for degraded nodes running nothing.
- A goodput breakdown whose parts do not sum to the gap, which means the instrumentation has a hole.
- One dashboard intended for everyone, which serves the researcher, the on-call and the planner equally badly.
Key takeaways
- Goodput is the top metric; everything else exists to explain a drop, and the breakdown must sum to the gap.
- Collect per-rank step timing and display the spread: one slow rank in 1,024 moves the mean by 0.015% and the job by 15%.
- Four layers: goodput, per-rank timing, collective timing, device telemetry, each explaining the one above.
- Page on job impact (goodput, hangs, fatal faults), ticket on recoverable hardware, never on component metrics with no job effect.