TL;DR: Measure useful progress over elapsed time, then trace the phases and dependencies that explain it. Separate local work from collective waiting; full-step timers can be similar on every rank because fast ranks wait for the slow one. Join job attempts, ranks, device identity and infrastructure events, and page on actionable impact or risk.
How to approach it
Define progress and its denominator first. Instrument lifecycle and phase boundaries with consistent identifiers, then preserve enough per-rank detail to locate delayed work. Design dashboards and alerts around the decisions each audience needs to make.
A strong answer
A run slows down while every GPU remains busy. A mean utilization chart cannot explain whether work is being repeated, one rank is late, or communication is exposed. Even a per-rank full-step chart may show nearly equal durations because each rank waited at a dependency. Stragglers and Hangs needs phase timing, not just more copies of the same end-to-end timer.
Define goodput as accepted, non-replayed training tokens or updates divided by total elapsed time. State whether queue time is included and how variable tokens per update are handled. A useful-time fraction is a different metric with a different unit. Loss and quality checks matter too: fast updates with invalid data or numerical failure are not successful learning.
Illustrative observation window: 3,600 seconds
Accepted unique progress: 90 million tokens
Goodput = 90,000,000/3,600 = 25,000 tokens/s
Executed tokens including replay: 100 million
Raw execution rate = 27,777.78 tokens/s
Sanity: counting replay as new progress overstates useful throughput by 11.11% here.
Collect lifecycle events for admission, launch, checkpoint start/completion, failure, restart and durable progress. Use job ID, attempt/generation, rank, node and device UUID. A timestamp coincidence suggests a relationship but does not prove causation; record confidence and the initiating evidence when attributing a restart.
For performance, mark input readiness, local compute, communication issue/completion and optimizer completion. GPU execution is asynchronous, so a host timer around a launch measures enqueue work unless its completion semantics are defined. Use supported GPU events or tracing where appropriate, and measure instrumentation overhead. Do not add a synchronization on every phase merely to simplify the dashboard.
In a simplified synchronized model, 1,023 ranks finish local work at 400 ms and one at 460 ms. The mean local-work time is (1,023×400+460)/1,024 = 400.0586 ms, only 0.01465% higher. But the rendezvous completes at 460 ms, producing 400/460 ≈ 86.96% of the former throughput, a 13.04% loss. If each timer includes the wait, all full-step times can approach 460 ms. Name which timing the chart contains.
NCCL collectives coordinate participating ranks, but not every collective is a universal host-side barrier. Algorithm, streams and dependencies affect progress. A long observed collective can include late arrival or dependency waiting; use aligned traces before calling it a transport fault. DCGM and GPU Telemetry supplies clocks, activity, health and path context.
Before adding phase times to a dashboard, decide how you will account for overlap. Training may compute while input arrives or a checkpoint is written. Adding all three durations would count some seconds more than once. Use mutually exclusive states for a wall-time budget, or show the dependencies on a critical-path trace. Leave uncertain time unclassified until the evidence explains it, and monitor gaps in the telemetry itself.
The researcher view shows useful progress, loss, recovery and completion estimate. The operator view shows affected jobs, delayed phases, implicated devices and recovery state. The capacity view shows allocated versus useful device-hours, queue demand, fragmentation and repair inventory. Preserve targeted traces for diagnosis while controlling high-cardinality labels and retention costs.
Page when a job stops advancing beyond its expected phase duration, an agreed goodput objective is breached, automated recovery fails or a credible risk requires action. Device health can justify action before visible job impact, especially for safety or data integrity. Expected checkpoint pauses should not look like hangs. Thresholds belong to the workload and service contract, not a universal 80% or 90% rule.
Reverse a local-straggler diagnosis when ranks arrive together and the transfer itself is slow. For many short jobs, emphasize queue wait, success and completion latency while retaining appropriate task-level traces. The data model can be shared even when the dashboard priorities differ.
What interviewers probe next
- "Why do all step timers show the same delay?" Fast ranks may include waiting for the late rank.
- "Can phase durations sum above wall time?" Yes, when work overlaps; attribution needs explicit semantics.
- "How do you avoid counting replay twice?" Use durable accepted progress and attempt-aware bookkeeping.
Common mistakes
- Calling a 15% duration increase a 15% throughput loss.
- Assuming equal full-step timers rule out a straggler.
- Assigning overlapping phase times to a mutually exclusive loss budget.
Key takeaways
- 90 million accepted tokens in one hour is 25,000 tokens/s.
- A 400-to-460 ms step change loses 13.04% throughput.
- Measure local readiness and waiting separately.