Compute the utilization first so you know how much time is unaccounted for, then read one timeline in a fixed order: GPU idle, then what the host was doing in the gaps, then whether the gradient all-reduce overlapped the backward pass, then the optimizer. Four causes, the evidence for each, and what each is worth.
Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.
Compute the utilization first so you know how much time is unaccounted for, then read one timeline in a fixed order: GPU idle, then what the host was doing in the gaps, then whether the gradient all-reduce overlapped the backward pass, then the optimizer. Four causes, the evidence for each, and what each is worth.
Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.
Scored on computing MFU before profiling, on the reading order of a system timeline rather than a kernel profile, on separating host stalls from collective exposure, and on converting each fix back into MFU.
No comments yet — be the first to share your approach.
