debugging

Half the expected bus bandwidth has four plausible causes needing different fixes: a link trained at reduced width, a fabric that never came up, a topology the job did not expect, or a test that measured latency. The sequence of commands that tells them apart, and what each output looks like on a healthy node.

Links that never come up and links that come up and flap are different faults with different causes. The reach change between 400G and 800G explains most first-time failures, and the counters separate the rest. The order to check, and the one physical measurement that settles it.

A rail-optimized fabric makes this fault findable because the eight rails are identical by construction, so any asymmetry is a defect. Bisecting by rail, then by switch, then by node, then by port narrows 512 links to one in four steps, and the counters say which of three physical causes it is.

Single-node fine and two-node halved is a specific signature, and it points at one thing before anything else. Why the copy path costs exactly about half, the five causes that produce it, and the one log line that confirms which.

It configures the NVSwitch fabric so eight GPUs present as one NVLink domain, and when it is absent nothing errors. Jobs run, results are correct, and everything multi-GPU is slower by a factor the team spends weeks not finding. The check that takes one command.

Acceptance measures the hardware and production measures the whole system, so the gap lives in everything acceptance did not cover. Four candidate layers, the order to eliminate them, and the two that account for most of these cases.

Modules do not fail randomly at three times a published rate, so the excess has a cause and it is usually thermal or a batch. The cohort cuts that localize it, the counters that separate a dying module from a dirty connector, and the environmental cause that looks like a hardware defect.

Registers are allocated in fixed steps out of a fixed budget per multiprocessor, so a small increase in live values can cost a whole resident block. The compiler output that shows it in two lines, the occupancy cliff arithmetic, and the two different failure modes that produce the same symptom.

A bug that scales with the number of tiles is invisible in a unit test that fits in one tile. The three candidates that produce exactly this signature, the measurement that separates them, and a test design that would have caught all three before the kernel shipped.

The watchdog says an all-reduce ran for ten minutes and never finished. That means one rank never entered it, entered a different one, or died. How to read the log line, the flight recorder that names the rank in seconds, the env vars that make the next one debuggable, and why the timeout is a symptom every time.

A flat median with a broken tail means one in a hundred requests is hitting something the others do not. There are five usual suspects, and the order you rule them out in is the answer.

Failing at startup and failing under load are different problems with different fixes, and the engine tells you which in its own log. Where the memory actually goes, the four flags that move it, and the arithmetic that says whether it can fit at all.

Expert parallelism reads fewer bytes and pays two all-to-alls, so it loses when the interconnect is wrong or when the routing is uneven. Three candidate causes, the measurement that separates them, and why the third one is the answer more often than the first two.

Fluent and wrong is a plumbing failure, not a model failure, because a model whose weights are broken produces obvious nonsense. Five places the plumbing goes wrong, the reproduction that isolates each, and the one that is invisible until an evaluation runs.

Forming and then hanging is a different fault from failing to form, and it points at a small set of causes. The rank field that is a cumulative count rather than an index, the collective that never completes, and the three checks that separate configuration from fabric.

A tail that wide is queueing or prompt-length variance, and the two are distinguishable in one measurement. Four causes ranked by how often they are it, the decomposition that attributes the wait, and the fix that is usually a flag rather than hardware.

Identical replicas serving identical traffic should produce identical numbers, so a difference is a defect and the fleet median is the tool that finds it. Five causes, the three that are about the replica and the two that are about what it was sent.

Engine upgrades change defaults, kernel selection and scheduling at once, so attribution is the whole problem. The bisection that works on a version rather than a commit, the four categories of cause, and the pre-upgrade baseline without which none of this is possible.

Tool calling passes through four layers and a break in any one produces the same symptom, so the debugging is a bisection rather than an investigation. Which layer the size of the drop points at, the deploy diff that usually contains the answer, and why no latency metric moved.

Graphs remove the per-kernel launch cost, which is a large share of a decode step at low batch and a small one at high batch, so whether it matters is a question about your operating point. The arithmetic, the four reasons an engine turns them off, and the measurement that settles it.

Find the scale at which it breaks before looking at any node, because the scale tells you which layer to open. The commands in order, the number each should return on a healthy fabric, and the four causes that this measurement sequence separates without guessing.

Four of them diagnose, four configure the machine correctly, and several popular ones override tuning that was already right. Which is which, the symptom each addresses, and why a cluster whose job scripts carry a dozen of these has a configuration problem rather than a tuning one.

A uniform loss across every node rules out hardware and points at something that changed everywhere at once. The differential procedure that isolates a reset default from a real regression, the settings firmware resets on its own, and the change process that catches it in an hour.

A node with eight healthy H100s that Kubernetes thinks has none. The layered check from kernel driver to container toolkit to device plugin to taints and allocatable, in the order that isolates the fault fastest, with the log lines each layer prints when it is the one that broke.

Free GPUs and a pending gang are not a contradiction: the gang needs 64 whole nodes at once and the free GPUs sit on 300 half-used nodes. The isolation order (quota, fit, per-node reasons, one node's device plugin), the command at each step and what it means, and the fixes ranked by displacement.

A hang is a collective that one rank never entered, and finding it means asking which rank is missing rather than what is broken. The flight recorder that answers that in seconds, what to do when it is not enabled, and the four causes that a missing rank turns out to be.

Five causes produce the same symptom and each has a check that takes under a minute. The order to run them in, what each one eliminates, and the arithmetic showing that one slow node in a thousand costs the whole job its speed rather than a thousandth of it.

Seven spans from arrival to the last token, with realistic durations for each. The three places latency hides that a naive trace misses entirely, why the decode phase is one span and not four hundred, and the attribute that explains most tail latency in a single field.

Full GPU utilization during a hang is the clue, because a spinning collective looks identical to real work. The isolation order that finds the mismatched rank in minutes, a runnable reproducer that hangs on demand, and the fix that gates the logging rather than the collective.

Everyone has this story and most candidates tell a flattering version of it. The bias that produces almost all of these mistakes, the cost you have to be willing to name, and the process change that is the difference between an anecdote and a lesson.

Arguing is the losing move and so is capitulating. What moves a vendor case is a reproducer small enough for their engineer to run, counters that isolate their layer from yours, and a bisection that removes your configuration as a variable entirely.