The Reliability Pushback Story
A reliability pushback story explains the risk you saw, the evidence available, and the cost of delaying or stopping work. Make your responsibility clear and describe what would have changed your decision.
TL;DR: Explain the situation, the signal you investigated, your decision and its owner, and what happened afterward. Include the cost of stopping as well as the risk of proceeding. Use your own experience; the scenarios below are hypothetical examples of how to organize it.
What the round is testing
A question about disagreement gives you a chance to explain how you handle a decision with competing costs. The listener needs enough detail to understand why you raised the concern, what you could decide, and how you worked with the people who wanted to proceed. Interview formats and leveling criteria vary by employer; this is a preparation framework, not a universal scoring rubric.
Infrastructure work offers concrete situations to discuss: admitting new nodes before burn-in (a stress test that catches early hardware failures) finishes, shortening a launch canary (a small share of traffic sent to the new version first), or changing checkpoint frequency to save step time. Choose a situation you actually handled. A routine decision explained accurately is more useful than a dramatic story with missing evidence.
The skeleton
| Beat | What it contains | The sentence it produces |
|---|---|---|
| situation | the system, its size, who depended on it, the deadline | "A 2,048-GPU pretraining run, three weeks from a milestone, and 64 new nodes arriving to join it" |
| the signal | one concrete measurement you read, and why it alarmed you | "The burn-in pass rate on the new batch was well below what the previous batch had shown, and two nodes had already logged uncorrectable ECC errors" |
| the decision | what you said no to, to whom, and what you offered instead | "I told the run owner we would not admit the batch that day, and offered a 48-hour extended burn-in with admission of whatever passed" |
| the evidence | the arithmetic or the log you put on the table | "At the failure rate the burn-in implied, the run would restart every few hours instead of every day, which cost more step time than waiting" |
| what changed | the gate, alert, or policy that made the argument unnecessary next time | "Admission now requires a burn-in pass, and the run owner reads the same dashboard I do" |
- Situation
Size, who depended on it, the deadline
- The signal
One measurement you actually read. Skip it and the story is an opinion
- The decision
What you said no to, to whom, and the alternative you gave. Skip it and it is a complaint
- The evidence
The arithmetic you put on the table
- What changed
The gate that made the argument moot. Skip it and it is an anecdote
This order gives the listener enough context to assess each step. The signal explains why you acted, the owner clarifies your responsibility, and the follow-up shows whether the problem was addressed beyond that one decision.
A worked telling: the node admission
Consider a hypothetical node-admission decision. A frontier run is short of its target size, a hardware batch lands, and the scheduler is ready to admit it. The person who owns the run wants the nodes today. You own fleet health.
Make the evidence specific enough for someone else to assess. The burn-in suite on the batch reported a pass rate visibly below the previous delivery's. Two GPUs had already shown row-remapping events in their first day. The NVLink counters on a handful of nodes were climbing during the fabric test. None of that is a proof; it is a pattern the previous batch did not have (Node Health Checks and Burn-In).
The decision names the authority. "I could block admission; the run owner could escalate over me. I said we would not admit the batch that day, and I put the alternative next to the refusal: a longer burn-in, admit whatever passed, and I would personally sit the extended run." The alternative matters. A no without an alternative reads as obstruction; a no with a dated alternative reads as ownership.
The evidence is one chain, said in two sentences. The run at its current size was restarting roughly once a day, which the team could live with. If the new batch failed at the rate its burn-in implied, restarts would come several times a day, and every restart cost the work since the last checkpoint plus the restart time itself (Training Interruption Statistics). Two days of waiting cost less step time than one week of that.
What changed: admission gained a burn-in gate, the run owner got the same fleet dashboard, and the escalation path was written down so the next disagreement went to a person with a name rather than to whoever argued longest.
Then the cost, unprompted. "Waiting cost the run two days of the extra capacity. If I had been wrong about the failure rate, that was two days of a 64-node contribution for nothing, and I would have owned that." This makes the tradeoff visible even if the eventual outcome favored your decision.
A second telling: the launch canary
A second hypothetical example applies the same reasoning to a model launch. A new model version is due at a date the product team announced. The launch plan skips the canary because the previous version's canary "never found anything." Your signal is the load test: p99 time to first token at the announced traffic sits above the SLO on the new engine build, and the number is reproducible (SLOs for AI Systems). The decision: ship on the date, behind a flag, at 5% of traffic, with the old version taking the rest, and a written rollback trigger on the error budget burn rate. The evidence: the p99 curve from the load test next to the SLO line. What changed: the canary is now the default for engine builds, and the rollback trigger is an alert rather than a judgment call at midnight. The cost: the full launch landed a week later than the announcement, and you owned the conversation with the product team.
The follow-up they hold back
Prepare to explain what would have changed your mind. For the node example, that might be additional tests showing the suspected failures were unrelated, or a revised recovery estimate that made immediate admission acceptable. State the cost of a mistaken delay as clearly as the cost of a mistaken launch.
Also be precise about authority. If you made the final call, say so. If another person decided, explain the evidence you supplied and how you carried out the decision. Being overruled does not make the story unusable; it can show how you handled disagreement and followed up on the outcome.
The answer that sounds right and fails
It goes like this: everyone wanted to ship, you alone saw the risk, you argued hard, they listened, disaster was averted. It fails on four counts. No signal was named, so the interviewer cannot tell whether you read data or had a hunch. No cost was stated, so the decision looks free. The other side is a straw figure, so there was no real trade to weigh. Nothing changed, so the same fight is due next quarter. The same story with the burn-in number, the two-day cost, a run owner who had a legitimate deadline, and the gate that followed is a different story in the interviewer's notes.
A related failure is the story where the pushback was "I raised it in the meeting" and the decision belonged to someone else. That is a story about noticing, which is fine for a mid-level loop and thin for a senior one.
Senior and staff tellings
The senior version stops the launch with data and owns the cost. The staff version does that and also changes the shape of the decision: the gate that makes the next batch admit itself, the dashboard both sides read, the written escalation path. Staff candidates also tend to name the trade honestly in the other direction, the times they let something through under a known risk because the expected loss was smaller than the delay. An interviewer who hears only refusals is hearing a gatekeeper. One who hears refusals and calculated admissions is hearing judgment (Leveling Signals: Senior vs Staff).
Working it in the room
Prepare one or two examples from work you know well. In each, give the listener the system and stakes, the evidence, the decision and alternative, and the outcome. A few minutes is a useful rehearsal target, but leave space for questions rather than treating the timing as a rule.
Use numbers you can support and mark estimates as estimates. If you do not remember a measurement, explain what it measured and acknowledge the missing value. Do not fill the gap with the hypothetical numbers on this page.
What to remember
- Five beats: situation, signal, decision and its owner, evidence, what changed. In that order.
- The signal is a measurement (a burn-in rate, a counter, a p99 against an SLO), never a feeling.
- Every no comes with a dated alternative.
- Say the cost of stopping before they ask, and the number that would have changed your mind.
- Broader-scope work can include changing admission rules or review processes across teams; leveling expectations vary.