Deciding Under Incomplete Information
When evidence is incomplete, compare the cost of waiting with the cost and reversibility of acting. Record the assumptions, the observation that would change the decision, and who will respond if it occurs.
TL;DR: Assess the effects of reversing the decision, including changes already seen by other systems or people. Compare acting and waiting, then choose a bounded next step where possible. A good process can still produce a bad outcome; use the result to update the assumptions.
The classification, and why it comes first
Ask what would remain changed if you tried to undo the action an hour later. A configuration rollback may be quick on the server but slow for clients that cache the old value. A deletion may be recoverable from a tested backup, or permanent if no usable copy remains. A schema change may have affected data that the old application cannot read.
Reversibility is therefore a matter of recovery cost, delay and residual effects, rather than a binary label attached to a command. Check those effects before deciding how much evidence and review the action needs.
Writing the cost of waiting next to the cost of being wrong
Write down the consequences of both choices, including what you cannot yet estimate. Waiting can preserve options while adding delay; acting can produce useful evidence while exposing users to a failure. Neither cost is always smaller.
Consider a hypothetical configuration change intended to reduce latency:
| Choice | Potential benefit | Cost or uncertainty to examine |
|---|---|---|
| Wait for another load test | More evidence before exposing users | Current latency problems continue during the test |
| Try a small canary | Observe the change on representative traffic | Bound affected traffic and preserve a usable rollback |
| Change the whole fleet | Reach more users quickly if correct | A mistaken assumption has a larger impact |
Choose the smallest experiment that can answer the important uncertainty in time. If the action cannot be safely contained or reversed, gather stronger evidence and involve the person who owns that risk. In an interview, explain the comparison using the information you actually had.
The trigger, which is what makes it a decision rather than a guess
Record the expected result, an observation that would cause reconsideration, the owner and the review time. For example: “If p99 has not improved by 30% within 48 hours under matched load, we will revisit the fragmentation hypothesis.” Those values are hypothetical; the real threshold should reflect the baseline, measurement noise and required benefit.
A trigger makes the next action clearer, but crossing it does not automatically prove one root cause. Check whether workload or measurement conditions changed. Then decide whether to reverse the change, gather more evidence or investigate another explanation.
Keep the record brief enough to use during follow-up. The Reliability Pushback Story explains how to connect the original concern, the decision and the later evidence without turning a favorable outcome into hindsight certainty.
Decision quality is not outcome quality
A well-supported decision can turn out badly because an uncertain event occurred. A poorly supported decision can succeed by luck. Assess both the information available at the time and what the result taught you.
In an interview, explain whether you would make the same decision with the same information, and why. You might keep the decision rule but improve monitoring, or conclude that an overlooked piece of evidence should have changed the call. Neither answer needs to defend the outcome as inevitable.
What interviewers are listening for
Give the listener enough information to assess the choice: what was reversible, what waiting cost, what uncertainty remained and who owned the follow-up. If the decision went badly, separate responsibility for the process from hindsight about the result.
Key takeaways
- Classify by reversibility before deciding; the test is whether anything would still be different if you undid it in an hour.
- Check residual effects: cached configuration, deleted data, public messages and state already consumed by another service may be costly or impossible to restore.
- Write the cost of being wrong next to the cost of waiting, which makes most of these tractable.
- Attach a trigger with an owner and a deadline, not just a review date, because a trigger forces a falsifiable prediction.
- Separate decision quality from outcome quality, and be able to say you would make the same call again.