TL;DR: Rank by measured hours, not by irritation. Two weeks of writing down what the team actually does, in half-hour granularity, reorders the list almost every time: the task everyone complains about turns out to cost three hours a week and the boring one nobody mentions costs eleven. Then apply two filters. How much damage a wrong automation can do, taking the reversible things first, since an automation that drains a node is recoverable and one that reimages it is not. And whether the task is stable, because automating a process that is still changing weekly means maintaining two things instead of one. Within any given task, detection ships before remediation. A system that tells you a node is unhealthy and does nothing is useful on day one, builds the confidence and the data for the remediation, and cannot cause an incident by itself. The most common failure is building the remediation first, having it act on a signal nobody validated, and taking out capacity at three in the morning.
How to approach it
Start by saying you would measure, and say how, because a candidate who ranks without data is telling you they will guess on the job too. Then give the two filters and the order they apply in. Then the detection-then-remediation rule with the reason. Close with how you would know the automation worked, which is a measurement of the same hours you started with.
A strong answer
A typical situation: a four-person platform team supporting 2,000 GPUs is spending most of its week reacting. Everyone agrees that node health handling is the worst part of the job. The lead is about to spend a quarter building automatic node remediation.
The measurement first, because it usually changes the plan:
| Task | Felt cost | Measured, two weeks | Per week |
|---|---|---|---|
| Node health triage | "The worst thing we do" | 6.5 hours | 3.25 h |
| Quota and access requests | "Annoying but quick" | 22 hours | 11 h |
| Job failure triage for users | "Part of the job" | 14 hours | 7 h |
| Capacity reporting for finance | "Once a month" | 9 hours | 4.5 h |
Quota requests cost more than three times node triage, they are individually trivial, and nobody named them because each one takes four minutes. That is the general pattern: high-frequency small tasks hide from intuition, and low-frequency painful ones dominate it. A self-service quota page is also far less risky to build than node remediation, so it wins on both filters at once.
The two filters, applied after the measurement:
filter 1: what a mistake costs
reversible and cheap first
a self-service quota page that grants too much: fixed by editing a number
an automatic drain that cordons a healthy node: fixed by uncordoning
irreversible or wide last
an automatic reimage: destroys local state, and a bad health signal takes out a rack
an automatic scale-down: releases capacity that may not come back for hours
the rule: the first automations should be ones where being wrong is embarrassing rather
than expensive
filter 2: is the process stable
a task whose steps changed three times this quarter is not ready to automate
automating it means every change now costs two edits, the runbook and the code
write the runbook first, run it unchanged for a month, then automate the runbook
Within a task, the ordering that keeps automation from becoming the incident:
stage 1 detect and report
the system watches the signal and files a ticket or posts to a channel
cost of being wrong: a false ticket
what you learn: the true positive rate of your signal, measured over weeks
stage 2 recommend and require approval
the system proposes the action and a human clicks
cost of being wrong: a human catches it
what you learn: whether the recommended action was the one the human would have taken
stage 3 act, with limits
the system acts, with a rate limit and a maximum concurrent count, and stops if it is
acting more often than usual
the rate limit is the part people skip, and it is what keeps a bad signal from draining
fifty nodes in four minutes
Incident Response for GPU Fleets covers the health loop this ladder produces and the runbooks that stage 1 is built from. Node Health Checks and Burn-In covers which signals are trustworthy enough to act on, which is the input to whether stage 3 is safe at all.
The reversal condition: when the manual work is a symptom of a design problem, automating it makes the design problem permanent. A team spending eleven hours a week on quota requests could build a self-service page, or could give teams quotas they manage themselves, which removes the task instead of speeding it up. Ask whether the task should exist before making it cheaper, and be suspicious of any automation that mainly moves work from a human to a script without changing who owns the decision. The strongest answers to this question include one example of a task the candidate deleted rather than automated.
What interviewers probe next
- "How would you measure toil without annoying the team?" A shared sheet with half-hour granularity for two weeks, filled in once each evening rather than in the moment. Precision matters less than relative ranking.
- "What would you never automate?" Anything irreversible acting on a signal with a false positive rate you have not measured. Say the rate you would want first.
- "How do you know the automation worked?" Re-measure the same hours. A team that automated node triage and is still spending six hours a week on it has built something that does not fit the work.
- "What happens when the automation is wrong at 3 a.m.?" It should have a rate limit, an off switch that one person can hit, and a log that says what it did and why.
Common mistakes
- Ranking by how much a task is disliked rather than by hours measured.
- Building remediation before detection, so the first action runs on an unvalidated signal.
- Automating a process that is still changing, which doubles the maintenance instead of removing it.
- No rate limit, so one bad signal drains a large fraction of the fleet before anyone notices.
- Automating a task that should have been deleted, which makes the underlying design problem permanent.
Key takeaways
- Measure two weeks in half-hour granularity first; high-frequency small tasks hide from intuition.
- Two filters after the measurement: what a mistake costs, and whether the process is stable.
- Ladder within a task: detect and report, then recommend with approval, then act with a rate limit.
- Every acting automation needs a rate limit, a concurrency cap and a one-person off switch.
- Ask whether the task should exist; deleting it beats automating it, and that story is the strongest version of this answer.