TL;DR: A migration is a trust problem wearing a technical costume. The technical part is well known: shadow the new path first, then move a small cohort, then move by cohort with a real rollback at every step, and keep both paths working until the last user is off. What decides whether it succeeds is different. Users have to hear about every problem from you before they hit it, which means a status page, a schedule they can plan against, and a message when the schedule slips. The rollback has to be exercised rather than documented, because a rollback nobody has run is a paragraph and not a plan. And the migration needs a stated end, since a half-finished migration is worse than either state and every organization has several. Tell the story with the cohort sequence, the one cohort that broke, and what you changed before moving the next one, because that recovery is the part that shows whether the plan had feedback in it.
How to approach it
Say what was being replaced and why, in one sentence each, and be honest if the reason was organizational rather than technical. Then the sequence, cohort by cohort, with the guard at each step. Then the failure, because there is always one. Then how users experienced it, which is the part most candidates omit entirely. Close with the number: how long it took against the estimate, and what the old path cost to keep alive in the meantime.
A strong answer
A typical situation: a platform team is moving 40 training teams from a homegrown job launcher to a scheduler with gang scheduling and quotas. Nobody asked for it. The launcher works. The reason is that the launcher cannot express a multi-node gang, so large jobs deadlock against each other, and the failure is invisible to the teams whose small jobs are fine.
That last sentence is the crux, and it generalizes. Migrations nobody wants are usually migrations where the pain is concentrated somewhere the users are not. Leading with the benefit to them when there is none is the fastest way to lose the room. Naming honestly who the migration helps, and what it costs the people you are asking to move, is what makes the rest of the conversation possible.
The sequence, with what each step is guarding against:
| Step | What runs | The guard |
|---|---|---|
| Shadow | New path receives a copy of real submissions and produces no side effects | Correctness, without any user exposure |
| Pilot | One friendly team, two weeks, both paths available | The unknown unknowns, at a scale where you can hand-hold |
| Cohort 1 | Five teams with simple jobs | The common case, at enough volume to see rates rather than anecdotes |
| Cohort 2 to N | Grouped by job shape, hardest last | Each group's specific breakage, discovered before the next group moves |
| Freeze | Old path accepts no new users | Backsliding, which otherwise runs forever |
| Decommission | Old path off, with a dated announcement and a grace window | The long tail that will otherwise never move |
Migrations and Deprecations covers the mechanics of that ladder and the deprecation policy that makes the last two rows possible. The example here is a move to Gang Scheduling with Kueue and Volcano, which explains why a launcher that cannot express an all-or-nothing placement lets large jobs deadlock against each other. What the interviewer is scoring is narrower: whether each step could be undone, and whether you know what the guard was at each one.
The social half, which is where these stories are won or lost:
the rule: users learn about a problem from you, before it reaches them
a status page for the migration, updated whether or not there is news
a schedule with dates, published early, with the slip announced as soon as it is known
a named person to ask, which is not a rotating queue
what this buys
a team that hears "cohort 3 is delayed a week, here is why" stays cooperative
a team whose job fails on Monday morning with no warning becomes an escalation, and the
escalation costs more time than the entire delay would have
what to say about the cohort that broke
name it, say what broke, say what you changed before moving cohort 4, and say whether you
rolled that cohort back. A migration story with no rollback in it is either a very small
migration or an incomplete telling
The reversal condition: if the old path is causing outages now and cannot be repaired, the cohort ladder is too slow and the honest move is a hard cutover with a heavy support commitment. The tradeoff is real and worth stating: cohorts cost calendar time and preserve trust, a cutover costs trust and preserves calendar time. What makes a cutover survivable is the same thing that makes cohorts work, which is that people knew the date well ahead and had a person to ask. What makes it fail is discovering the cutover through a broken job.
What interviewers probe next
- "How long did it take against your estimate?" Have both numbers. Migrations overrun, and a candidate who says theirs did not is either describing a small one or misremembering.
- "What did keeping both paths cost?" Engineering time, duplicated infrastructure, and the bugs that only appear when two systems disagree. If you never estimated it, say so.
- "Who did you move last, and why?" The hardest cohort, usually, and the reason is that moving them first would have stalled everything behind them.
- "What did you leave behind?" Almost every migration leaves a residue: a compatibility shim, three teams that never moved, a flag that is still there. Naming it is more credible than claiming a clean finish.
Common mistakes
- Selling a benefit the users do not get, when the honest framing is that the cost lands on them and the benefit lands elsewhere.
- A rollback that was documented and never exercised, which is discovered at the worst moment.
- Moving the hardest cohort first because it seemed brave, which stalls the whole sequence.
- No dated end, so the old path survives for years and the organization pays for both.
- Telling the story with no failure in it, which reads as either a trivial migration or an edited one.
Key takeaways
- Shadow, pilot, cohorts grouped by job shape with the hardest last, freeze, then a dated decommission.
- Every step reversible, and the rollback exercised at least once before it is needed.
- Users hear about problems from you first: a status page, a published schedule, a named person.
- State the honest reason, including when the benefit lands on someone other than the people moving.
- Know what the migration overran by and what running both paths cost while it did.