TL;DR: Explain why the migration was necessary, who had to do extra work, and how you reduced that burden. Describe the rollout gates, recovery options and communication you actually used. A credible story can finish on schedule without an incident; do not invent a rollback or failure to make it sound important.
How to approach it
Choose a real migration you helped deliver. State the reason, your responsibility and the users affected. Follow one important decision from the evidence available at the time through its outcome. Close with the cost of transition and what you would change next time.
A strong answer
Consider a hypothetical move from a homegrown launcher to a scheduler that supports whole-job placement and quotas. Large training teams need that capability; teams with small jobs already have a working launcher. Their resistance may be reasonable: they must change scripts, retest jobs and learn a new failure path without an immediate benefit.
Start by explaining that distribution of costs. “The new scheduler supports the larger jobs we need to run” is more honest than promising that every team will immediately move faster. Ask affected teams about dependencies before setting a date. Some may rely on behavior that the replacement does not yet support.
Migrations and Deprecations covers the rollout mechanics. Gang Scheduling with Kueue and Volcano explains the all-or-nothing placement that motivates this example. Neither concept removes the need to agree who owns compatibility work and which constraints can change.
Use cohorts to learn, not just to count progress
Begin with representative submissions in a side-effect-free test path where feasible. Confirm that a shadow request cannot accidentally launch another training job. Use a small pilot to test integration, support and recovery before moving more teams.
Choose cohorts by risk and the questions they answer. A simple common workload can establish the basic path. Test a difficult dependency early enough to discover a blocker before everyone else commits; “hardest last” is not a universal rule. Expand only when the agreed acceptance checks pass.
The hypothetical five-team cohort has one unresolved recovery failure.
- Five teams try the new path
Run agreed submission, checkpoint, cancellation and recovery checks.
- Four pass; one restore fails
Pause that team’s move. Keep a supported recovery path while compatibility is fixed.
- Retest before advancing
Accept the blocked team only after its required workflow passes.
For a worked status example, suppose 40 teams are in scope and a five-team cohort starts. Four pass submission, checkpoint and cancellation checks; one cannot restore an old checkpoint. That is 4/5 accepted, with one blocked, not a completed cohort because all five submitted a job. Keep the blocked team's supported path available if feasible, assign the compatibility fix and retest before claiming completion.
Describe what recovery could actually restore
A routing switch is reversible only if the old system can still interpret the state created on the new path. Data formats, credentials and irreversible writes can make rollback incomplete. Test the recovery option before the migration reaches that boundary. Where full rollback is impossible, explain the backup, forward-repair or bounded cutover plan and the exposure users accepted.
Tell users about known problems promptly, with the impact, current action and next update time. You cannot guarantee they will never encounter an issue first. A shared status page helps when someone owns it and keeps it accurate; it does not replace direct coordination for a team whose deadline is affected.
Track remaining blockers, active use of the old path and support demand. Before decommissioning, verify that scheduled jobs, automation and infrequent users have moved. A dated retirement plan should include exceptions and an owner for resolving them. Keeping both paths indefinitely has infrastructure and maintenance costs, but removing the old one too early can strand work.
Tell the outcome without improving the facts
Report the planned and actual dates, remaining exceptions, user impact and transition effort you can substantiate. For an illustrative overlap cost, four weeks at $1,500/week plus 20 engineering hours means $6,000 of infrastructure and 20 hours, before other costs. Do not add those quantities into one dollar total without an explicit labor-cost assumption.
If the migration went smoothly, explain which checks helped and what evidence supports that conclusion. If a cohort failed, describe the detection, recovery and changed gate. If the old path was unsafe to retain, a faster cutover may have been justified; explain why its risk was preferable and what support or recovery capacity made it acceptable.
What interviewers probe next
- “Why did users resist?” Describe their actual costs or constraints without labeling them difficult.
- “What could not be rolled back?” Name the state transition and the recovery alternative.
- “What was your contribution?” Separate your decisions from implementation and support performed by others.
Common mistakes
- Claiming benefits that affected users did not receive.
- Calling traffic movement complete before checking real workflows.
- Treating a written rollback command as a tested recovery plan.
- Inventing an incident, delay or clean finish to fit a story template.
Key takeaways
- Four accepted teams out of five is a partial cohort, not completion.
- Rollout order should resolve important uncertainty early while containing exposure.
- Trust depends on honest costs, usable recovery and timely communication.