Node Lifecycle: Drain, Upgrade and Return

TL;DR: Stop admitting new work to a node before maintaining it. Let existing jobs finish or apply an agreed checkpoint and preemption policy, then validate the node before returning it. Limit how much capacity can be unavailable at once and pause the rollout when failures or demand reduce the margin.

The cycle

Cordoning and draining are separate on purpose. Cordon is reversible but reduces capacity available for new work, so it is the correct first action for a suspected problem: the node stops taking new work while you decide whether it needs to come out at all. Draining commits to losing the node's current capacity.

Draining is where the wall-clock goes

a fleet-wide operation on 256 nodes
  the work itself: reimage and reboot, about 20 minutes per node
  total node time = 256 x 20 min = 85 hours

  but nodes must drain first
  assume fixed 12-hour jobs sampled uniformly through their runtime: mean remainder 6 hours
  with 16 nodes drained and worked at a time: 256 / 16 = 16 batches
  wall-clock = 16 x (6 h drain + 0.33 h work) = 101 hours = about 4.2 days

  what changes it
    upgrading at checkpoint boundaries agreed with job owners cuts the drain term sharply
    more nodes per batch, bounded by how much capacity can be offline at once
    a drain deadline, after which work is preempted rather than waited for
sanity: 85 hours of work becomes 4.2 days of calendar, and the whole difference is drain, so
        the plan is a scheduling document and the estimate belongs in job boundaries rather
        than node counts

Returning a node, which is where repairs regress

A repaired node is a new node as far as trust is concerned. Returning it without the same checks a fresh node passes is how a fault comes back, and the fault will look new because nobody associates it with the repair.

Node Health Checks and Burn-In covers what belongs in each of those. The lifecycle point is narrower: the checks run on the return transition and not only at provisioning, and a node that fails them goes back out rather than into the pool with a note.

The two policies

• A drain deadline. Without one, a node waits on the longest job indefinitely and a rolling operation stalls. With one, work is preempted at the deadline and the fleet's checkpoint discipline is what makes that survivable, which is a good reason to have both.
• A concurrent-out limit. Cap how many nodes may be out of service at once, expressed as a fraction of the fleet or as whole racks, so a rolling operation cannot remove more capacity than the queue can absorb. This is the policy that stops an automated repair loop from draining the cluster when a bad health signal fires fleet-wide.

That second policy connects to automation directly: any system that cordons nodes on a signal needs a rate limit and a concurrency cap, or one wrong signal drains a large fraction of the fleet in minutes.

What interviewers are listening for

Estimate the drain time as well as the maintenance time. Explain the deadline for long-running jobs, the allowed disruption and the tests required for a node’s return. Show how the operation stops when too many nodes are unavailable or validation reveals a recurring fault.

Key takeaways

• Five states: schedulable, cordoned, draining, out of service, validating. Cordon stops new admission and is reversible; drain commits.
• A 256-node operation is 85 hours of work and about 4.2 days of calendar, and the gap is entirely drain time.
• Return a node through the same checks a new node passes, including a collective benchmark against the fleet median.
• Set a drain deadline, after which work is preempted, or a rolling operation stalls on the longest job.
• Cap how many nodes may be out at once, which is what stops an automated cordon loop from draining the fleet on a bad signal.