TL;DR: Separate durable job intent from physical allocation ownership. Index eligible jobs by policy, reserve whole gangs and reconcile uncertain launches. Preemption requests start a bounded checkpoint grace period; either a new committed checkpoint or the last durable checkpoint becomes the resume point. Release GPUs only after old workers are stopped or fenced from using them.
How to approach it
Clarify whether 100,000 means registered jobs, queued jobs or running jobs. Establish GPU count, minimum gang size, topology constraints, quota and deadlines. Draw launch and preemption separately. Explain the durable records, bounded placement work and the invariant that two allocation owners cannot use the same device.
A strong answer
A gang is the full set of GPUs a distributed job needs at the same time; preemption means stopping a running job to free its GPUs for higher-priority work, ideally after it saves a checkpoint. A worker can disappear after receiving a launch command but before acknowledging it. The scheduler cannot safely interpret that missing reply as proof that the GPU is free. The same uncertainty appears when a preempted worker fails to acknowledge termination.
In a toy 16,384-GPU fleet, 100,000 registered jobs are plausible, but 20,000 simultaneous jobs each requiring at least one exclusive GPU cannot fit. Queue cardinality sizes the control plane; feasible concurrent allocations size the data plane. Gang Scheduling with Kueue and Volcano adds a stronger condition: an eight-GPU job needs a valid eight-GPU placement, not eight unrelated free entries.
Use a job record for requested resources, tenant, priority, constraints and retry policy; an allocation record for device IDs, ownership epoch and lifecycle; and checkpoint metadata for committed manifests and resume state. Store each policy decision and the command it requires in the same database transaction. This pending-command record is often called an outbox. A reconciler retries the command until the observed worker state matches the decision; repeated delivery must not create another allocation. The database transaction itself cannot launch a remote process atomically.
The normal path is submitted, queued, reserving, launching, running, then terminal cleanup. A reservation either covers the entire gang or is released; partial holds need a bounded protocol or they can deadlock competing jobs. Launch failure enters reconciliation before reuse. Reservation leases prevent indefinite control-plane holds, but an expired lease alone does not stop a process already using hardware.
On preemption, record the request and grace deadline before sending the command. Start the timer then, not after an acknowledgment that might never arrive. The worker may publish a new checkpoint only after every required shard and its manifest meet the durability contract. If grace expires, select the last verified checkpoint. Both outcomes enter stopping. Confirm termination through the node authority or quarantine an unreachable node until old execution cannot continue. An epoch fences stale commands and result writes; physical GPU reuse additionally requires the old process to be unable to run.
Checkpoint-aware victim selection uses observed state. If the last durable checkpoint is 25 minutes old and restart takes 10 minutes, immediate eviction costs roughly 35 minutes of replay plus restart, before requeue delay. For a 512-GPU job:
Observed checkpoint age = 25 minutes; restart = 10 minutes
Replayed work plus restart = 25 + 10 = 35 minutes, before requeue
Lost allocation time = 512 × 35/60 ≈ 298.7 GPU-hours
Sanity: 512 GPUs lose over half an hour each, not 35 GPU-minutes total.
A fresh checkpoint that takes 40 seconds to create can be cheaper when the incoming job can tolerate the wait. The T/2 lost-work estimate in Checkpoint Cadence is an expectation for randomly timed interruption under assumptions; it is not the observed age of a selected victim.
Multi-Tenancy, Quotas and Fair Share sets eligibility before cost: protect non-preemptible guarantees, enforce quotas, then consider borrowers and priority. Kueue's fair-sharing documentation distinguishes admission and preemption policies. Do not flatten all configurations into one usage/quota formula. Preempt whole non-elastic gangs, include checkpoint/restore cost and impose a cooldown or bounded disruption policy. A permanent preemption cap can conflict with promised reclaim rights, so resolve that contract explicitly.
Index jobs by tenant and resource class with eligible-head queues, delayed-job timers and cached topology summaries. Bound placement attempts and time spent per scheduling cycle; do not scan all 100,000 jobs on every heartbeat. Ordering 150 queue heads is cheap, but it does not prove that topology placement finishes within a second. Aging, reservations for large jobs and backfill keep a stream of small jobs from starving a wide gang. Backfill jobs need an enforceable finish or preemption plan before the reserved window.
Crash tests should kill the leader after durable intent but before command delivery, after worker launch but before acknowledgment, and during checkpoint publication. The invariant is one valid allocation owner, recoverable job intent and no resume pointer to partial state. Track p99 queue age by class, placement failure reasons, checkpoint grace expiries, quarantined capacity and replay GPU-hours. A low average queue wait can hide one tenant that never starts.
What interviewers probe next
- What if the node never acknowledges stop? Quarantine its resources and reconcile through an authoritative node or infrastructure action. Do not let lease expiry alone authorize a conflicting launch.
- What if the cheapest victim is guaranteed capacity? It is ineligible unless the policy explicitly permits that disruption. Apply contract eligibility before optimizing replay cost.
Common mistakes
- Starting the grace timer only after an acknowledgment, leaving a hung worker an unbounded wait.
- Treating T/2 as the current checkpoint age, or reserving fragments of a non-elastic gang indefinitely.
Key takeaways
- 100,000 queued records do not imply 100,000 running allocations.
- Grace starts when preemption is requested; both checkpoint outcomes require a safe stop barrier.
- A 25-minute-old checkpoint plus 10-minute restart costs about 298.7 GPU-hours for 512 GPUs.
- Durable intent, idempotent commands and reconciliation handle uncertainty; no acknowledgment is not proof of no execution.