TL;DR: Gang scheduling coordinates workers that need to run together before the job can make progress. It avoids allocating mutually dependent workers as unrelated jobs. Group admission does not make pod binding instantaneous, guarantee physical placement, or define every failure policy. Check the scheduler and controller's actual contract.
How to approach it
State the minimum group size and what happens if fewer workers start. Draw a resource-holding example, then distinguish quota admission, placement constraints, readiness and retries.
A strong answer
Two synchronous training jobs each require 40 GPUs in a 64-GPU cluster. Under a policy that launches workers independently and lets them hold allocations while waiting, each job might obtain 32 GPUs. Neither can reach its required world size; all 64 GPUs are allocated without either job training.
That is a possible hold-and-wait stall under the stated policy, not proof that every default scheduler deadlocks forever. Application timeouts, controller cleanup or preemption can release resources. Gang Scheduling with Kueue and Volcano covers coordinated admission; Topology-Aware Scheduling covers whether the admitted group fits the machines.
If job A receives its complete 40-GPU group, B stays queued without consuming GPUs, and 24 remain. Other compatible work can use them. “All ready” means the required workers satisfy the application's readiness conditions; it does not mean every process starts at the same instant.
Volcano's PodGroup documentation describes group-level scheduling concepts such as minimum membership. Kueue's workload model separates admission and quota accounting from the pod scheduler. An admitted quota reservation is not evidence that every pod fits the current node layout. Check the installed integration, version and configuration.
| Stage | Evidence to inspect |
|---|---|
| Admission | Group's quota and admission conditions |
| Placement | Nodes, affinity, devices and topology fit |
| Readiness | Required workers actually become ready |
| Recovery | Timeout, release, retry and checkpoint policy |
Consider 16 nodes with four free GPUs each. The free count is 64. A job requiring each eight-GPU tensor-parallel group to fit entirely on one node still has zero feasible eight-GPU groups. Capacity totals cannot satisfy a placement constraint. Relaxing that constraint changes the network path and may change performance; do not silently assume it is equivalent.
A controller also needs a bounded response when an admitted group cannot become ready. Kueue's all-or-nothing readiness documentation describes readiness waiting and requeue behavior. Verify the configured timeout, whether admissions are blocked while another group starts, and how retries affect fairness. Those are operational choices, not one universal definition of gang scheduling.
For a hypothetical two-minute stall, count held capacity separately from training progress:
held GPUs = 32 + 32 = 64
capacity-time held = 64 GPUs × 2 minutes = 128 GPU-minutes
complete-group allocation = 40 GPUs
remaining capacity = 64 − 40 = 24 GPUs
sanity: 40 allocated + 24 available = 64 total.
That is allocated capacity-time, not proof the devices consumed their full rated power. Successful admission can avoid this specific waste, but reserving resources for a large group can also leave capacity idle while waiting for the last required node.
Start with group-aware admission for fixed-world-size jobs that cannot make useful partial progress. Add bounded readiness and a documented recovery policy. Reverse an aggressive reservation or preemption choice if it starves smaller jobs or spends more time restarting than it saves. Independent one-GPU jobs generally do not need a shared gang; elastic jobs need the supported minimum and resize rules stated explicitly.
Measure median and p99 queue wait separately from admitted-but-not-ready time and actual useful training time. Alert on missing workers with their placement reasons. A single utilization percentage cannot tell whether a large group is correctly queued, blocked on images, or unable to fit a topology constraint.
What interviewers probe next
- “Does a worker failure kill the entire group?” The training framework and controller decide whether to retry, restart, shrink or fail. Admission alone does not specify teardown.
- “Can priorities fix the stall?” Priority combined with effective preemption can change allocation; a priority number alone does not release held resources.
- “How do small jobs avoid starvation?” Use explicit fairness, reservations or bounded backfill policies and measure their consequences.
Common mistakes
- Equating quota admission with all pods successfully placed.
- Promising atomic binding or simultaneous process startup.
- Ignoring fragmentation, readiness deadlines and retry churn.
Key takeaways
- Two 40-GPU jobs can strand all 64 GPUs under partial holding.
- One complete 40-GPU group leaves 24 GPUs available.
- Group size, placement and recovery are separate contracts to verify.