Slurm or Kubernetes for a 2,000-GPU training cluster? Make the case, and tell me what you lose either way.

TL;DR: For a cluster whose work is multi-node synchronous training, Slurm wins out of the box: gang scheduling, topology-aware placement, backfill, fair share, MPI and container launch through Pyxis are native, and the failure modes are twenty years old. Kubernetes wins where the same fleet also serves models, runs pipelines and notebooks, and needs autoscaling, RBAC and the operator ecosystem, but it needs Kueue or Volcano for gang semantics, a topology model, and an operator for the GPU stack before a training job runs well. Large fleets increasingly run both: Slurm partitions for the big runs, Kubernetes for everything else, or Slurm on Kubernetes (Slinky, Soperator) to get one control plane.

How to approach it

Ask what fraction of the GPU-hours are multi-node training versus serving, notebooks and pipelines, and who the users are (researchers who write sbatch, or product teams who write Helm charts). Then list what each scheduler has natively and what has to be added, in the order a training job needs it: admission, placement, launch, fairness, recovery. Give a decision tied to the workload mix, with the mix ratio that flips it, and name the hybrid as the thing large clusters actually run.

A strong answer

A typical situation: a lab buys 250 nodes of 8 H100s, the research team has run Slurm for years, and the new platform team wants Kubernetes because the serving stack, the observability and the CI already live there. Both sides are right about their own workload.

What a training job needs, and who provides it:

The Slurm for AI Clusters page covers the native path in detail; Slurm vs Kubernetes carries the ecosystem comparison. The answer that earns the round is the arithmetic of what the gap costs.

Without gang scheduling, the default Kubernetes scheduler places pods one at a time, and two large jobs can each hold half the cluster forever. The cost of a missing feature is not slowness, it is an idle fleet that reads as full:

cluster: 2,000 GPUs = 250 nodes × 8
two jobs of 1,200 GPUs each, submitted minutes apart, no gang scheduling
  the scheduler interleaves: job A gets 1,000, job B gets 1,000, cluster full
  both block at the first NCCL collective waiting for ranks that will never come
  useful work: 0 of 2,000 GPUs, at $2.50 per GPU-hour ≈ $5,000 per hour of nothing
with gang admission: A runs on 1,200 (800 free for smaller jobs), B waits holding nothing
sanity: the failure is a live-lock visible only as "every GPU allocated, 0% SM active";
        Slurm has never had it; Kubernetes has it until Kueue or Volcano is installed

Topology is the second gap, and it costs throughput rather than deadlock. A 64-GPU job that Slurm places on 8 whole nodes under one leaf runs its tensor-parallel groups on NVLink; the same job placed as scattered pods runs some TP groups across the NIC and can lose more than half its step time (Topology-Aware Scheduling works this to a 2.5x step-time gap). Kubernetes reaches parity with a topology model and Kueue's TAS, but someone has to build and maintain the labels.

What Kubernetes gives that Slurm cannot is everything that is not a batch job. Serving needs replicas, readiness, rolling upgrades and autoscaling on custom metrics; pipelines need CRDs and operators; notebooks need per-user isolation and idle reclaim; all of it needs RBAC and network policy. Slurm has no answer to a Deployment, and a lab whose GPU-hours are 40% serving cannot run serving through sbatch.

So the decision follows the mix:

GPU-hours by workload, illustrative
  ≥ 80% multi-node training, users write sbatch:      Slurm, with Pyxis for containers
  ≥ 50% serving, pipelines, notebooks:                Kubernetes, with Kueue and a training operator
  mixed, one fleet, one team owning it:               hybrid: Slurm partitions for large runs
                                                      (or Slurm-on-Kubernetes), Kubernetes for the rest
sanity: the cost of the wrong choice is asymmetric; a Slurm cluster with a serving need is a
        second cluster; a Kubernetes cluster with a big-training need is three add-ons and a
        year of scheduler tickets

For the 250-node lab, the hybrid: a Slurm partition of 200 nodes for the big runs, where gang, topology and fair share are native, and 50 nodes under Kubernetes for serving, evaluation and notebooks, with a quota policy that moves nodes between them monthly. Slinky and Soperator, which run the Slurm control plane and nodes as Kubernetes workloads, give one control plane and one node pool with Slurm's batch semantics, and by 2026 that is what several GPU clouds offer as their default training environment.

The condition that reverses the hybrid toward pure Kubernetes is a fleet where the biggest job is 64 GPUs and the users already write Kubernetes manifests: Kueue with gang admission and a training operator covers that without the topology work, and one control plane is worth more than Slurm's polish. The condition that reverses it toward pure Slurm is a fleet where nothing serves traffic, which is rarer every year.

squeue and kubectl get pods answer the same question in two vocabularies, which is most of the operational cost of running both.

What interviewers probe next

• "Your researchers refuse to write YAML. What changes?" Nothing about the scheduler; give them a submission CLI or run Slurm-on-Kubernetes so sbatch still works.
• "How does a 512-GPU job ever get a window on a busy Kubernetes cluster?" The same way it does on Slurm: reservation (hold freed nodes until the gang fits) or priority aging in Kueue; without one, small jobs starve it forever.
• "What does Slurm do badly for training?" Elasticity and services; a job that wants to shrink when a node dies needs a framework that handles it, and Slurm's job model is fixed-size.
• "Which would you choose for a 64-GPU startup?" Kubernetes with Kueue; the batch gaps are small at that size and the serving stack has to exist anyway.

Common mistakes

• Answering with a preference ("Kubernetes is the future") instead of a workload mix and a reversal condition.
• Forgetting that stock Kubernetes has no gang scheduling, then describing a training platform that deadlocks on its first busy day.
• Claiming Slurm cannot run containers; Pyxis and Enroot have run images under srun for years.
• Treating the hybrid as a cop-out; it is what most large fleets run, and the interviewer wants to hear how the boundary is managed.

Key takeaways

• Slurm has gang, topology, backfill, fair share and MPI launch natively; Kubernetes needs Kueue or Volcano, a topology model and a training operator to match.
• Kubernetes has services, autoscaling, RBAC and the operator ecosystem; Slurm has none.
• Two 1,200-GPU jobs on 2,000 GPUs without gang scheduling can hold 1,000 each and do nothing.
• Decide by GPU-hour mix; the hybrid (Slurm partitions or Slurm-on-Kubernetes, plus Kubernetes for the rest) is the common landing.