Slurm vs Kubernetes
Every GPU platform team has this argument, and the two schedulers were built for different jobs: Slurm for long, large, all-or-nothing training on bare metal; Kubernetes for many services that scale up and down. Training fleets run Slurm because gang scheduling, topology and MPI-style launch are native there; serving fleets run Kubernetes because autoscaling and rolling deploys are native there. A platform that does both picks a hybrid: Slurm on Kubernetes (Slinky, Soperator) or a batch scheduler on Kubernetes (Kueue, Volcano, KAI). The interview question is which, for which workload, and why.
TL;DR: Training wants a scheduler that hands out whole nodes together, knows the fabric, launches N identical ranks at once, and runs a job for days without touching it: Slurm does this by default, Kubernetes needs Kueue or Volcano plus topology labels plus a launcher operator to approximate it. Serving wants a scheduler that runs many replicas of many services, scales them on metrics, rolls out new versions without downtime and integrates with everything else in the company: Kubernetes does this by default, Slurm cannot. Most organizations end up with both, and the hybrids (Slurm running inside Kubernetes via Slinky or Soperator, or a Kubernetes batch layer) exist so one fleet can serve both without two teams.
Two schedulers, two jobs
| Need | Slurm | Kubernetes (default) | Kubernetes with the AI stack |
|---|---|---|---|
| Gang (all-or-nothing) placement | native, the default | no; pods placed one at a time | Kueue, Volcano, KAI |
| Whole-node exclusive allocation | one flag | node affinity and anti-affinity hacks, or a DaemonSet-shaped job | supported by the batch schedulers |
| Fabric topology awareness | topology plugin, --switches | no | Kueue TAS, custom schedulers at the clouds |
| Launch N ranks with rank ids | srun sets SLURM_PROCID | a Job with indexed completions, or a launcher operator (Kubeflow, JobSet) | the operators |
| Fair share across teams from history | slurmdbd, fair-share priority | ResourceQuota only (static) | Kueue cohorts and borrowing |
| Time limits and pre-limit signals | native | activeDeadlineSeconds and preStop, weaker | the same |
| Backfill of small jobs | native | no | Kueue partially |
| Services, autoscaling, rolling deploys | no | native (Deployments, HPA, KEDA) | native |
| Containers | via Pyxis/Enroot plugins | native | native |
| Health-aware node drain | drain state with reasons; epilog scripts | cordon, taints, node problem detector | the same plus DCGM exporters |
| Multi-tenant isolation and RBAC | accounts and partitions | namespaces, RBAC, network policy, richer | the same |
| Who runs it well | HPC and lab teams, every supercomputer | every cloud-native team | platform teams willing to assemble it |
The table's shape is the argument. Slurm's left column is a solved problem for the training workload and a blank for services; Kubernetes' middle column is the reverse; the right column is what it takes to make Kubernetes serve training, and it is a stack of add-ons rather than one system.
Why training teams keep Slurm
A 512-GPU run needs 64 nodes at once, placed close on the fabric, launched as 512 identical ranks with a known rank id each, left alone for a week, and given warning before it is killed. Slurm does each of those with one line of the batch script. On Kubernetes each is a separate mechanism from a separate project, and the failure modes of the assembly (a gang half-placed by a scheduler that did not know it was a gang, a launcher that did not set the rank, a preemption that killed pods with no warning signal) are exactly the ones training engineers cannot afford in week two of a run. Researchers also know Slurm: sbatch and squeue are muscle memory from university clusters, and a platform that makes them learn Kubernetes YAML to run an experiment pays for it in adoption.
Why serving teams cannot use Slurm
An inference fleet is 40 services, each with several replicas, each needing a rolling upgrade when the model or the engine changes, each scaling on queue depth through the day, each fronted by a load balancer and an ingress, each emitting metrics into the company's observability stack and secrets from its vault. Slurm has no concept of a service, a replica set, a rolling deploy or an autoscaler; it schedules jobs that start and finish. Kubernetes was built for precisely this, and the serving ecosystem (KServe, llm-d, the engines' Helm charts, KEDA scalers on custom metrics) assumes it.
The hybrids
Because most organizations run both workloads on one pool of expensive hardware, the industry built bridges in both directions.
Slurm on Kubernetes: Slinky (SchedMD's own project) and Soperator (Nebius) run the Slurm controller and node daemons as Kubernetes pods, so a Kubernetes cluster can host a Slurm partition. Researchers get sbatch; the platform team gets one control plane, Kubernetes' node lifecycle, and the ability to move nodes between the Slurm partition and the serving pool. This is the direction a Kubernetes-first organization takes to keep researchers happy.
Batch schedulers on Kubernetes: Kueue (upstream, admission-based), Volcano and KAI make Kubernetes itself gang-aware, topology-aware and fair-share-aware, with Kubeflow's training operators or JobSet handling the multi-rank launch. This is the direction a team takes when it wants one scheduler and is willing to assemble it; it is also what the managed clouds' AI platforms increasingly ship as their default.
The choice between the two hybrids is mostly about who the users are: if they are researchers who live in Slurm, run Slinky or Soperator; if they are engineers who live in Kubernetes, run Kueue.
The decision, as a table
| Situation | Choice |
|---|---|
| a lab or an HPC center whose main workload is large training runs and whose users know sbatch | Slurm on bare metal, with Pyxis for containers; serving elsewhere |
| a product company with a serving fleet and occasional fine-tuning | Kubernetes with Kueue for the batch jobs |
| a GPU cloud selling both training capacity and inference | Kubernetes as the substrate, a Slurm partition via Slinky/Soperator for training customers, a custom topology-aware scheduler underneath |
| one team, one fleet, both workloads, Kubernetes skills | Kubernetes with Kueue plus a training operator, and topology labels maintained carefully |
| the reversal condition | when the training workload's share falls below the serving share, the Kubernetes-first hybrid wins; when researchers dominate, the Slurm-first one does |
Working it in the room
"Slurm or Kubernetes for our cluster?" is a reported question at the GPU clouds and platform teams, and the scored answer refuses the binary: it names what each is native at, gives the table, and picks a hybrid conditioned on the workload mix and the users, with the reversal condition stated. The follow-up held back is "what breaks first if you run training on plain Kubernetes?", answered with gang scheduling (the deadlock) and topology (the scattered placement). The answer that sounds right and fails is "Kubernetes, because it is the standard": it is the standard for services, and a training job on it without the batch stack is a deadlock waiting to happen.
What to remember
- Slurm is native at gang placement, whole nodes, topology, N-rank launch, time limits and fair share from history; blank at services and autoscaling.
- Kubernetes is native at services, scaling, rollouts and the ecosystem; it needs Kueue or Volcano, topology labels and a launcher operator to run training safely.
- Hybrids: Slurm inside Kubernetes (Slinky, Soperator) for Slurm-native users; a Kubernetes batch layer (Kueue, KAI) for Kubernetes-native ones.
- Pick by workload mix and by who the users are; state the reversal condition.
- Training on plain Kubernetes breaks at gang placement first and topology second.