Slurm vs Kubernetes
Slurm and Kubernetes provide different foundations for running GPU workloads. Choose the operating model and supporting components that fit the users and jobs.
TL;DR: Large batch jobs need coordinated admission, placement and launch. Serving workloads need replica management, rollouts and demand-based scaling. Both ecosystems can be extended, so compare the complete proposed stack, including operational effort, rather than choosing from the scheduler name alone.
Two schedulers, two jobs
Start with what each one is. Slurm is a batch scheduler from high-performance computing: you submit a job script asking for, say, 64 nodes for a week, it waits in a queue, and Slurm starts it when the nodes are free. Kubernetes is a container orchestrator from web services: you declare that a service should always have, say, four copies running, and it keeps them running, restarts failures and rolls out new versions. Its unit is a pod, one or more containers scheduled together.
Three batch terms appear in the table. Gang scheduling starts all of a job's workers at once or none of them, because a distributed training job cannot make progress with half its ranks. Backfill lets small jobs run in gaps while a large job waits for its full allocation, as long as they finish before it starts. Fair share lowers the priority of teams that have recently used more than their share.
| Need | Slurm | Kubernetes (default) | Kubernetes with the AI stack |
|---|---|---|---|
| Gang (all-or-nothing) placement | native, the default | no; pods placed one at a time | Kueue, Volcano, KAI |
| Whole-node exclusive allocation | one flag | node affinity and anti-affinity hacks, or a DaemonSet-shaped job | supported by the batch schedulers |
| Fabric topology awareness | topology plugin, --switches | no | Kueue TAS, custom schedulers at the clouds |
| Launch N ranks with rank ids | srun sets SLURM_PROCID | a Job with indexed completions, or a launcher operator (Kubeflow, JobSet) | the operators |
| Fair share across teams from history | slurmdbd, fair-share priority | ResourceQuota only (static) | Kueue cohorts and borrowing |
| Time limits and pre-limit signals | native | activeDeadlineSeconds and preStop, weaker | the same |
| Backfill of small jobs | native | no | Kueue partially |
| Services, autoscaling, rolling deploys | no | native (Deployments, HPA, KEDA) | native |
| Containers | via Pyxis/Enroot plugins | native | native |
| Health-aware node drain | drain state with reasons; epilog scripts | cordon, taints, node problem detector | the same plus DCGM exporters |
| Multi-tenant isolation and RBAC | accounts and partitions | namespaces, RBAC, network policy, richer | the same |
| Who runs it well | HPC and lab teams, teams experienced with batch scheduling | teams experienced with Kubernetes operations | platform teams willing to assemble it |
The table's shape is the argument. Slurm's left column is a solved problem for the training workload and a blank for services; Kubernetes' middle column is the reverse; the right column is what it takes to make Kubernetes serve training, and it is a stack of add-ons rather than one system.
Why training teams keep Slurm
A 512-GPU run needs 64 nodes at once, placed close on the fabric, launched as 512 identical ranks with a known rank id each, left alone for a week, and given warning before it is killed. Slurm does each of those with one line of the batch script. On Kubernetes each is a separate mechanism from a separate project, and the failure modes of the assembly (a gang half-placed by a scheduler that did not know it was a gang, a launcher that did not set the rank, a preemption that killed pods with no warning signal) are exactly the ones training engineers cannot afford in week two of a run. Researchers also know Slurm: sbatch and squeue are muscle memory from university clusters, and a platform that makes them learn Kubernetes YAML to run an experiment pays for it in adoption.
Why serving often needs a separate service layer
An inference fleet is 40 services, each with several replicas, each needing a rolling upgrade when the model or the engine changes, each scaling on queue depth through the day, each fronted by a load balancer and an ingress, each emitting metrics into the company's observability stack and secrets from its vault. Slurm has no concept of a service, a replica set, a rolling deploy or an autoscaler; it schedules jobs that start and finish. Kubernetes was built for precisely this, and the serving ecosystem (KServe, llm-d, the engines' Helm charts, KEDA scalers on custom metrics) assumes it.
The hybrids
Because most organizations run both workloads on one pool of expensive hardware, the industry built bridges in both directions.
Slurm on Kubernetes: Slinky (SchedMD's own project) and Soperator (Nebius) run the Slurm controller and node daemons as Kubernetes pods, so a Kubernetes cluster can host a Slurm partition. Researchers get sbatch; the platform team gets one control plane, Kubernetes' node lifecycle, and the ability to move nodes between the Slurm partition and the serving pool. This is the direction a Kubernetes-first organization takes to keep researchers happy.
Batch schedulers on Kubernetes: Kueue (upstream, admission-based), Volcano and KAI make Kubernetes itself gang-aware, topology-aware and fair-share-aware, with Kubeflow's training operators or JobSet handling the multi-rank launch. This is the direction a team takes when it wants one scheduler and is willing to assemble it; it is also what the managed clouds' AI platforms increasingly ship as their default.
The choice between the two hybrids is mostly about who the users are: if they are researchers who live in Slurm, run Slinky or Soperator; if they are engineers who live in Kubernetes, run Kueue.
The decision, as a table
| Situation | Choice |
|---|---|
| a lab or an HPC center whose main workload is large training runs and whose users know sbatch | Slurm on bare metal, with Pyxis for containers; serving elsewhere |
| a product company with a serving fleet and occasional fine-tuning | Kubernetes with Kueue for the batch jobs |
| a GPU cloud selling both training capacity and inference | Kubernetes as the substrate, a Slurm partition via Slinky/Soperator for training customers, a custom topology-aware scheduler underneath |
| one team, one fleet, both workloads, Kubernetes skills | Kubernetes with Kueue plus a training operator, and topology labels maintained carefully |
| the reversal condition | revisit the choice when workload needs, operational cost or user interfaces change; one workload percentage is not a universal threshold |
Mostly training
- Slurm on bare metal
- Pyxis for containers
- Serving runs on a separate stack
A deliberate hybrid
- Kubernetes as the substrate with a Slurm partition
- Or Kueue plus a training operator
- Needs a clear owner for the extra layers
Mostly serving
- Kubernetes with Kueue
- Batch jobs are the guests
- Topology labels kept current by hand
Working it in the room
Describe the workload mix and the interfaces users need. Explain how the chosen stack handles batch placement, service lifecycle, quotas and failures. A hybrid may be useful, but its extra control layers need a clear owner. State what workload or operating requirement would make you reconsider the choice.
What to remember
- Slurm is native at gang placement, whole nodes, topology, N-rank launch, time limits and fair share from history; blank at services and autoscaling.
- Kubernetes is native at services, scaling, rollouts and the ecosystem; it needs Kueue or Volcano, topology labels and a launcher operator to run training safely.
- Hybrids: Slurm inside Kubernetes (Slinky, Soperator) for Slurm-native users; a Kubernetes batch layer (Kueue, KAI) for Kubernetes-native ones.
- Pick by workload mix and by who the users are; state the reversal condition.
- Training on plain Kubernetes breaks at gang placement first and topology second.