Slurm and Kubernetes both put work on GPU nodes, and they start from opposite assumptions. HPC batch scheduling, where Slurm comes from, assumes a machine of fixed size and a workload that never runs out. Kubernetes was built to manage long-running processes on resources that can grow [12]. Slurm vs Kubernetes for a GPU cluster comes down to which assumption fits the work, and the three facilities whose designs this explainer reads run both side by side [13, 31, 32].
What each was built for
Slurm's documentation names three functions. It allocates access to compute nodes to users "for some duration of time", it provides a framework for starting and monitoring work on those nodes, and it arbitrates contention by managing a queue of waiting work [3]. A 2023 SchedMD talk sums it up in a colleague's words: "Slurm is a policy engine" [12]. Because the system is fixed and the work is endless, queue prioritization is critical, while cloud orchestration expects all work to run at once by default [12].
Kubernetes works from a declared state. Each object carries a spec that holds the desired state, and controllers are control loops that move the cluster's current state toward it [18].
SchedMD's operator for running Slurm on Kubernetes draws the line in its README. Kubernetes suits workloads that run for an indefinite time, with vague resource needs, on a single node and with loose policy, on a pool that can grow to meet demand. Slurm suits workloads that run for a finite time, with well-defined resource needs and topology, on multiple nodes and with strict policy, on a pool of known size [15].
Slurm vs Kubernetes: the unit of scheduling
A Slurm job declares the tasks, CPUs, GPUs and nodes it needs. Its tasks are allocated as one atomic unit: all of them get resources, or the job does not start [1]. Once the job holds its nodes, it runs job steps in any shape inside that allocation [3].
The default Kubernetes scheduler places one pod at a time. It filters the nodes, scores the ones that pass and binds the pod to the node it ranks first [1, 17]. It does not know that eight pods belong to one training job [1]. A CNCF whitepaper of 5 November 2026 gives the consequence in an example [2]. A job needs 8 GPUs and 6 are free, so the default scheduler starts 6 pods and leaves 2 waiting [2]. The 6 GPUs sit idle, and several partial jobs holding GPUs can deadlock [2].
"Gang scheduling" means two different things in the two projects. In Slurm's documentation it is time slicing: two or more jobs share the same resources and take turns, each suspended while another runs [4]. In Kubernetes it means all-or-nothing placement: a group of pods starts together, or none of them is bound to a node [20]. A Slurm user who wants an all-or-nothing start gets it from the allocation itself [1].
What Kubernetes adds for training jobs
A CNCF whitepaper of 4 November 2026 describes a layered stack above the default scheduler: workflow orchestrators, queue managers or batch schedulers, and device plugins or Dynamic Resource Allocation for the hardware [1]. For training jobs, these public projects fill the gaps:
- Job and Indexed Job. Indexed Jobs, stable since v1.24, give each pod an index and, with a Service, a fixed hostname its peers can reach through DNS [19]. A pod failure policy, stable since v1.31, can end a Job when an exit code shows a software bug, and keep failures caused by preemption or eviction from counting against its retry limit [19].
- JobSet models a distributed training run as a group of Jobs. Its failure policy can recreate the whole set, so the run can resume from its last checkpoint [28].
- Kueue decides when a job waits, is admitted or is preempted. It manages quotas that queues can borrow and lend inside a cohort, and it leaves pod placement to the default scheduler [24, 25]. Its all-or-nothing mode evicts and requeues a job whose pods are not all ready within a timeout [26].
- Volcano runs its own scheduler, which places and binds pods, with gang scheduling, queue quotas and network topology-aware placement [29].
- Kubeflow Trainer offers the TrainJob API, built on JobSet and LeaderWorkerSet, and works with Kueue and with Slurm's bridge for hybrid clusters [30].
The difference behind the "Kueue vs Volcano" question is where placement happens. Kueue admits jobs and lets the default scheduler place their pods [24]. Volcano's scheduler places them itself [29].
Core Kubernetes gang scheduling graduated in v1.37
The Workload and PodGroup APIs, which bring gang scheduling into the default scheduler, graduated in v1.37, released on 26 August 2026 [21]. If the cluster cannot place a pod group's minimum count of pods, none of them is bound, and the group waits in the unschedulable queue while other work is scheduled [20]. The feature is off by default. It needs the GenericWorkload feature gate on the API server, the controller manager and the scheduler [21]. It guards the start only: the running count can drop below the minimum if pods are later deleted or evicted [20]. In its post of 8 September 2026, the Kubernetes working group listed moving these APIs to stable among its aims for v1.38 [21].
Slurm on Kubernetes, and one cluster for both
SchedMD's Slinky project joins the two in both directions [14]. Its slurm-operator runs Slurm clusters inside Kubernetes, with sets of Slurm worker pods that are drained before they are removed for a scale-in or an upgrade [15]. Its slurm-bridge lets Slurm schedule Kubernetes pods. Each Kubernetes workload becomes a representative Slurm job, and a shared node runs native Slurm work or bridged Kubernetes work, never both at the same time [14].
NERSC described in 2023 how it runs Slurm's controller and accounting daemon as Kubernetes deployments, so that Kubernetes moves them to another worker when a node stops responding [13].
Two schedulers in one cluster have a cost. A 2026 paper notes that Slinky makes resource accounting across two scheduling domains harder to keep consistent [31]. SchedMD's 2023 talk lists the drawbacks of running Slurm under Kubernetes: Kubernetes workloads run outside Slurm's view, and prioritizing between the two is difficult [12].
Slurm vs Kubernetes for training, evaluation and inference
An April 2026 paper from a Swiss national supercomputing centre writes that large-scale pre-training, with its data parallelism and synchronous communication, has fit naturally on HPC clusters run by queue-based workload managers [32]. The Swiss centre keeps that pre-training on Slurm and runs inference pipelines, agentic systems and retrieval-augmented generation (RAG) applications on Kubernetes. It also put fine-tuning on Slurm, because hosting it on both would duplicate work such as debugging GPU-to-GPU communication and managing shared access to datasets [32]. The servers such runs are built on, and what their names cover, are set out in DGX, HGX and MGX explained for GPU system buyers.
For inference, the unit is often a request, typically an HTTP or gRPC call [1], and long-running, highly available services conflict with the batch model [32]. In a CNCF survey announcement of January 2026, 66% of organizations that host generative models used Kubernetes for some or all of their inference [35]. The serving side is covered in Deploying LLMs on Kubernetes: serving, scaling and rollbacks.
Evaluation can run as batch work that an event starts. The CNCF paper's example is a finished training job that triggers an evaluation job [1], and Kueue can manage batch jobs together with serving workloads such as Deployments [24].
A UK national supercomputing service, in a paper of August 2026, says Kubernetes "is not considered a replacement for batch scheduling systems" and expects Slurm "to schedule the largest tenants and partitions" [31].
How to choose: a decision table
| Question | Points to Slurm | Points to Kubernetes |
|---|---|---|
| Team skills | Users who want low-level control through job scripts on the command line [12, 32] | Users who want "one-click" abstractions, and teams that manage services as declared state [18, 32] |
| Job shapes | Finite multi-node jobs with known resource needs and topology [15] | Long-running services and requests, with batch jobs through Kueue or Volcano [1, 15, 24, 29] |
| Tenancy | Partitions that limit their users, with account and quality of service (QOS) limits [3, 9] | Namespaces with role-based access and network policies scoped to them, and Kueue quotas [23, 25] |
| Failure handling | Requeue with the same job ID, and health checks that drain nodes [5, 11] | Job failure policies, JobSet restarts from a checkpoint, Kueue requeue after a timeout [19, 26, 28] |
| Accounting | Records for every job and job step once accounting is configured, reported by sacct and sreport [10] | Kueue metrics in Prometheus format [24] |
Failure handling matters more as jobs grow, on either scheduler. On two research clusters running Slurm, 1,024-GPU jobs had a mean time to failure of 7.9 hours, against 47.7 days for 8-GPU jobs [34]. A failure costs the work done since the last checkpoint (derived from [2, 28]), the subject of What research shows about training on spot GPUs.
Rillor designs and builds Kubernetes platforms for superintelligence (SI) work on GPUs, on servers the client controls, where workloads ask for GPUs and the scheduler places them on the nodes that have them.
Read the full report
The full report adds the queue, priority and backfill rules of both schedulers, GPU requests and topology, failure handling, the four ways Slurm and Kubernetes share a cluster, what three facilities report and all 35 sources.