LLM deployment on Kubernetes uses the same objects as any other service: Pods that request GPUs, Deployments that roll out versions, Services that spread traffic and autoscalers that add replicas. Round-robin load balancing suits short, uniform web requests [9]. LLM requests are slow, uneven and tied to caches held in GPU memory [2, 9], so routing, scaling and rollouts each need settings or tools built for that traffic.
In its January 2026 survey announcement, the CNCF reported that 66% of organizations that host generative models use Kubernetes for some or all of their inference, and that 7% deploy models daily [1].
Kubernetes LLM deployment, layer by layer
This explainer splits an LLM endpoint on Kubernetes into four layers, each described in [3, 4, 6, 27].
| Layer | Its job | Kubernetes object or tool | Sources |
|---|---|---|---|
| Inference engine | Loads model weights and runs forward passes | vLLM, SGLang or TensorRT-LLM in a container | [6] |
| Pod | Holds one engine and the GPUs it requests | A Pod with a GPU limit, volumes and health probes | [27, 29] |
| Model server set | Replicas with the same accelerator type, base model and engine | An InferencePool | [4] |
| Gateway | Sends each request to one replica | A Gateway, an HTTPRoute and an endpoint picker | [3] |
Loading weights and running forward passes are the engine's work [6], and engines differ. Kubernetes places Pods on nodes that have the GPUs they request [27], sends traffic only to Pods that pass their readiness checks [28] and rolls out new versions [25]. vLLM's own Kubernetes example asks for one GPU, mounts a volume as a model cache and adds shared memory for tensor-parallel inference [29]. How much memory the weights and the key-value (KV) cache take on each replica is worked out in How much GPU memory LLM inference needs.
Why round-robin load balancing fits LLM traffic badly
The documentation of llm-d, a project for serving LLMs on Kubernetes [6], states the problem: ordinary HTTP requests are fast, uniform and cheap, and LLM requests break all three assumptions [9]. A single request can generate tokens for over a minute, and requests range from thousands of reasoning tokens to more than 100,000 tokens of context [9].
Replicas also hold state. A GPU-backed model server may keep several sessions active with in-memory token caches [2], and the KV cache of each request grows and shrinks as it runs [33]. At any moment, replicas differ in queue length and free cache memory [4, 33]. Requests in a conversation resend the same growing prefix [9], and a replica that already holds that prefix in cache can skip prefill work that another replica would repeat [10]. Request counts hide size too: a count rates an 8,192-token prompt the same as a 512-token one, though the prefill work differs 16 times [8]. Traditional load balancers that route on HTTP path or round-robin lack the capabilities this traffic needs [2].
The inference gateway: InferencePool and the endpoint picker
The Gateway API Inference Extension is an official Kubernetes project for self-hosting generative models [3]. Its InferencePool groups Pods that share a compute configuration, accelerator type, base model and model server [4]. In practice it takes the place of a Kubernetes Service [4]. InferencePool has been v1 since the extension's v1.0.0 release on 9 September 2025 [4, 5].
The gateway asks an endpoint picker where each request should go [3]. The picker tracks KV-cache use, the queue of waiting requests and the active LoRA adapters on each model server, and picks a replica from those metrics [4]. In a comparison published in June 2025, on 10 replicas of one model, with traffic ramped from 100 to 1,000 queries per second, throughput was roughly on par with a standard Service, and p90 latency per output token was significantly lower at 500 queries per second and above [2].
Prefix-aware routing and separate prefill and decode
llm-d's v1.0.0 release came on 20 October 2026 [12]. Its router builds on InferencePool and scores Pods on live metrics, KV-cache affinity and policy [6]. Its prefix-cache-aware routing sends a request to a replica that already holds the cache for the prompt's prefix, so the prefill is not computed twice [10]. An approximate version splits prompts into blocks of 64 tokens by default, counted approximately from characters, and can drift from a server's real cache when a Pod evicts a prefix [10]. A precise version reads cache events from the servers [10].
llm-d also separates the two phases of a request. Prefill, the evaluation of the prompt, is compute-bound. Decode, the generation of tokens, is memory-bandwidth-bound, and running both on one accelerator causes interference and head-of-line blocking [11]. In the 2024 DistServe paper, serving the phases separately handled 7.4 times more requests, or met 12.6 times tighter latency objectives, than the systems it was compared with, with over 90% of requests inside their latency limits [32]. KServe's LLMInferenceService is a Kubernetes custom resource built on llm-d [13].
Models bigger than one node: LeaderWorkerSet
When a model is sharded across several nodes, the unit of replication becomes a group of Pods. LeaderWorkerSet (LWS) deploys such a group from one leader template and one worker template, gives each Pod an index, exposes a scale endpoint for the HorizontalPodAutoscaler and rolls out updates group by group [17]. LWS reached v1.0.0 on 16 October 2026 [17], and KServe's LLMInferenceService starts LWS when its spec declares multi-node workers [13].
LLM inference autoscaling: what to scale on
GPU utilization is the obvious signal and a weak one. llm-d's documentation says GPU utilization can sit near 100% during active batching "whether the server is lightly loaded or saturated" [7]. It scales on queued requests, active concurrency, token backlog or estimated latency against a target instead [7]. vLLM publishes the inputs on a Prometheus endpoint: running and waiting requests, KV-cache use, time to first token and time per output token [30].
The HorizontalPodAutoscaler (HPA) checks every 15 seconds by default and sets replicas to the current count times the ratio of the metric to its target, rounded up [22]. It ignores changes within a tolerance of 0.1 and applies a five-minute stabilization window to scale-downs [22]. KEDA watches event sources and feeds them to an HPA it creates [23]. In llm-d's setup, KEDA's Prometheus scaler reads the endpoint picker's metrics and owns the HPA [7].
Scale to zero and the cold start
HPA scale to zero has been on by default since Kubernetes v1.37, released on 26 August 2026: an HPA with minReplicas: 0 and an object or external metric can take a workload to zero and back [20, 21]. Kubernetes Services do not buffer requests while no Pods are ready, so an HTTP endpoint needs a separate layer that holds them [21]. llm-d's endpoint picker can buffer requests while the pool is at zero, and a request queued then is lost if the picker restarts before a server is ready [8].
The price of scaling out is the cold start. A new LLM replica may need to load weights, initialize GPU resources, compile kernels, capture execution graphs and warm up before it serves anything [31]. In one llm-d example, a replica of an 8-billion-parameter model on one GPU took about 240 seconds to become Ready, and the example sets the scale-up window to 300 seconds [8]. Image volumes, stable since Kubernetes v1.36, mount content from an OCI registry into a Pod, so model files can ship through the same registries and versioning as container images [18, 19]. Whether idle replicas are worth paying for is a utilization question, set out in Self-hosted LLM vs API cost.
Rollouts and rollbacks
A rollback needs a fixed version to go back to. Kubernetes' documentation notes that tags can be moved while digests are fixed, and that pinning an image by digest means a change at the registry cannot change what runs [26].
A Deployment rolls out only when its Pod template changes, and by default keeps at least 75% of its Pods up and allows up to 125% during the rollout [25]. With four replicas of one GPU each, that allows a fifth Pod, which can start only where a fifth GPU is free (derived from [25, 27]). kubectl rollout undo returns the Pod template to an earlier revision, and a Deployment keeps 10 old revisions by default [25].
Model versions roll out through weights. An HTTPRoute splits traffic between backends in proportion to their weights [24]. KServe's canary rollout for LLMInferenceService runs two versions side by side. Weights of 9 and 1 give a 90/10 split, and a rollback lowers the new version's weight or restarts the old one [14]. Weights move traffic and do not judge answers (derived from [14, 24]), so a new model version needs testing on tasks with known answers first, with measures such as those in Agent evaluation metrics and what each one tells you.
When the desired state lives in Git, the rollback is a revert. The OpenGitOps principles ask for desired state that is declarative, versioned and immutable, pulled automatically and continuously reconciled by software agents [35]. A reverted commit becomes the state the cluster returns to (derived from [35]). Rillor designs and builds Kubernetes platforms for superintelligence (SI) work on GPUs, on servers the client controls, where every change ships from Git and a revert rolls it back.
Read the full report
The full report adds the scaling signals and startup mechanics in detail, multi-node and disaggregated serving, rollout settings for GPU Pods, each project's dated state and all 35 sources.