We want to run LLM inference on Kubernetes. What does the deployment actually need beyond a Deployment with a GPU limit?

A plain Deployment reports ready before it can serve and scales on a signal that never moves. What an inference operator adds: weight delivery, readiness tied to warm-up, scaling on queue depth with cold starts accounted for, a prefix-aware router, and rollouts that keep streams alive. KServe and llm-d as of 2026.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🚀 Inference & Serving🔒 Premium

Inference Autoscaling and Cold StartsScaling an LLM fleet is harder than scaling a web service because a replica takes minutes to become useful (pull an image, load 141 GB of weights, warm the cache) and costs several dollars an hour while idle. The signals that work are queue depth and TTFT against the SLO, not GPU utilization, which is misleading for memory-bound decode. The design is a warm pool sized for the burst, hysteresis so the fleet does not thrash, and a cold-start path measured in seconds through snapshots and weight streaming.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Ray on KubernetesRay is a distributed compute framework that turns a cluster into one Python program: tasks and actors placed by Ray's own scheduler, with libraries for data loading, training, tuning and serving on top. KubeRay runs Ray clusters as Kubernetes resources. The combination is common in ML platforms because it gives researchers a single API for pipelines, distributed training and serving, and it is awkward in exactly one place: Ray has a scheduler and Kubernetes has a scheduler, and a platform engineer has to decide which one owns the GPUs. This page covers what Ray is good for, how KubeRay works, and where the two schedulers fight.

Advanced

🚀 Inference & Serving🔒 Premium

Serving Engines: vLLM, SGLang and TensorRT-LLMThree engines serve most open-weight models in production, and they converged on the same mechanisms (paged KV, continuous batching, chunked prefill, prefix caching, speculation, disaggregation) while differing in what they optimize first. vLLM is the default for breadth and hardware coverage, SGLang leads on prefix reuse and structured generation, TensorRT-LLM squeezes the most from NVIDIA hardware at the cost of a compile step. This is a dated page, September 2026; the decision table is what to carry into the room, not the version numbers.

Advanced

📐 AI Systems Design🔒 Premium

Request Routing and Load Balancing for LLMsA load balancer for stateless web services spreads requests evenly and is done. A router for LLM replicas has two things a web balancer never had to think about: each replica holds a cache (the KV pages of recent prefixes) that makes some replicas far cheaper than others for a given request, and each request costs a wildly different amount, so counting connections is meaningless. This page builds the router that handles both: prefix-aware placement with load-aware fallback, cost-aware queue estimates, session affinity, and the failure handling when a replica restarts and its cache is gone.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the scheduling and isolation for a multi-tenant fine-tuning service: hundreds of customers, a few base models, shared GPUs.Next in this trackThe cluster dashboard says 90% allocated and 30% utilized. What is happening, how do you prove it, and what policy fixes it?Next in this trackWe have GPUs in two clouds. Can we run one training job across both, and what would you do instead?

FEDITOR'S NOTE

Scored on the specific gaps of a naive Deployment (readiness, scaling signal, weight loading, rollout), on scaling with cold-start awareness, and on knowing what the current operators (KServe, llm-d) provide versus what the platform still owns.

DISCUSSION · 0

No comments yet — be the first to share your approach.