We want to run LLM inference on Kubernetes. What does the deployment actually need beyond a Deployment with a GPU limit?

Make readiness mean that the engine can serve, scale from measured demand and ready capacity, and drain streams before termination. Kubernetes can implement these controls; an operator packages and reconciles them.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🚀 Inference & Serving🔒 Premium

Inference Autoscaling and Cold StartsInference autoscaling adds or removes model replicas as demand changes. Queue pressure, latency and startup time help determine how much ready capacity to keep.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Ray on KubernetesRay schedules Python tasks and actors across workers; KubeRay manages those workers on Kubernetes. The two scheduling layers need compatible resource and placement rules.

Advanced

🚀 Inference & Serving🔒 Premium

Serving Engines: vLLM, SGLang and TensorRT-LLMAn inference engine coordinates model work and request state. Memory fit and batch throughput do not establish a response deadline; test the actual execution loop before choosing an engine.

Advanced

🚀 Inference & Serving🔒 Premium

Reasoning Budgets and Answer LatencyReasoning tokens consume generation time and retained state before an answer becomes usable. Separate the first generated token from the first answer token, reserve output capacity, and evaluate reasoning limits against task success.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the scheduling and isolation for a multi-tenant fine-tuning service: hundreds of customers, a few base models, shared GPUs.Next in this trackThe cluster dashboard says 90% allocated and 30% utilized. What is happening, how do you prove it, and what policy fixes it?Next in this trackWe have GPUs in two clouds. Can we run one training job across both, and what would you do instead?

FEDITOR'S NOTE

Separate Kubernetes primitives from operator policy. Derive backlog during startup and recovery after scale-out, and distinguish request routing from pod placement.

DISCUSSION · 0

No comments yet — be the first to share your approach.