← Open-Weights Models & Serving Engines / 47

mediumNewBasetenModalTogether AI

Deploy one open-weights model on Kubernetes with vLLM. What goes in the manifest, and what breaks first?

Walk a single-GPU vLLM Deployment from pod scheduling to the first token, then show the three settings that turn a working manifest into restart loops and stuck rollouts.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Walk a single-GPU vLLM Deployment from pod scheduling to the first token, then show the three settings that turn a working manifest into restart loops and stuck rollouts.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🧮 Open Weights & Serving Engines

vLLM Server Arguments That MatterSeparate executor memory, allocated cache pools and logical occupancy. Map context, sequence and token-batch limits to the constraints they control, then tune a pinned deployment against per-rank memory and latency.

🗂️ Scheduling & Orchestration🔒 Premium

Containers, Images and GPU Cold StartsGPU cold starts include scheduling, image loading, model loading and engine initialization. Measure each stage before choosing caches, warm pools or supported snapshot mechanisms.

🧮 Open Weights & Serving Engines

Expert Parallel and All-to-All BackendsCompare tensor and expert placement without confusing either with sparsity. Derive logical dispatch traffic and redundant-expert memory, then choose backends and balancing from per-rank measurements.

🚀 Inference & Serving🔒 Premium

Inference Autoscaling and Cold StartsInference autoscaling adds or removes model replicas as demand changes. Queue pressure, latency and startup time help determine how much ready capacity to keep.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYou serve a model with vLLM on GPUs. What changes when you move the same model to TPUs?Closest to this topicDesign a platform that serves twenty open-weights models of varying size to internal teams.Closest to this topicIs speculative decoding worth enabling on a trillion-parameter mixture-of-experts model?

DISCUSSION · 0

No comments yet — be the first to share your approach.