sglang
AI infra interview questions tagged sglang, across every topic.
3 questions · 1 unlocked for you
Concepts behind "sglang"
The curriculum that explains the ideas these questions test.
Foundational
SGLang Server Arguments That MatterSGLang's tuning model is different from vLLM's in one way that matters: it exposes the scheduler's aggressiveness and the static memory fraction as direct knobs, and its own documentation gives target values for the runtime signals those knobs move. That makes tuning it a measurement loop rather than guesswork. Aim for a queue of a hundred to a couple of thousand requests, token usage above 0.9, and five to eight gigabytes of free GPU memory after startup, then adjust the flags that move each one.🧮 Open Weights & Serving Engines
Advanced
Chunked PrefillA long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.🚀 Inference & Serving🔒 Premium
Advanced
Prefix Caching and KV ReuseMost requests to a production LLM share a prefix: the same system prompt, the same few-shot examples, the same conversation up to the latest turn. Prefix caching keeps the KV blocks for those tokens resident and skips their prefill, so a 4,000-token system prompt costs compute once instead of once per request. Radix trees make the lookup cheap, block-aligned hashing makes it safe, and the hit rate is what decides whether it is a 2x or a 10x win. The interview question is how you would route to make it hit.🚀 Inference & Serving🔒 Premium
Advanced
Serving Engines: vLLM, SGLang and TensorRT-LLMThree engines serve most open-weight models in production, and they converged on the same mechanisms (paged KV, continuous batching, chunked prefill, prefix caching, speculation, disaggregation) while differing in what they optimize first. vLLM is the default for breadth and hardware coverage, SGLang leads on prefix reuse and structured generation, TensorRT-LLM squeezes the most from NVIDIA hardware at the cost of a compile step. This is a dated page, September 2026; the decision table is what to carry into the room, not the version numbers.🚀 Inference & Serving🔒 Premium