SGLang Server Arguments That Matter

TL;DR: Tune SGLang for the workload's objective. An offline throughput job can maintain a deep queue; an interactive service needs bounded waiting time. --mem-fraction-static budgets weights plus the KV pool, while chunk size, running-request limits and scheduler conservativeness affect how work uses that pool. Measure occupied cache, request retractions, allocation peaks and latency together. Prefix reuse helps when matching state is retained and reachable, but its token-saving fraction is not an end-to-end speedup.

Choose the objective before copying tuning targets

A typical situation: an interactive deployment is configured from a throughput guide. It achieves high token usage while users wait behind hundreds of requests. The engine may be doing exactly what the configuration encourages.

The project's Hyperparameter Tuning guide explicitly frames its large-batch advice around offline batch inference. Its queue, token-usage and memory-headroom heuristics are starting points for that setting. They are not universal interactive SLOs. Pin the engine version; the live documentation and available flags change.

For a synthetic steady-state example, suppose completions average 20 requests/s and the mean waiting queue holds 100 requests. Little's law gives mean queue time = 100 / 20 = 5 seconds. At a mean queue of 2,000, it is 100 seconds under the same stable flow assumptions. Neither calculation predicts p99, and neither applies unchanged to an unstable queue whose arrivals exceed completions. It does explain why a throughput-oriented queue target can defeat a 1-second interactive budget before inference begins.

A zero queue can be healthy at the offered load. Increase load in a controlled test to find capacity; do not manufacture production waiting merely to fill the queue.

Budget static memory, then verify runtime peaks

SGLang defines the static fraction around weights and the cache pool. vLLM's executor fraction includes weights too, but uses different profiling/accounting for other allocations. Copying the same numeric fraction between engines does not establish equivalent cache capacity. See vLLM Server Arguments That Matter for that distinction.

Use this synthetic single-rank example, with a hypothetical 70 GB resident weight allocation and 327,680 B of KV per position:

The 43.2 GB is needed for activations, graph buffers and other allocations; it is not automatically wasted capacity. Measure the peak for the accepted input shapes, concurrency and model features. Raising the fraction to 0.90 adds 14.4 GB of theoretical pool while removing the same amount of runtime headroom. Do it only when measurements justify that transfer.

Real models add placement, rounding and architecture-specific state constraints. For a hybrid model, inspect recurrent state and retained prefix snapshots rather than applying a single full-attention coefficient to every layer. Device memory alone cannot reveal occupied logical blocks inside a preallocated pool.

Connect each observation to an experiment

The server-argument reference, checked September 7, 2026, defines the controls below. Record resolved values because several defaults are inferred.

Low token usage with waiting work does not uniquely identify excessive conservativeness. Check token/batch limits, dependency stalls and the request mix. Conversely, a full cache is not useful if repeated retractions redo work and miss deadlines. Change one control at a time under reproducible load, record the response, and stop increasing load when the acceptance criteria fail. Use a separate test environment to explore failure limits.

Prefix reuse saves a specific part of the work

For a synthetic 2,200-token request with a retained, matching 2,000-token prefix, only 200 prompt positions need new prefill. That is 90.9% fewer newly computed prompt positions. It is not necessarily 90.9% less prefill time: the suffix still attends to existing context, cache lookup/restore has costs, and kernel efficiency changes. Decode still runs.

RadixAttention manages reusable prefix state. Measure hit length and saved prefill time after routing, including cold requests and eviction. A shared system prompt can help even when each document differs. Changed tokenization, model revision or preceding tokens can invalidate reuse. Hybrid architectures must restore a coherent boundary across their state components. Prefix Caching and KV Reuse covers those conditions.

Other serving engines also support prefix caching. Pick SGLang when its supported implementation, reliability and measured workload results win your comparison. Disabling caching can be a useful diagnostic or a justified configuration for a workload with little reuse and measurable overhead; it is not categorically forbidden in production.

Key takeaways

• At 20 completions/s, a stable mean queue of 100 implies 5 seconds of mean waiting, not an interactive tuning goal.
• In the synthetic 288 GB example, fraction 0.85 leaves 174.8 GB for KV and 43.2 GB outside static allocation.
• Moving to 0.90 transfers 14.4 GB from runtime headroom to the estimated pool; validate that trade before keeping it.
• A 2,000-token hit out of 2,200 saves 90.9% of new prompt positions, not the same fraction of end-to-end latency.