vllm

Generation is a decode workload and the update is a training workload, and they want different software on different GPUs. The per-token arithmetic for each side, the 141 GB broadcast that has to happen every step, the ratio of actor to learner GPUs that follows, and the idle time that on-policy training builds in.

A batch of eight requests finishes when the longest one does, and seven slots compute padding until then. The fix is to make the scheduling unit one decode step, and the throughput math shows why that was worth several times the hardware.

Before vLLM, engines reserved the maximum context for every request up front and wasted most of it. The fix borrowed virtual memory from the operating system, and the numbers show how much capacity it returned.

Three engines, one hardware roofline, and the difference between them is which part of the roofline each one reaches first on your workload. The decision is a table, and the table has reversal conditions.

The answer comes from the model card and config.json in five minutes, before downloading anything. The weight footprint, the KV per token that is far smaller than the parameter count suggests, and the parallel degree that has to divide the head count.

Failing at startup and failing under load are different problems with different fixes, and the engine tells you which in its own log. Where the memory actually goes, the four flags that move it, and the arithmetic that says whether it can fit at all.

Expert parallelism reads fewer bytes and pays two all-to-alls, so it loses when the interconnect is wrong or when the routing is uneven. Three candidate causes, the measurement that separates them, and why the third one is the answer more often than the first two.

Both engines cache prefixes, so the choice is not about whether the feature exists. It is about how much of your traffic shares a prefix, what the cache eviction does under pressure, and which tuning model your team can actually operate.

Two point eight trillion parameters fits in one node, because the experts ship in a four-bit format. The footprint arithmetic that reproduces the project's own published hardware minimum, the hybrid cache that needs specific engine support, and why batch-one throughput is a tenth of the bound.

These two flags decide how much memory the engine reserves and how many users share each step, and both are usually left at values that come from the model rather than from the product. The derivation, the SLO that bounds the second one, and the measurement that confirms both.

The two phases want opposite machines, so running them together makes every setting a compromise. What splitting buys, the KV transfer that couples the pools, and the arithmetic that says whether the coupling is affordable for this model.

Forming and then hanging is a different fault from failing to form, and it points at a small set of causes. The rank field that is a cumulative count rather than an index, the collective that never completes, and the three checks that separate configuration from fabric.

It helps most exactly where these models are weakest, which is single-user latency at low batch, and the published gains are larger than on dense models for a reason. The acceptance arithmetic, what it costs at high concurrency, and the measurement that decides.