Self-Hosting Frontier Open-Weight Models

What the diagram shows

Teaching example in GiB per rank: EP 8 has 40 expert + 12 other = 52; EP 16 has 20 expert + 12 other = 32. The naive whole-checkpoint division at EP 8 predicts only 41.5. The bars share a zero baseline. Runtime and cache are excluded.

Figure 3. Replicated weights stay on every rank

At sixteen EP ranks, the same layout needs 320/16 + 12 = 32 GiB per rank. Expert storage falls, but non-expert replication stays. The cluster now holds 512 GiB of weights in total. Adding GPUs has increased total resident bytes while reducing bytes per GPU. Both statements can be true.

The last row is an idealized comparison. Embeddings, normalization parameters and particular projections may not follow that simple split. Use the engine's actual placement and the most heavily loaded rank. An average across eight GPUs cannot protect the ninth tensor allocation on the fullest one.

Reserve space before admitting traffic

Suppose a rank exposes 80 GiB of usable memory in a laboratory example. Its weights need 52 GiB; measured graph and communication allocations need 8 GiB; the operator reserves another 4 GiB for variation. That leaves 16 GiB for request state. These assumed values are not an H100 specification. They describe how to read a measured budget.

Measure after loading, after graph capture and during the worst tested mix of prefill and decode. Record the maximum per rank. Multimodal inputs, speculative drafts and unusually large prefills can produce peaks that a short text prompt never exercises. Memory-utilization settings control an engine's allocation policy; they do not prove that the remaining space covers every transient allocation.

What the diagram shows

Full attention retains state that grows with sequence length. Sliding attention can retain a bounded recent window if implemented that way. Recurrent attention carries an updated fixed-shape state instead of one full K/V record per old token. A hybrid model combines these components; temporary prefill allocations are additional.

Figure 4. Growing, bounded and recurrent state

These examples are not a memory-efficiency ranking. They leave out weights and different architecture-specific state, and the models do different work. Their purpose is to show why one universal “bytes per token” number is unreliable.

Sparse attention still has storage

Sparse attention chooses a subset of positions to attend to. That can reduce work without deleting every unselected position from the cache: a later query may select a different subset. GLM-5's DSA should therefore not be sized by multiplying cache bytes by the fraction of attended positions. Include the retained compressed state and any index state required by the implementation.[4][23]

DeepSeek V4 adds a different compressed-attention design. Its configuration includes compression-related fields, so inserting its head count into a conventional GQA formula would be especially misleading. Use the documented implementation's allocations and a sweep across sequence lengths; do not transfer the Kimi MLA result to V4 merely because both reduce attention costs.[21]

Convert logical state into an admission limit

Return to Chapter 3's 16-GiB cache allowance on one rank. Suppose measurements for a particular layout establish an effective growing cost of 80 KiB per live token on the limiting rank, after including its placement and block overhead. Reserve another 1 GiB for bounded per-sequence state and allocation variation at the proposed concurrency.

The remaining 15 GiB supports floor(15 × 2³⁰ / (80 × 2¹⁰)) = 196,608 live-token equivalents. Reserving 8,192 prompt tokens plus 1,024 output tokens for each request gives 9,216 tokens/request and a conservative cap of 21 such requests. The 22nd would require 202,752 tokens, which exceeds the budget. Our earlier demand estimate was about 42 concurrent requests, so this layout needs another complete replica, a smaller memory footprint, a different traffic policy or a measured reduction in state per rank.

The break-even progress is 55/30 = 1.83 tokens per cycle. At 1.5 tokens per cycle the speculative path takes 36.67 ms/token and is slower than the original 30-ms step. All timings here are assumed. They show what to measure and how to combine it.

What the diagram shows

Assumed cycle: 8 ms drafting, 42 ms verification, 5 ms other overhead, total 55 ms. At 3.2 advanced tokens/cycle, cost is 17.19 ms/token versus 30 ms ordinary decode, about 1.75 times faster. At 1.5 advanced tokens, cost is 36.67 ms/token, slower than ordinary decode.

Figure 8. Measure progress per verification cycle

“Accepted tokens” and “tokens advanced” may be different counters. A cycle may also emit a correction or bonus token. Define the numerator from the implementation's behavior before computing speedup. Likewise, an acceptance fraction for draft tokens does not reveal how much verification work was performed.

Compare at the same offered load

Speculation consumes draft compute, target verification work and additional state. At low concurrency, spare compute may make that trade attractive. At higher concurrency, the target may already use its resources efficiently, and the extra work can reduce aggregate goodput. Run both configurations at the same arrival rate and length distribution, then repeat near the service's saturation point.

Keep input processing in the experiment. A decode-only benchmark can miss worse first-token latency when verification work shares the GPU with prefill. Include long outputs and short outputs: a setup cost that pays back across 1,000 tokens may lose on an answer of 20 tokens.

SGLang documents several speculative paths with different options and limitations. A model's MTP capability does not establish that every path is supported by every engine release. Start with a documented combination and save the exact options with the release record.[15]

Account for the second footprint

A separate draft has weights and cache. Prediction heads and verification buffers also consume space even when there is no standalone draft model. Recompute Chapter 4's admission limit after warmup. A latency improvement that halves safe concurrency can fail the original demand target.