Capacity Planning for Open-Weights Fleets

TL;DR: Forecast the joint workload, measure a supported replica on that mix, and calculate surviving capacity under the failures you promise to tolerate. Include prompt work, output generation and live state. Use average delivered work for cost and peak demand for capacity. Existing hardware is a valid constraint too: calculate the supported service envelope and state any gap.

Output tokens are only one demand measure

A typical situation: product forecasts 12,800 output tokens/s at peak and asks for a GPU count. The missing information includes prompt lengths, output lengths, arrivals, prefix reuse, modalities, latency objectives and failure policy. Two services can produce the same output volume with very different prefill and cache demands.

The second has twenty times the incoming prompt volume. Actual prefill work depends on reusable prefixes and architecture; live-state demand depends on concurrent sequence lengths and state placement. Preserve their joint distribution, not just separate averages.

Reading config.json to Size a Model You Have Never Run starts the memory estimate. A sparse model may have far fewer selected parameters than resident parameters, but active count is not a throughput formula. Expert union across a batch, kernels, attention, communication and placement matter. Open-Weights Models of 2026 links the pinned artifact sizes; do not substitute an advertised active count for measured HBM traffic.

Derive a candidate, then test its failure states

Serving Benchmarks That Do Not Lie explains how to obtain a sustainable operating point using offered traffic and explicit acceptance criteria. The vLLM benchmark CLI distinguishes request rate from a concurrency cap; a cap that throttles arrivals changes the experiment.

For a synthetic planning example, assume a supported eight-GPU replica sustains 2,400 output tokens/s on the exact target mix while meeting the agreed quality and latency objectives. This is an assumed capacity, not a published GLM or Kimi result.

peak demand = 12,800 output tokens/s
replicas required = ceil(12,800 / 2,400) = 6
six replicas supply 14,400 tokens/s under the stated assumptions
one additional replica gives 7 × 8 = 56 GPUs
sanity: after losing one of seven replicas, six remain above peak demand.

Seven covers one unavailable replica under this model. A failure during a rollout that already removes one replica leaves five, supplying only 12,000 tokens/s. Eight are needed to retain six under those two simultaneous losses. This calculation still assumes routing and surviving replicas maintain the measured service envelope.

Replica failures are not the only domain. Seven one-node replicas placed four and three across two racks lose more than one replica when a rack fails. Nine replicas distributed three per rack retain six after one rack loss, assuming independent infrastructure and adequate surviving network capacity. That illustrates a different requirement, not a universal nine-replica recommendation.

Turn nodes into a facility plan

Use the actual server envelope, rack power/cooling limits and network topology. In a hypothetical layout with four physical node slots per rack, seven nodes need at least two racks by space. If each node budgets 12 kW, their node power totals 84 kW; occupying two racks does not turn that into 100 kW. Add the applicable network, storage and facility overhead separately and check each rack's loading.

Parallel layouts also need implementation validation. A degree that divides an attention-head count is not a universal support rule or proof of best latency. Host memory, recurrent/cache state, temporary buffers and collective communication can change the feasible replica shape.

Keep the cost denominator consistent

Suppose this synthetic fleet serves a daily mean of 4,000 output tokens/s and the stated peak of 12,800. Its peak-to-mean ratio is 3.2. Assume, purely for arithmetic, a GPU charge of $2.50 per GPU-hour:

GPU-only hourly charge = 56 × $2.50 = $140
mean delivered output/hour = 4,000 × 3,600 = 14.4 million tokens
GPU-only cost/million output tokens = $140 / 14.4 = $9.72
mean / forecast peak = 4,000 / 12,800 = 31.25%
mean / installed nominal rate = 4,000 / 16,800 = 23.81%
sanity: these two utilization ratios have different denominators.

Neither ratio is measured GPU activity. Do not replace the 4,000-token mean with an unrelated utilization assumption mid-calculation. Cost per Million Tokens covers input/output mix and additional expenses. Include host, network, storage, operations and reserved capacity in a full comparison; API prices and workload charges need current evidence rather than an assumed universal crossover.

Replan when the operating envelope changes

Measure arrival bursts, cache reuse, output budgets, prompt growth and cold readiness. A twenty-percent drop from 2,400 to 1,920 tokens/s raises the base requirement from six to seven replicas; inverse scaling and integer rounding prevent a simple twenty-percent GPU rule.

The reversal condition: if the workload or failure requirement changes, remeasure and resize. Autoscaling or burst capacity can help only if it becomes usable in time and is available under the relevant failure. Inference Autoscaling and Cold Starts covers that delay. Peak-to-mean ratio alone does not decide whether owning or renting the peak is preferable.

What interviewers are listening for

State the measured envelope and its assumptions, the failure state the fleet survives, and the capacity or admission limit when it cannot. Planning from demand and evaluating an existing GPU budget are both useful when the gap is explicit.

Key takeaways

• The same 12,800 output tokens/s can accompany 12,800 or 256,000 prompt tokens/s in the examples.
• Six assumed 2,400-token/s replicas cover peak; seven cover one unavailable replica, not two.
• At the stated 4,000-token/s mean, the hypothetical 56-GPU charge is $9.72 per million output tokens before other costs.
• Test placement, readiness and workload changes alongside the arithmetic.