Capacity Planning for Open-Weights Fleets
Planning a fleet for a sparse open-weights model works differently from planning one for a dense model, because memory follows total parameters and throughput follows active parameters, and those now differ by more than twenty times. The sizing goes in one direction only: from a traffic forecast to tokens per second, to replicas at a measured operating point, to GPUs, to racks and kilowatts. Doing it in the other direction, from an available GPU count, produces a fleet that fits the hardware rather than the demand.
TL;DR: Size forward from traffic in six steps and check the answer against cost. Convert the forecast into peak output tokens per second, using the peak-to-mean ratio from your own logs rather than an assumed one. Divide by the measured per-replica throughput at the operating point where latency meets the SLO, which is a number from a concurrency sweep and never a peak. Multiply by a headroom factor for failures and deployment. Multiply by GPUs per replica, which comes from the model's footprint and the parallel degree that divides its head and expert counts. Convert to racks and kilowatts, because that is what a facility can actually accept. Then compute cost per million tokens and compare against an API price, because for many workloads self-hosting a trillion-parameter model does not pay until utilization is high.
The six steps, worked
inputs, from a product forecast and production logs
mean output tokens per second across the day: 4,000
peak-to-mean ratio, measured from logs: 3.2
target: serve peak within the SLO
step 1: peak output tokens per second
4,000 x 3.2 = 12,800 tok/s
step 2: per-replica throughput at the operating point
from a concurrency sweep on the actual model and hardware, the level where p99 TTFT and
per-user output rate both meet the SLO
say a GLM-5.3 replica on 8 x B300 sustains 2,400 aggregate output tok/s there
(this number comes from measurement; assuming it is the single largest error in the chain)
step 3: replicas needed at peak
12,800 / 2,400 = 5.33 -> 6 replicas
step 4: headroom
N+1 for a replica in maintenance or failed: 7 replicas
plus room for a rolling deployment (one replica down at a time is covered by the N+1)
step 5: GPUs
7 replicas x 8 GPUs = 56 GPUs
rounded to whole nodes of 8: 56 GPUs = 7 nodes
step 6: racks and power
at 4 B300-class nodes per rack and over 50 kW per rack:
7 nodes = 2 racks, over 100 kW
sanity: the chain multiplies four estimates, so the answer is a range. The measured
per-replica throughput is the term worth spending real effort on, because a 20% error
there is a 20% error in GPUs and kilowatts and every downstream number
What changes for a sparse model
the decoupling that dense models did not have
a dense 70B in bf16
memory 140 GB, active 140 GB: one number governs both
GLM-5.3 in FP8
memory 760 GB of weights, active about 43 GB per token
ratio 17.7 to 1
Kimi K3 with MXFP4 experts
memory roughly 1,619 GB, active about 63 GB per token
ratio 25.7 to 1
what this means for the fleet
the minimum viable replica is set by MEMORY and is large
the throughput of that replica is set by ACTIVE parameters and is high for its size
so sparse models have a high floor and a good slope: they are expensive to deploy at all
and cheap per token once deployed
sanity: this is why sparse open-weights models suit high-volume serving and suit occasional
internal use very badly, and the crossover is a utilization number rather than an
opinion
The cost check that decides self-hosting
cost per million output tokens, self-hosted
fleet: 7 nodes x 8 GPUs = 56 GPUs at $2.5 per GPU-hour = $140 per hour
delivered at peak: 12,800 tok/s
but the fleet runs at peak only part of the day; at a 40% average utilization of capacity
average delivered = 12,800 x 0.40 = 5,120 tok/s
tokens per hour = 5,120 x 3,600 = 18.4M
cost per million = 140 / 18.4 = $7.60
the same at higher utilization
at 70%: 12,800 x 0.70 x 3,600 = 32.3M tokens/hour, cost per million = 140 / 32.3 = $4.33
at 20%: 9.2M tokens/hour, cost per million = $15.20
sanity: utilization moves the cost per token by more than any tuning flag will, which is why
capacity planning and cost control are the same activity, and why the first question
about self-hosting is what the duty cycle will be rather than what the hardware costs
Cost per Million Tokens covers the comparison against published API pricing. The planning point is that the self-hosted number is a function of utilization and the API number is not, so the two curves cross at a volume, and finding that volume is the analysis.
The inputs worth measuring rather than assuming
- Peak-to-mean ratio. Assume 2 and it is 3.2, and the fleet is a third too small at the worst possible moment.
- Per-replica throughput at the SLO, from a sweep on the real model and hardware. Every other number in the chain is arithmetic; this one is measurement.
- Output length distribution. A shift from 200-token to 800-token answers quadruples decode work with no change in request rate.
- Prefix sharing rate. It decides how much prefill is real work, and it differs enormously between a chat product and a document pipeline.
- Growth rate. The lead time on more GPUs is weeks to months, so the forecast horizon has to exceed the procurement horizon.
What interviewers are listening for
The direction of the derivation. Sizing forward from traffic is the answer; sizing backward from an available GPU count is the failure, and it is common. The second signal is separating memory from throughput, since a candidate who says "we need 8 GPUs for memory and that gives us 2,400 tokens per second, so 6 replicas at peak" has the two-number model that sparse models require. The third is utilization: naming it as the dominant term in cost per token, ahead of any engine flag, is what shows the candidate has owned a fleet rather than a benchmark.
Key takeaways
- Six steps forward: peak tokens per second, per-replica throughput at the SLO, replicas, headroom, GPUs, then racks and kilowatts.
- Memory follows total parameters and throughput follows active ones, and the ratio is 17.7 to 1 for GLM-5.3 and about 25.7 to 1 for Kimi K3.
- Sparse models have a high deployment floor and a low per-token cost, so they suit high volume and suit occasional use badly.
- Per-replica throughput at the operating point is the one measured input; every other step is arithmetic on top of it.
- Cost per million tokens moves from $15.20 at 20 percent utilization to $4.33 at 70 percent on the same fleet, which is a larger effect than any tuning flag.