Serve Kimi K3, a 2.8 trillion parameter model. What is the minimum viable configuration?

TL;DR: The headline number is misleading and the arithmetic explains why. Moonshot released Kimi K3 in July 2026 at 2.8 trillion total parameters with 104 billion active per token, and the mixture-of-experts weights ship in MXFP4, which is 4.25 bits or about 0.53 bytes per parameter rather than two. If roughly 90 percent of parameters are expert weights, the footprint works out near 1,619 GB against 5,600 GB in bf16, which is why the vLLM project's own launch note states it needs at least one eight-GPU B300 node, or a minimum of sixteen B200 or GB200 GPUs on the previous generation. The attention is hybrid: published descriptions give 93 text layers as 69 linear-attention layers with fixed recurrent state and 24 gated latent-attention layers with a growing cache, so only about a quarter of the layers accumulate KV, and the engine needs a cache manager that handles both kinds of state at once. The project reports 111 tokens per second at tensor parallel 8 at batch one, which is roughly 11 percent of the bandwidth bound, and their speculative configuration raises it to 331.

How to approach it

Do the footprint arithmetic from the quantization format rather than from the parameter count, since that is what makes the answer surprising. Then the attention structure and what it demands of the engine. Then the launch configuration from the project's own note. Then interpret the published throughput against the bound, because that number is what a capacity plan needs. Close with what you would verify first.

A strong answer

A typical situation: a team is asked whether a 2.8 trillion parameter model is deployable at all. The instinct is that it needs dozens of GPUs, and the correct answer is one node of the current generation, which changes the conversation entirely.

The footprint:

bytes per parameter, from the released formats
  MXFP4: 4 bits per value plus one 8-bit shared exponent per block of 32
    = 4 + 8/32 = 4.25 bits = 0.53125 bytes
  the non-expert parts (attention projections, routers, shared experts, embeddings) stay at
    higher precision; take 1 byte

the estimate, assuming 90 percent of parameters are expert weights
  experts: 2.8e12 x 0.90 x 0.53125 = 1,339 GB
  other:   2.8e12 x 0.10 x 1.0     =   280 GB
  total                             = 1,619 GB

against the hardware
  8 x B300 at 288 GB = 2,304 GB   -> fits, with 685 GB for KV, activations and overhead
  8 x B200 at 180 GB = 1,440 GB   -> does not fit; 16 gives 2,880 GB, which does
  in bf16 it would be 5,600 GB, needing 24 B300s or three nodes

the check
  the vLLM project's launch note for this model states at least one 8-GPU B300 node, or a
    minimum of 16 B200 or GB200 GPUs
  the estimate reproduces both, which is the confirmation that the 90 percent expert share
    is close to right
sanity: an estimate that independently reproduces the project's published hardware guidance
        is one you can present, and the assumption it rests on is stated rather than hidden

Weight Formats: FP8 Blocks, MXFP4 and AWQ covers the format arithmetic. Multi-Head Latent Attention and Sparse Indexers covers the hybrid attention below.

The attention, and what it demands:

published descriptions of the architecture give
  93 text layers: 69 linear-attention layers and 24 gated latent-attention layers
  interleaved roughly three linear to one full
  hidden 7,168, 896 routed experts with 16 active and 2 shared
  a vision encoder of roughly 401M parameters
  1,048,576 token context

what that does to the cache
  linear-attention layers keep a fixed-size recurrent state that does not grow with context
  only the 24 latent-attention layers accumulate a KV cache
  so KV per token is roughly 24/93 = 26 percent of what an all-latent stack of the same
    depth would cost

what it demands of the engine
  a single cache manager handling paged blocks for the full-attention layers and fixed
    recurrent state for the linear ones, simultaneously
  the vLLM launch note describes exactly this, and notes that prefix caching started
    disabled for this architecture while the hybrid cache design settled, so it has to be
    passed explicitly
sanity: an architecture that saves memory can still be unservable on day one if no engine has
        the cache manager, which is why the support check comes before the sizing

The launch, from the project's own note:

vllm serve moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

why each is there
  --trust-remote-code        the architecture's modelling code ships with the weights
  --load-format fastsafetensors  1,619 GB has to be read from disk on every restart
  --enable-prefix-caching    explicitly, because it starts disabled for this architecture
  --tool-call-parser and --reasoning-parser  model-specific output formats; without them the
                             structured output is not parsed and tools are never called
sanity: the two parser flags are the ones that produce a deployment that runs and is subtly
        wrong, because output is still fluent when they are missing

Interpreting the published throughput:

the bandwidth bound at batch 1 on 8 x B300
  active bytes per token, using the same 85 to 90 percent expert share:
    104e9 x 0.85 x 0.53125 = 47 GB
    104e9 x 0.15 x 1.0     = 16 GB
    total                   = 63 GB
  aggregate bandwidth: 8 x 8 TB/s = 64 TB/s
  bound = 64e12 / 63e9 = about 1,020 tokens/s

what the project measured
  111 tokens/s at tensor parallel 8, 118 at tensor parallel 16, at batch 1
  achieved fraction = 111 / 1,020 = about 11 percent
  with their speculative configuration: 331 at TP8 and 370 at TP16, which they describe as a
    3.14 times improvement

why the gap
  at batch 1 the per-layer work is small against kernel launches, the expert all-to-all and
    sampling, and expert weights are read as many scattered pieces rather than one stream
  speculative decoding amortizes those fixed costs over several tokens, which is why it wins
    so much here rather than a little
sanity: the bound is the ceiling and 11 percent is the batch-1 reality, so a capacity plan
        built on the bound overstates by nine times and one built on the measured number at
        batch 1 understates throughput at production concurrency

Serving Benchmarks That Do Not Lie covers why the bound belongs in every report.

What to verify first on a running replica:

the checks, in order
  the engine's reported model size against the 1,619 GB estimate
    a large discrepancy means the MXFP4 weights were not read as MXFP4
  the engine's reported KV cache size in tokens
    compare against 24 full-attention layers rather than 93, which is the check that the
    hybrid cache manager is doing what it should
  nvidia-smi topo -m
    NVLink between all 8 GPUs, since a tensor-parallel deployment without it runs and is slow
  NCCL_DEBUG=INFO on the first collective
    confirms the detected topology and the all-to-all path the engine chose
  a tool-calling request end to end
    the parser flags produce correct structured output, or tools are silently never called
  a concurrency sweep, not a single-request latency number
    p99 TTFT and aggregate tokens/s at each level, against the SLO
sanity: the KV-size check is the strongest one here, because it fails if either the layer
        composition or the attention design was misread, and both are easy to get wrong on a
        hybrid model

The reversal condition: if the product does not need this model's capability, none of this is worth doing. A 2.8 trillion parameter model occupies an entire eight-GPU node of the newest hardware before serving a single user, so its cost floor is high and its per-token cost is only good at volume. A 320 billion parameter model of the same family, or a 284 billion parameter alternative, fits in a fraction of the hardware and serves the same requests adequately for many products. The question to answer before the sizing is whether the evaluation on your own traffic justifies the floor, and the honest sequence is to measure that on a rented node before committing capacity.

What interviewers probe next

• "Where does the 90 percent expert share come from?" It is an assumption, stated as one, validated by the estimate reproducing the project's published hardware minimum.
• "Why does speculative decoding help so much?" It amortizes fixed per-step costs across several tokens, and those costs dominate at batch one on a sparse model with scattered expert reads.
• "What breaks without the parser flags?" Tool calls are never parsed and reasoning leaks into user-visible output, both of which produce fluent, wrong behaviour rather than errors.
• "How does the hybrid attention change capacity planning?" KV per token is roughly a quarter of an all-full-attention stack, so long-context concurrency is much cheaper than the layer count suggests.

Common mistakes

• Sizing 2.8 trillion parameters at bf16 and concluding it needs three nodes.
• Skipping the engine support check, when a hybrid cache manager is required and was not universally available at release.
• Omitting --enable-prefix-caching because it is on by default for other models.
• Omitting the tool and reasoning parsers, which yields fluent output with tools silently never called.
• Planning capacity from the bandwidth bound, which overstates batch-one throughput by about nine times.

Key takeaways

• MXFP4 at 0.53125 bytes per parameter puts 2.8T at roughly 1,619 GB, which fits one 8-GPU B300 node and matches the project's published minimum.
• The stack is 69 linear-attention layers and 24 full-attention layers, so KV accumulates in about a quarter of them and the engine needs a mixed cache manager.
• The launch needs --trust-remote-code, a fast load format, explicit prefix caching, and the model's tool and reasoning parsers.
• Published measurements give 111 tokens/s at TP8 at batch 1, about 11 percent of the roughly 1,020 tokens/s bandwidth bound.
• Speculative decoding raises that to 331 because it amortizes fixed per-step costs, not because the model got faster.