TL;DR: Kimi K3's pinned checkpoint contains 1.561 TB of stored tensors. Its routed experts use MXFP4, while other components retain higher precision. vLLM's July 27, 2026 launch guidance gives eight GPUs or sixteen /GB200 GPUs as NVIDIA starting configurations. Treat that as supported starting hardware, not proof of any concurrency or latency SLO. Budget hybrid attention state, verify model-specific parsing, and reproduce performance on the intended topology.
How to approach it
The product team wants a 2.8T-parameter model on one node. Start by checking whether it improves the task enough to justify the hardware. Then work through what the team needs from the deployment: it must load, produce correct outputs, serve the expected traffic and recover from failure. A minimum configuration that only passes the first check is useful for evaluation, but it is not yet production capacity.
A strong answer
The model card reports 2.8T total and 104B active parameters, with 16 of 896 routed experts selected per token. The pinned tensor index, checked September 7, 2026, reports 1,560,860,324,864 bytes. This is more useful for initial capacity than multiplying a rounded headline count by an assumed average precision.
This is a mixture-of-experts model: each token passes through 16 of the 896 expert blocks, so only 104B parameters do work for any one token, but all 2.8T must be stored and loaded. MXFP4 is a 4-bit floating-point format in which every group of 32 values shares one 8-bit scale:
MXFP4 packing for an eligible tensor
32 values × 4 bits + 8-bit shared scale = 136 bits = 17 bytes
17 / 32 = 0.53125 bytes per value
Why not apply that to all 2.8T parameters?
The config excludes attention, shared experts, embeddings/output-related
components and vision components from this expert quantization path.
Use actual tensor bytes; do not invent a 90% expert share.
Illustrative aggregate memory comparison, using nominal decimal capacities
stored tensors: 1,560.86 GB
8 × 288 GB = 2,304 GB: 743.14 GB remainder
8 × 180 GB = 1,440 GB: -120.86 GB remainder
16 × 180 GB = 2,880 GB: 1,319.14 GB remainder
The quantization configuration specifies four-bit groups of 32 and higher-precision exclusions. Weight Formats: FP8 Blocks, MXFP4 and AWQ explains scale overhead. The negative remainder rules out the eight-by-180 GB resident-weight example before cache. A positive remainder is only an aggregate lower-bound check; use actual GPU-reported capacity and the largest rank's allocation to decide fit.
GB
1,440 GB
1,560.9 GB
2,304 GB
The architecture has 69 KDA layers and 24 gated MLA layers. Growing KV is only part of its request state. Recurrent and convolution states occupy memory even at short context; retained snapshots add more. Dividing 24 by 93 gives the fraction of layers with growing attention state, not a promise that total cache memory falls to 26%. Multi-Head Latent Attention and Sparse Indexers covers the underlying mechanism.
For example, suppose an engine retained an additional 100 MiB snapshot per reusable prefix. A thousand retained prefixes would consume about 97.7 GiB beyond their other state. This is a synthetic sensitivity check, not Kimi's measured snapshot size. Measure snapshot bytes and hit rate together; keeping more boundaries can save prefill while evicting useful capacity.
The vLLM launch note provides the hardware guidance above and links exact container recipes. Its quick start uses TP8, explicit prefix caching and Kimi-specific tool/reasoning parsers. Pin the recipe image and model revision; review any repository code before enabling remote-code execution. We have not run this multi-terabyte GPU deployment here.
| Acceptance check | Evidence required |
|---|---|
| Load and memory | Per-rank stored/loaded weight difference, peak initialization memory, all cache groups and workspaces |
| Hybrid prefix reuse | Identical-prefix requests plus divergent suffixes; verify reuse and output correctness across recurrent state |
| Tool contract | Expected tool name and schema, empty/malformed results, retry and fallback behavior |
| Capacity | Offered load, prompt/output distributions, completed requests, p99 TTFT and token gaps |
| Recovery | Cold load, readiness, an interrupted stream and capacity while a replica is unavailable |
The launch note's batch-one measurements are on : 111 to 331 tokens/s at TP8, and 118 to 370 at TP16 with DSpark. The ratios are 2.98× and 3.14×, respectively. These are published results, not measurements from this site or a generic B300 node. Serving Benchmarks That Do Not Lie explains why workload and topology belong beside the number.
Do not divide aggregate peak HBM bandwidth by 104B × guessed bytes and call the gap “utilization.” Routed weights need not be evenly read from every GPU; batch reuse, cache reads, kernel overhead and communication change the bound. Speculation also spends draft and verification work. At production concurrency it can lose the capacity advantage it showed for one stream.
The reversal condition: choose a smaller model if it passes the same quality evaluation and meets the SLO more economically. If Kimi wins that evaluation but this layout misses the SLO, test a different supported topology before treating the published minimum as a production design.
What interviewers probe next
- “Does matching the published minimum validate a 90% expert assumption?” No. Many incorrect byte estimates fall between the same two hardware capacities. The tensor index provides independent evidence.
- “Does enabling the tool parser prove tools work?” No. Test the complete request/schema/output path, including malformed or empty tool calls. A parser flag cannot guarantee model behavior.
- “What changes when context grows?” Growing attention state rises with positions, while current recurrent state is bounded per sequence. Retained snapshots and concurrency add separate costs.
- “What could make loading exceed stored bytes?” Conversion, alignment, replicated components, temporary buffers, and backend-specific layouts. Investigate the difference rather than assuming the quantization failed.
Common mistakes
- Treating every parameter as MXFP4, or assuming the higher-precision remainder is one byte per value.
- Presenting a positive aggregate remainder as available cache on every rank.
- Calling both published speedups 3.14×, or transferring NVL72 results to a different topology.
- Treating an eight-GPU starting recipe as demonstrated long-context production capacity.
Key takeaways
- Start from the pinned 1.561 TB tensor payload, then check the largest rank after loading.
- A positive aggregate memory remainder does not establish long-context capacity.
- Validate hybrid cache reuse, request parsing and latency under the intended traffic before adopting a launch recipe.