Reading config.json to Size a Model You Have Never Run
Read a model configuration as an architecture description, then combine it with tensor metadata and engine allocation rules. Learn which fields determine attention state, why mixed precision needs tensor-level accounting, and how to turn a first estimate into a testable per-GPU capacity plan.
TL;DR: A configuration describes tensor shapes and architectural choices. It does not fully specify checkpoint bytes, runtime allocations, valid distributed layouts, or throughput. Read it alongside the tensor index and the selected engine implementation. Derive each attention component separately, state the workload, then check the largest GPU allocation. Treat the result as a candidate to test, not a guaranteed minimum GPU count.
What to open before downloading weights
A new model appears and a team asks whether it fits existing hardware. Start with its exact repository revision, model card, config.json, tensor index, and supported engine recipe. Reading these small metadata files avoids downloading terabytes for a combination the engine cannot load. Some indexes are large themselves; a Git LFS pointer is not the resolved JSON.
| Evidence | What it establishes | What it does not establish |
|---|---|---|
| Model card | Intended tasks, published counts and evaluation conditions | Performance on your prompts |
| Architecture and configuration | Layer types, widths, expert routing, context configuration | Exact physical cache placement |
| Tensor index and tensor headers | Stored tensor names, shapes/dtypes where available, byte totals | Peak loading or steady-state GPU allocation |
| Engine implementation and recipe | Supported attention, packing, kernels and distributed layouts | Your workload's sustainable capacity |
| Measured allocation and load sweep | Fit and performance under stated conditions | Every possible context/concurrency combination |
Do not execute downloaded Python merely to inspect a configuration. Parse the JSON first. If serving needs custom code, review and pin it with the rest of the runtime.
Resolve the architecture before choosing a formula
A multimodal model may put language fields under text_config. A hybrid model may provide layer_types; multiplying its full layer count by a conventional KV formula is wrong. Field names can survive architecture changes for compatibility. num_key_value_heads: 1 does not prove a model uses ordinary multi-query attention.
| Fields to inspect together | Capacity implication |
|---|---|
architectures, model_type, nested text_config | Select the implementation whose fields you are interpreting |
layer_types, num_hidden_layers, prediction/vision configuration | Count each main, recurrent, auxiliary and modality component |
| KV heads, actual K/V widths, cache dtype | Conventional attention cache per retained position |
kv_lora_rank, rotary width, backend cache layout | Possible compressed-latent representation; verify retained tensors |
| Expert counts, selected experts, expert widths | Resident expert capacity versus routed work |
Quantization groups, scales, exclusions, expert_dtype | Mixed storage formats and kernel requirements |
| Context/rotary configuration, sliding windows | Supported position range and retained-state policy |
For example, GLM-5.3 exposes head_dim: 192 but also qk_head_dim: 256 and v_head_dim: 256. Its compressed cache needs a different derivation altogether. Reading one familiar field and ignoring the implementation produces a precise-looking wrong answer. See its pinned configuration.
A reproducible partial calculation
The Qwen3.8-27B configuration, checked September 7, 2026, has 16 full-attention and 48 linear-attention layers. Save that JSON as config.json. This Python computes only its BF16 full-attention KV component, before sharding, padding, recurrent state and prediction state:
import json
from pathlib import Path
config = json.loads(Path("config.json").read_text())
text = config.get("text_config", config)
assert text["model_type"] == "qwen3_5_text"
assert len(text["layer_types"]) == text["num_hidden_layers"]
full_layers = text["layer_types"].count("full_attention")
cache_bytes = 2 # Explicit BF16 cache assumption.
per_position = (
full_layers * 2 * text["num_key_value_heads"]
* text["head_dim"] * cache_bytes
)
print(full_layers, per_position, per_position * 32768)
Executed against that configuration, the output is 16 65536 2147483648: 64 KiB per position and 2 GiB at 32,768 positions. An all-64-layer calculation would give 8 GiB for this component. Recurrent layers still need their own state, and a retained prefix snapshot adds another allocation. KV Cache Sizing and Multi-Head Latent Attention and Sparse Indexers explain those terms.
GiB in BF16
2 GiB
8 GiB
Count expert parameters without guessing the whole model
For a bias-free gated MLP with three matrices, input/output width h and intermediate width m, one expert has 3hm weights. In a synthetic layer with h=4,096, m=1,024, 64 routed experts and 2 selected, the resident routed bank has 805,306,368 weights, while one token uses 25,165,824 routed weights. Add shared experts, attention, embeddings and auxiliary components separately.
That 32-to-1 ratio does not imply 32 times the throughput. A batch can activate many experts and reuse their weights, and communication can dominate. Nor does hidden_size always equal expert input width: latent-MoE architectures introduce another projection. Use actual tensor shapes rather than extending this synthetic formula blindly.
Convert the estimate into a placement decision
For each rank, sum loaded weights, persistent request state, retained shared state, and peak simultaneous workspaces. Replace a blanket 15% allowance with separately stated estimates, then measurements. Weight Formats: FP8 Blocks, MXFP4 and AWQ explains why index bytes and loaded bytes can differ.
Validate the chosen tensor/expert/pipeline degrees against the engine. Divisibility can constrain a layout; an unsupported degree may fail rather than silently pad. Other backends can replicate or distribute components differently, so head count is not a universal GPU-count rule.
Start with the smallest supported candidate that fits the required workload under your stated assumptions. Reverse that choice if allocation profiling or the load sweep fails it. The useful interview answer names the uncertain term and the measurement that will settle it.
Key takeaways
- Configuration, stored tensors and runtime allocations are three different sources of evidence.
- The Qwen example yields 64 KiB per position for one specified component, not the whole hybrid cache.
- A synthetic 64-expert, top-2 layer stores 32 times as many routed weights as one token selects; that is not a throughput ratio.
- Fit is a per-rank constraint under a workload. No constant overhead percentage or universal divisor rule completes the calculation.