Weight Formats: FP8 Blocks, MXFP4 and AWQ
Derive the storage cost of low-precision tensors, including scales, zero points and higher-precision exceptions. Distinguish formats from quantization methods, understand fused dequantization, and test whether a smaller checkpoint actually improves serving capacity without unacceptable quality loss.
TL;DR: A format specifies encoded values and metadata; a quantization method chooses how to map weights into that representation. Count value bits, scales, zero points, padding and excluded tensors. Then inspect the execution path: packed low-bit weights can speed up bandwidth-bound inference through fused dequantization even without native multiplication in that format. Neither a smaller checkpoint nor a published evaluation proves the new deployment meets your quality and latency requirements.
One tensor, all its bytes
An engine loads a “four-bit” model, but GPU memory barely falls. Before blaming fragmentation, determine whether it keeps packed weights resident or expands them at load time. The checkpoint's label does not specify its runtime representation.
For complete groups without padding, effective bits per value are:
value bits + (scale bits + zero-point bits) / group size
Real implementations may add permutations, alignment, extra scale levels and tensor headers. A tiled scheme also rounds each matrix dimension to its tile boundary; dividing the total parameter count by tile area can undercount partial tiles.
| Example storage layout | Effective bits/value | Bytes/value |
|---|---|---|
| BF16 | 16 | 2 |
| FP8 plus FP32 scale per 128×128 tile | 8 + 32/16,384 | 1.00024414 |
| MXFP4, E8M0 scale per 32 values | 4 + 8/32 | 0.53125 |
| Symmetric INT4, FP16 scale per 128 values | 4 + 16/128 | 0.515625 |
| INT4, FP16 scale and packed 4-bit zero point per 128 | 4 + 20/128 | 0.51953125 |
The INT4 rows are explicit example layouts, not universal AWQ or GPTQ encodings. Some formats omit zero points; others pack or store them differently. AWQ and GPTQ are quantization methods, and their exported tensor layout determines storage and kernel compatibility. The AWQ implementation demonstrates activation-aware weight quantization and optimized serving kernels.
The OCP microscaling specification defines MXFP4 with E2M1 values, a shared E8M0 scale and groups of 32. A group holds 32 × 4 + 8 = 136 bits, or 17 bytes. Scale overhead is 6.25% relative to the 16-byte value payload, not 6.25% of an original BF16 tensor.
Numerics: FP32, BF16, FP8 and FP4 explains representable values. A shared scale can improve local range, but values sharing that scale still compete for limited precision. Outliers and clipping require evaluation rather than a promise that the format “degrades gracefully.”
Mixed precision changes the whole-model result
Consider a synthetic 70B-weight artifact, with 90% of its values eligible for MXFP4 and 10% kept in BF16:
quantized portion = 70e9 × 0.9 × 0.53125 = 33.46875 GB
BF16 portion = 70e9 × 0.1 × 2 = 14.00000 GB
total 47.46875 GB
If every value were eligible: 70e9 × 0.53125 = 37.1875 GB
All BF16: 70e9 × 2 = 140 GB
The exclusions add 10.28125 GB relative to the all-MXFP4 shortcut. The actual reduction is 2.95× from BF16, not 4×. This example excludes other metadata and runtime allocations; it is not an estimate of a named model.
GB
140 GB
37.19 GB
47.47 GB
Real 2026 releases make this distinction necessary. GLM-5.3 mixes FP8 and higher-precision tensors. Kimi K3 applies MXFP4 to routed experts with exclusions. DeepSeek V4 post-trained artifacts combine FP4 expert weights with FP8 and other tensors even though their quantization config says fp8. Qwen3.8's official BF16 artifacts show that open releases are not all pre-quantized. Open-Weights Models of 2026 links their pinned byte totals and primary configurations.
Three execution paths, different consequences
| Path | Persistent weight storage | Possible outcome |
|---|---|---|
| Native low-bit multiplication | Packed weights and scales | Less traffic and supported low-bit compute; actual shapes and kernels still matter |
| Fused dequantization within a matmul kernel | Packed weights; expanded tiles in registers/shared memory | Less HBM traffic can outweigh conversion work |
| Whole-tensor expansion before execution | Higher-precision weights, perhaps packed copies too | Checkpoint is smaller but GPU savings may disappear; temporary peaks can grow |
A synthetic decode step reading 100 GB at an effective 2 TB/s has a 50 ms transfer lower bound. Packing to 25 GB gives 12.5 ms. If a fused conversion path adds 5 ms on the critical path, the simplified total is 17.5 ms, a 2.86× speedup without native four-bit multiplication. This example assumes bandwidth is the limiting term and ignores unchanged work. At a compute-bound batch size, that conclusion may reverse.
Native support is therefore useful evidence, not the entire decision. Check GPU capability, backend, tile shapes, scaling layout, activation dtype and accumulation. Weight storage, activation precision and KV-cache precision are independent choices.
The release is the reference, not the acceptance test
Begin with a supported released format to establish a reproducible baseline. Still evaluate it on your product tasks: tokenizer/template mistakes, parser differences and backend numerics can break a nominally unchanged release. If you requantize, compare against that baseline using held-out prompts and slice-level quality results.
Measure short and long inputs, code/tool tasks where relevant, clipping or saturation indicators when available, and output-length changes. Do not assume which task degrades first. Also record peak allocated memory and offered-load latency: a candidate can pass quality yet lose throughput through a poor kernel path.
Prefer the smaller representation when it preserves required quality and improves the resource that actually constrains the service. Reverse the choice if conversion cost, lost accuracy, or expanded runtime storage outweighs the gain. Quantization for Inference covers that decision process.
Key takeaways
- MXFP4 is 17 bytes per 32 eligible values; block FP8 and grouped INT4 have different metadata costs.
- AWQ/GPTQ names do not uniquely specify bytes per value or a compatible kernel.
- In the synthetic example, keeping 10% in BF16 raises 37.19 GB to 47.47 GB.
- Fused conversion can save time through reduced HBM traffic; loading expanded weights can erase memory savings.
- Every deployment needs task and capacity evaluation, including one using the authors' released format.