TL;DR: Choose quantization for the resource the workload lacks, then verify the executed kernels and task quality. Weight storage, linear-layer arithmetic and KV-cache precision are separate choices. Smaller payloads can improve fit or performance, but neither a format name nor a peak-rate ratio guarantees a serving speedup.
How to approach it
First identify whether weights, active cache or execution time is limiting the service. Name the model, device and supported backend. Derive the changed payload, follow it through the kernel, then compare quality and latency under the same workload before selecting the candidate.
A strong answer
A checkpoint becomes much smaller, yet users still wait just as long. The smaller file may load faster while leaving the slowest part of request execution unchanged. Quantization for Inference separates three decisions that often get bundled together.
Quantization stores numbers in fewer bits. Each group of small integers carries a scale, a stored multiplier that maps them back to approximate real values, and sometimes a zero point, an offset applied before that multiplication. For weight-only quantization, a fused kernel can unpack or dequantize tiles (convert them back to a wider format for the multiply) as it executes. The stored bit width does not establish the arithmetic format of every operation. For weights-and-activations quantization, supported low-precision matrix operations may also change execution throughput. KV quantization changes retained attention state independently. Check the vLLM supported-method and hardware matrix for the installed backend rather than inferring compatibility from the GPU brand.
Start with payload, not a latency promise. Use a hypothetical tensor set containing 70.6 billion weight values. Assume every value uses the stated storage width and ignore scales, zero points, padding and any tensors retained at higher precision:
BF16 payload = 70.6e9 × 2 bytes = 141.2 GB
8-bit payload = 70.6e9 × 1 byte = 70.6 GB
4-bit payload = 70.6e9 × 0.5 byte = 35.3 GB
sanity: four-bit values occupy one quarter of the BF16 raw payload.
Actual checkpoint and loaded memory can be larger. Do not use 35.3 GB to promise a particular cache budget on an 80 GB card: runtime allocations, activations, graphs and temporary buffers still need space. An unquantized configuration that cannot load on the comparison device is not a valid single-device latency baseline.
A separate illustrative cache with 80 layers, eight KV heads, head width 128 and two-byte values uses 320 KiB per position. One-byte values would use 160 KiB of raw payload. At 4,096 positions, that changes 1.25 GiB to 0.625 GiB, before scales and allocation overhead. The KV Cache explains the geometry. Weight-only quantization leaves those cache bytes unchanged unless the cache format also changes.
Measure the changed boundary. If a feasible configuration reads 16 GB of weight payload through an assumed effective 2 TB/s path, the counted transfer takes 8 ms. Halving those bytes makes that term 4 ms. It does not halve an entire request that also spends time on attention, communication or queueing. The Roofline Model helps form this hypothesis; actual timing decides whether the saved traffic was on the critical path.
| Observed constraint | Candidate experiment |
|---|---|
| Weights prevent a useful deployment from fitting | Supported weight-only low-bit method |
| Linear kernels limit throughput | Supported weights-and-activations method |
| Growing attention state limits admission | Supported cache quantization |
| Queueing or transport dominates | Fix that path before crediting quantization |
AWQ and GPTQ are weight-quantization approaches with different calibration/optimization procedures. Do not rank them by the label alone. The AWQ work uses activation information when protecting salient weights. Representative calibration data and the deployed kernel both matter. A weight-only method can still help at larger batches when memory or traffic remains important; claiming it gives nothing beyond a fixed batch size is too strong.
Build the acceptance test before choosing the fastest run. Compare against the intended reference checkpoint using the same prompts, generation settings and scoring rules. Include long-context tasks, structured output and important rare categories. Set allowable regressions from the product requirement; there is no universal one-percent threshold. Record uncertainty and sample counts when scores are close.
Then run a length and offered-load sweep on the actual deployment. Measure per-rank peak memory, qualified completions, client TTFT and p99 token gaps. Include cache growth and a cold start. A format that passes a small warm test can fail at a larger context or a different kernel shape. Repeated tokens or malformed output deserve investigation, but neither symptom uniquely diagnoses quantization.
Keep the candidate only if it improves the required service objective and passes quality checks. Reverse the choice when quality falls outside the agreed bound, backend conversion erases the performance benefit, or the new layout leaves inadequate memory headroom. Numerics: FP32, BF16, FP8 and FP4 explains representation; a format's range and precision alone do not choose the model's scaling scheme.
What interviewers probe next
- "Does W4 mean four-bit compute?" Not necessarily. Inspect the execution path, including unpacking, accumulation and higher-precision operations.
- "Why not quantize the cache too?" It changes another state path and needs separate quality/backend validation; compose it after measuring each change.
- "Can perplexity certify shipping?" It is one signal. Task correctness, long-context behavior and required output structure need their own checks.
- "What if a new format has twice the peak FLOPS?" Determine whether the workload uses those operations efficiently and whether another resource limits the request.
Common mistakes
- Reporting ideal weight-byte savings as end-to-end speedup.
- Omitting scales, higher-precision tensors and workspace from memory.
- Comparing different resource budgets or a baseline that does not fit.
- Using one aggregate quality score to hide a failing task category.
Key takeaways
- The 70.6-billion-value exercise gives 141.2, 70.6 and 35.3 GB raw payloads.
- Weight, execution and cache precision are independent choices.
- Halving a counted transfer term does not halve every latency component.
- Select with task quality and workload-matched measurements on supported kernels.