FP8 vs INT8 vs AWQ in vLLM: Accuracy and Speed, Measured on One GPU

Quantization promises three things at once: a smaller model, faster decoding and about the same answers. We tested all three on one GPU, with every arm answering the same 1,319 GSM8K test questions, and found that the third promise is the hardest one to measure. Two BF16 servers with identical arguments at temperature 0 disagreed on 57 answers before any quantization was involved.

The short version, on an RTX 5070 with vLLM 0.31.0: FP8 and INT8 versions of Qwen3-1.7B decoded 1.45 to 1.50 times faster than BF16 and left about 1.4 GiB more room for the KV cache. INT8 GPTQ stayed inside the BF16 noise. FP8 averaged 1 to 2 points lower. A 4-bit AWQ version of the larger Qwen3-4B beat the BF16 1.7B model by about 9.6 points while using less memory.

What we measured

Six arms, each in a fresh vLLM server with the same memory fraction: Qwen3-1.7B in BF16, cast to FP8 at load, the publisher's block-scaled FP8 checkpoint and its INT8 GPTQ checkpoint, plus Qwen3-4B in FP8 and in 4-bit AWQ. Every arm ran twice, in a forward and a reverse pass, and we added a second BF16 server to measure noise. Questions used greedy decoding with thinking disabled, 32 in flight. Speed used a fixed 256-token output so every arm did the same work.

The memory you actually save

FP8 is one byte per weight, so a 1.7-billion-parameter model should be 1.7 GB. It was 1.90 GiB (2.04 GB). Quantization converts the linear layers, 1.41 billion parameters here, and these checkpoints keep the tied embedding of 0.31 billion parameters in BF16. That arithmetic, from config.json, predicted every arm's weight size within 2% of what the engine reported. The saving still mattered: at the same memory fraction, the 8-bit arms fit 41 to 42% more KV cache tokens.

Speed: real, but short of the byte ratio

At batch one, a decode step reads every weight, so halving the weight bytes could at best make decoding 1.7 times faster here. We measured 1.45 to 1.50 times. The 8-bit arms reached 66 to 71% of their bandwidth ceiling against 77 to 79% for BF16, because the fixed costs in a step, the KV reads, small kernels and launch gaps, did not shrink with the weights.

Accuracy, and the noise you have to beat

Comparing two percentages hides which answers changed, so we compared every question against the first BF16 server. The second BF16 server, identical in every argument, produced the same text on only 691 of 1,319 questions and flipped 57 answers. Greedy decoding at high concurrency is not batch-invariant: a request shares each step with different neighbours, and the kernels' reduction order follows the batch.

Against that floor, INT8 flipped 91 to 95 answers with no consistent direction. FP8 flipped about 150, with a net loss of 15 to 36. One FP8 comparison reached p = 0.004 on an exact McNemar test, but INT8 compared with itself across the two passes reached p = 0.009. A single pair of runs reads batching noise as signal. The defensible conclusion is that FP8 cost between nothing and about two points on this task, and INT8 showed no measurable cost.

When the bigger 4-bit model wins, and when it does not

Qwen3-4B in AWQ used 2.50 GiB against 3.22 for the BF16 1.7B model, decoded 175 tokens per second on one stream against 150 to 154, and answered 9.6 points more questions correctly. For one user at a time, it is the better deployment.

Under load it lost. At a batch of 32 it produced about 3,500 output tokens per second against 3,840 to 3,960. A big batch shares every weight read, so the 4-bit saving matters less, while the 4B model's linear layers still do 2.6 times the arithmetic per token at 16 bits. It also wrote 13% longer answers, which left it at 8.6 correct answers per second against 11.0 to 11.5. The 4B models also followed the instructed Answer: <number> line only 25 to 37% of the time, against about 60% for the 1.7B arms. A parser expecting that line would have failed more of the more accurate model's answers.

Running it yourself

In vLLM 0.31.0, serve a publisher's quantized checkpoint by name, or cast a BF16 checkpoint at load with --quantization fp8_per_tensor (the older --quantization fp8 spelling still works and logs a deprecation warning). Block-scaled FP8 checkpoints failed to start on our machine until we set VLLM_USE_DEEP_GEMM=0, because that path compiles kernels at startup against a CUDA toolkit the machine did not have. Pin every checkpoint revision, read the Model loading took and KV cache size lines on every start, and keep a second BF16 server in the plan. The full hands-on guide has the scripts, the paired report and the exact environment.

What interviewers ask about it

Expect "how would you decide whether to ship an FP8 model?" A strong answer names the tensor being quantized and predicts its bytes, separates the speed gain from the memory gain, and describes a quality comparison with a noise floor rather than a single benchmark number. The quantization concept covers scales, formats and the arithmetic in more depth.