AI Infra Interviews logo

Quantization: measure what FP8, INT8 and AWQ cost

Serve one model in BF16, FP8 and INT8 and a larger one in FP8 and 4-bit AWQ, then measure memory, decode speed and GSM8K accuracy question by question.

PremiumIntermediate

AI Infra Interviews editorial · Updated

How do you measure whether FP8, INT8 or AWQ quantization hurts a model's accuracy?

Serve the BF16 model and each quantized checkpoint with identical arguments, score them on the same full test set, and compare answers question by question rather than only comparing two percentages. Run a second BF16 server first: greedy decoding at high concurrency is not batch-invariant, so some answers change with no quantization at all, and that floor is what a candidate must beat. Predict weight bytes from the config, read the engine's memory lines, and time a fixed output length. Accept a format when its accuracy stays within the measured noise and the speed or memory gain matters for your load.

Read the primary sources
What you build
Thirteen server records, answer journals and speed records, plus a paired report that separates quantization effects from run-to-run noise.
Environment
Linux · Python 3.12 · One NVIDIA GPU with 12 GB · 15 GB disk
Plan your session
2–3 hours plus about 35 minutes of GPU time. A planning estimate, not measured runtime.
Before you begin
Complete the free streaming client project and understand greedy decoding. Use a single-GPU Linux host with the pinned engine environment and about 15 GB of disk for five checkpoints.
Paired GPU experiment executed. 13 fresh servers on an RTX 5070: six arms in a forward and a reverse pass plus a second BF16 server, each scored on all 1,319 GSM8K test questions and timed at one stream and a batch of 32. Weight sizes matched config predictions within 2%. Greedy decoding at concurrency 32 is not batch-invariant, so single-pair significance tests are reported beside same-arm replicates. One task, short outputs and one GPU; long context, other tasks and FP8 KV cache are not measured.
  1. 01Predict the bytes

    Compute each checkpoint's weight bytes and a decode ceiling from config.json before loading anything.

  2. 02Measure every arm

    Score the same 1,319 questions and time a fixed output length in fresh servers, twice, in opposite orders.

  3. 03Defend the decision

    Compare question by question against a BF16 noise floor, then weigh accuracy, speed, memory and answer format.

Follow the changing state through the project. Each boundary has a check in the walkthrough.

Continue with Premium

Premium includes this guide’s complete walkthrough and downloadable project files. The technical overview, prerequisites and verification notes above are public.

View Premium access

Start with Stream vLLM responses with Python and detect incomplete answers (Free to read).

Primary sources

Checked . Source review and execution checks are described separately above.