Checking model quality after quantisation

What quantisation changes

Quantisation stores weights, and sometimes activations or the KV cache, in fewer bits to cut memory, while trying to preserve accuracy. The Hugging Face Transformers documentation lists many methods: some need calibration data, others quantise on the fly, and support differs by hardware. vLLM publishes a compatibility table by GPU generation, so a format that works on one card may not run on another.

Kurtic and colleagues ran over 500,000 evaluations on the Llama 3.1 family. They found 8-bit floating point weights and activations effectively lossless, well-tuned 8-bit integer formats losing 1 to 3 percent, and 4-bit weight-only quantisation more competitive than expected. Those results are averages over academic benchmarks and real-world tasks for one model family. Your model, task and format can behave differently, especially at the edges.

The tests to rerun

Run the original and the quantised model side by side on identical inputs, with the same prompts, decoding settings and scoring code.

• The held-out task set from the original approval, scored per class and on each critical slice, not just overall
• Structured output: parse rate, schema validity and tool-call arguments, where a single wrong token breaks the result
• Your longest real inputs, especially if the KV cache is quantised as well
• Rare categories and less common languages, where small shifts in probability change the answer
• Behaviour checks you run for the original, such as refusals and instruction priority
• An agreement check: the share of cases where both models give the same answer, with every disagreement on a critical slice read by a person

Public benchmarks screen, your tests decide

EleutherAI's lm-evaluation-harness runs dozens of standard benchmarks and can use vLLM as a backend, so comparing original and quantised checkpoints on a few of them is cheap. Treat a clear drop as a reason to stop early. Treat no drop as permission to run your own tests, not as approval. Standard benchmarks rarely contain your document formats, your categories or your output schema.

A worked comparison with cost per GPU

Quantisation pays only if it changes how many GPUs you run or how much each one serves at your latency target. Kurtic and colleagues found that the most cost-efficient format depended on the deployment: 4-bit weights for synchronous serving, 8-bit weights and activations for continuous batching. Measure, do not assume. A hypothetical readout for an extraction model:

• Original, 16-bit, two GPUs per replica: field F1 0.871, valid JSON 99.9 percent, 6 requests per second at target, so 3 per GPU
• 8-bit weights and activations, one GPU per replica: field F1 0.868, valid JSON 99.8 percent, 4 requests per second at target, so 4 per GPU
• Critical slice, bank details on scanned invoices: 97 of 100 correct for both

Capacity per GPU rises by a third, the critical slice is unchanged and the overall F1 drop of 0.003 sits within the 0.005 tolerance agreed before testing. Under those criteria the quantised model goes to a shadow run on live traffic, with the original kept ready for rollback. Had bank details dropped by even two cases, the answer would have been no, whatever the saving.

When the comparison has to be repeated

A quantised checkpoint is a new model. Repeat the comparison after every new fine-tuned version, after changing the quantisation method or its calibration data, and after upgrading the inference server or its kernels, since vLLM lets you choose between kernel backends for quantised layers. Keep the test set, scoring code and results versioned together so the next comparison reuses the same setup instead of rebuilding it.