LLM quantization: the complete practical guide

LLM quantization stores a language model's weights, and sometimes its activations, in fewer bits than the 16 it was trained with, usually 8 or 4. Done well, it cuts memory roughly 2–4× and often speeds up generation, with a quality loss you can measure and usually accept. This guide covers why it works, the memory math, the main method families, and how to pick a format for your hardware.

Why quantize an LLM

There are three reasons, and they compound:

• Memory. The weights have to fit in GPU VRAM, unified memory or RAM. At 16 bits, every billion parameters takes 2 GB.
• Bandwidth. To generate each new token, the model reads essentially all of its weights from memory. The AWQ paper's analysis shows that in FP16 this decoding step has an arithmetic intensity of only about 1 FLOP per byte, far below what GPUs can compute. It is limited by memory, not by compute. Fewer bytes per weight means more tokens per second.
• Cost. A smaller model fits on fewer or cheaper GPUs, or on a laptop, and leaves room for longer contexts and bigger batches.

A rough ceiling for single-stream decoding is memory bandwidth ÷ bytes read per token. At a hypothetical 1,000 GB/s, a 16 GB model can't exceed about 62 tokens/s, while a 4.9 GB version could approach 204. Real throughput is lower.

Core concepts: what gets quantized and when

Weight-only vs weight-and-activation

Weight-only schemes (written W4A16 or W8A16) store weights in low precision and dequantize them on the fly. The matrix multiplications themselves still run in 16-bit. This directly attacks the bandwidth bottleneck, which is why it dominates local and single-user inference.

Weight-and-activation schemes (W8A8, FP8, FP4) quantize both operands so the multiply runs on low-precision tensor cores. That helps compute-bound work such as long prompts and large batches. It is harder, because activations contain large outliers.

PTQ vs QAT

Post-training quantization (PTQ) converts a finished checkpoint. Some variants need nothing extra (round-to-nearest). Others need a small calibration set (GPTQ, AWQ, imatrix). Quantization-aware training (QAT) simulates rounding during training or fine-tuning, so the model learns weights that survive it. QAT usually holds up better at very low bit widths but needs a training pipeline. Most downloadable quantized models are PTQ, though some vendors now ship low-precision checkpoints directly: the gpt-oss models were post-trained with MXFP4 MoE weights. More in PTQ vs QAT.

Granularity

Every format shares a scale across a group of values. Per-tensor scales are cheap but crude. Per-channel or per-group scales, typically every 32–256 weights, follow the local range better. They cost a fraction of a bit per weight, which is why "4-bit" formats land at about 4.1–4.9 bits per weight.

Memory math

The weight footprint is simple:

The VRAM calculator runs these numbers for any model and context.

Measuring quality loss: perplexity and KL divergence

Bit width alone doesn't tell you quality. The two standard measures are:

• Perplexity (PPL) is the exponential of the average negative log-likelihood the model assigns to each true next token in a test text. Lower is better. It is comparable only between versions of the same model with the same tokenizer and test text, never across different models.
• KL divergence (KLD) measures how far the quantized model's full next-token probability distribution has moved from the original model's, averaged over tokens. Zero means identical. It is more sensitive than perplexity, because errors in opposite directions can cancel in PPL but not in KLD.

llama.cpp's llama-perplexity tool can record the FP16 model's output distributions and then report mean KLD, the change in perplexity, and how often both models pick the same top token. On its published Llama 3 8B scoreboard, mean KLD is about 0.0014 for Q8_0 and 0.031 for Q4_K_M, which shows the gap between near-lossless and "good 4-bit". Always follow up with task tests (code, maths, long-context retrieval, your own prompts), where low-bit damage often appears first.

The major method families

Round-to-nearest (RTN)

Divide by a scale, round, and store. No data is needed. Applied to weights at 8 bits with small blocks, it is close to lossless: llama.cpp's Q8_0 is exactly this. At 4 bits and below, plain RTN loses noticeably more quality, and the methods below were built to recover it.

GPTQ

GPTQ (Frantar et al., arXiv 2210.17323) quantizes one layer at a time. It processes weights column by column, using approximate second-order (inverse-Hessian) information from calibration inputs to adjust the remaining weights after each rounding. The paper reports quantizing 175B-parameter models to 3–4 bits in about four GPU hours with little accuracy loss, using 128 random 2,048-token segments of C4 as calibration data.

AWQ

AWQ (Lin et al., arXiv 2306.00978) observes that protecting about 1% of salient weights greatly reduces error, and that you find them by looking at activation magnitudes, not weight magnitudes. Instead of keeping those weights in mixed precision, it scales the salient channels up before quantizing, an equivalent transformation. It needs no backpropagation, and the paper's default setting is 4-bit weights in groups of 128.

bitsandbytes: LLM.int8() and NF4

LLM.int8() (Dettmers et al., arXiv 2208.07339) quantizes matrix multiplications vector-wise to 8 bits. It routes the rare outlier feature dimensions through 16-bit, so more than 99.9% of values are still multiplied in 8-bit. QLoRA (arXiv 2305.14314) introduced NF4, a 4-bit type whose 16 levels sit at quantiles of a normal distribution, with blocks of 64. It also introduced double quantization, which quantizes the scales themselves and saves about 0.37 bits per parameter. Together these let a 65B model be fine-tuned on a single 48 GB GPU. In Hugging Face Transformers you can apply NF4 on the fly:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=cfg)

GGUF k-quants and i-quants

llama.cpp's GGUF formats use blocks with their own scales. K-quants (Q2_K–Q6_K) use 256-weight super-blocks, and the _S/_M/_L mixes give sensitive tensors more bits. I-quants (IQ1_S–IQ4_XS) use lattice codebooks and non-linear levels to reach very low bit rates, guided by an importance matrix computed from calibration text. The GGUF quant types guide covers every type.

SmoothQuant (W8A8)

SmoothQuant (Xiao et al., arXiv 2211.10438) makes INT8 activations practical. Activation outliers sit in a few channels, so it divides each activation channel by a smoothing factor s_j = max|X_j|^α / max|W_j|^1−α and multiplies the matching weights by the same factor. The model computes exactly the same function, but both operands become easy to quantize. The paper's general default is α = 0.5.

FP8

FP8 (Micikevicius et al., arXiv 2209.05433) defines two 8-bit floating-point encodings: E4M3, which favours precision, and E5M2, which favours range. vLLM's FP8 path stores weights as E4M3. Floating-point levels follow the data's magnitude, which makes FP8 W8A8 more forgiving than INT8. FP8 compute needs compute capability 8.9 or higher (Ada Lovelace, Hopper, Blackwell). On Turing and Ampere GPUs, vLLM runs FP8 checkpoints as weight-only W8A16.

MXFP4 and NVFP4

Both formats store 4-bit E2M1 floating-point elements with a shared block scale. MXFP4, from the Open Compute Project's Microscaling (MX) specification, uses blocks of 32 with a power-of-two E8M0 scale: 4 + 8/32 = 4.25 bits per value. NVFP4 uses blocks of 16 with an FP8 E4M3 scale plus a per-tensor FP32 scale, about 4.5 bits per value, and is native to NVIDIA Blackwell. The finer scaling reduces error at a small cost in size. The number formats guide compares the bit layouts.

Choosing a format by hardware

See hardware for local LLMs, GPTQ vs AWQ vs GGUF, and how to quantize a model.

Decision checklist

1. Budget memory first. Weights + KV cache at your target context + about 10–20% headroom must fit in fast memory.
2. Pick the highest precision that fits. 8-bit or FP8 if it fits, a good 4-bit (Q4_K_M, AWQ, GPTQ, NF4) if not, and 3-bit or below only when a larger low-bit model beats a smaller high-bit one on your tasks.
3. Match the runtime. Use GGUF for llama.cpp and its derivatives, AWQ, GPTQ or FP8 for vLLM-style servers, and MLX on Macs if you use that stack.
4. Prefer calibrated methods below 5 bits. GPTQ, AWQ or an imatrix usually beat plain rounding.
5. Measure. Check KLD or perplexity against the 16-bit model, then run your own evaluation prompts.
6. Revisit the KV cache. At long contexts it can rival the weights.