GPTQ vs AWQ vs GGUF vs EXL2: LLM quantization methods compared

GPTQ and AWQ are calibration-based 4-bit methods built for GPU inference, GGUF is llama.cpp's file format for CPUs, Apple silicon and mixed CPU/GPU setups, and EXL2/EXL3 trade portability for precise bitrate control on NVIDIA cards. bitsandbytes and HQQ quantize on the fly without calibration, which makes them the usual choice for fine-tuning, while FP8 is the default serving format on recent data-centre GPUs. This guide explains how each one works and which to pick for your hardware.

Algorithm, file format or runtime?

These names are not the same kind of thing. GPTQ, AWQ and HQQ are algorithms for choosing low-bit weights that keep a model's outputs close to the original. GGUF is a file format whose quantization types are defined by the llama.cpp project. EXL2 and EXL3 are formats tied to the ExLlamaV2 and ExLlamaV3 libraries. bitsandbytes is a library that quantizes at load time. FP8 is a number format that hardware executes natively.

In practice, your hardware decides your runtime, and your runtime decides your method. For the basics of scales, bits and rounding error, start with what quantization is and the broader LLM quantization guide.

How each method works

GPTQ: second-order error correction, layer by layer

GPTQ (Frantar et al., 2022) quantizes one linear layer at a time. It feeds a small calibration set through the model, records the inputs each layer sees, and looks for low-bit weights that minimise the squared difference between the original and quantized layer outputs. After each column of weights is rounded, the remaining unquantized columns are adjusted to absorb the error, using approximate second-order (inverse Hessian) information. Because that Hessian depends only on the layer's inputs, one update serves every row, which keeps the method fast on very large models.

The paper's calibration set was 128 random 2,048-token segments of C4 web text, and it reports quantizing a 175B-parameter model in roughly four GPU hours. Grouping (one scale per 128 consecutive weights) costs about 0.15 extra bits per weight and helps most at 3 bits. Transformers now runs GPTQ through GPT-QModel (AutoGPTQ is no longer supported), with 2, 3, 4 and 8 bits.

AWQ: protect the weights that see large activations

AWQ (Lin et al., MLSys 2024) starts from an observation: around 1% of weights matter far more than the rest, and you find them by looking at activation magnitudes, not weight magnitudes. Keeping those channels in FP16 helps, but mixed-precision storage is awkward for hardware. AWQ instead scales each salient input channel up before quantizing and scales the matching activations down, an equivalent transformation that shrinks the relative rounding error on important weights. The scale is average activation magnitude raised to a power α, chosen by a small grid search.

There is no backpropagation or layer reconstruction, so AWQ uses calibration data only for activation statistics and is less prone to overfitting it. The paper targets weight-only INT4/INT3 with group size 128. Note the tooling shift: the AutoAWQ repository was archived in May 2025, and AWQ support moved into vLLM's llm-compressor; MLX-LM also supports AWQ on Macs.

GGUF: k-quants, i-quants and the importance matrix

A GGUF file is a single binary holding tensors plus key-value metadata, designed for fast memory-mapped loading; it replaced the older GGML, GGMF and GGJT formats. llama.cpp, which reads it, is plain C/C++ with optimised x86 and ARM NEON paths, treats Apple silicon as a first-class target, and has GPU backends including CUDA, HIP, Metal, Vulkan and SYCL. Its defining feature is hybrid inference: you can offload some layers to a GPU (the -ngl option) and run the rest on the CPU, so a model larger than your VRAM still runs.

K-quants group weights into 256-weight super-blocks split into smaller blocks, each with its own quantized scale (and, for "type-1" variants, a minimum). Q4_K, for example, uses eight blocks of 32 weights with 6-bit scales and mins, which works out to 4.5 bits per weight. The suffixes are recipes: in the original k-quants design, Q4_K_M stored half of the attention value and feed-forward down-projection tensors in Q6_K and everything else in Q4_K, spending extra bits where errors hurt most. I-quants (the IQ types) borrow ideas from QuIP#: IQ2_XXS, for example, encodes groups of 8 weights as points on a grid derived from the E8 lattice, which makes sizes near 2 bits per weight usable.

Both families can use an importance matrix (imatrix): run llama-imatrix over calibration text to accumulate squared-activation statistics, then pass the file to llama-quantize --imatrix. It is optional for higher-bit k-quants but effectively required for 2-bit i-quants. Measured on Llama 3.1 8B, Q4_K_M is 4.89 bits per weight, Q6_K 6.56 and Q8_0 8.50. For the full list, see GGUF quant types explained.

EXL2 and EXL3: fractional bitrates for NVIDIA GPUs

EXL2, the ExLlamaV2 format, uses the same optimisation as GPTQ but mixes 2, 3, 4, 5, 6 and 8-bit settings within a model and even within a layer. The converter measures error for several settings per matrix against calibration data and picks the mix that minimises worst-case error at a target average anywhere from 2 to 8 bits per weight. ExLlamaV2 is now archived; its successor ExLlamaV3 uses EXL3, a streamlined variant of QTIP (trellis-coded quantization from Cornell). EXL3 accepts fractional targets such as 3.75 bits per weight and requires CUDA. TabbyAPI is the recommended OpenAI-compatible server.

bitsandbytes: LLM.int8(), NF4 and QLoRA

bitsandbytes quantizes as the model loads, with no calibration data. Its 8-bit mode implements LLM.int8(): matrix multiplications run in INT8 with per-vector scales, while rare outlier feature dimensions (above a default threshold of 6.0) are split out and multiplied in 16-bit. Its 4-bit mode offers FP4 and NF4; the library default is FP4, but the Transformers documentation recommends NF4 when training 4-bit base models.

The 4-bit path is the foundation of QLoRA (Dettmers et al., 2023): freeze an NF4-quantized base model, compute in BF16, and train small low-rank adapters on top. QLoRA adds double quantization of the scale constants and paged optimisers, and the paper reports fine-tuning a 65B model on a single 48 GB GPU.

HQQ: calibration-free, optimisation-based

HQQ (Half-Quadratic Quantization, Mobius Labs, 2023) needs no calibration data. It optimises the zero-point against a sparsity-promoting error between original and dequantized weights, using closed-form updates instead of gradients; the authors report processing Llama-2-70B in under five minutes. Transformers runs HQQ on the fly at 8, 4, 3, 2 and 1 bits, with PEFT support.

FP8: hardware-native 8-bit floating point

FP8 stores weights and usually activations as 8-bit floats, typically E4M3 (maximum magnitude 448). Unlike INT4 weight-only methods, FP8 W8A8 lets tensor cores compute in 8 bits, which helps batched serving. vLLM documents full FP8 compute on GPUs with compute capability 8.9 or higher (Ada Lovelace, Hopper, Blackwell) and a weight-only fallback on Turing and Ampere. Its recommended dynamic scheme uses per-channel weight scales and per-token activation scales computed at run time, so no calibration data is needed. See number formats for AI for how E4M3 compares with BF16 and FP4.

Side-by-side comparison

Which should I use?

A single consumer NVIDIA GPU

If the model fits in VRAM, use EXL3 to tune the bitrate precisely to your card, or GGUF with all layers offloaded for the widest choice of ready-made files and tools. If it does not fit, use GGUF with partial offload; nothing else here degrades as gracefully when VRAM runs out.

A Mac with Apple silicon

Use GGUF through llama.cpp, where Metal is enabled by default on macOS, or Apple's MLX format via MLX-LM. GPTQ and AWQ tooling is optimised mainly for CUDA, and ExLlama requires it, so neither is the natural starting point on a Mac.

CPU only

Use GGUF. Token generation on a CPU is usually limited by memory bandwidth, because every weight is read once per token, so a smaller file generally means faster output. Q4_K_M is the usual starting point; move to Q5_K_M or Q6_K if you have RAM to spare and care about quality.

A multi-user GPU server with vLLM

On Ada, Hopper or Blackwell, start with FP8 W8A8: it halves weight memory versus BF16, needs no calibration in its dynamic form, and uses 8-bit tensor-core math. When memory is tighter, or on Ampere, use INT4 weight-only AWQ or GPTQ checkpoints, which vLLM runs with Marlin kernels. On Blackwell, llm-compressor can also produce NVFP4. Avoid GGUF here: vLLM's documentation calls its GGUF support highly experimental and under-optimised.

Fine-tuning

Use bitsandbytes NF4 with QLoRA, with double quantization and BF16 compute; HQQ is a calibration-free alternative that also works with PEFT. Only the adapters train. A common workflow is to merge the adapter into a 16-bit copy of the base model, then quantize the merged model for deployment.