GGUF quant types explained: legacy, k-quants and i-quants

GGUF is llama.cpp's model file format, and names like Q4_K_M, Q6_K or IQ2_XS describe how its weights were quantized: the number is roughly the bits per weight, "K" marks a k-quant built on 256-weight super-blocks, "IQ" marks a codebook-based i-quant, and _S/_M/_L says how many tensors got extra bits. Q4_K_M is the sensible default for most people. This page explains every type, what it costs in memory, and how to build one yourself.

What GGUF is

GGUF is a binary format for running models with GGML, the tensor library behind llama.cpp, and with other GGML-based programs. It replaced the earlier GGML, GGMF and GGJT formats. The main change was swapping an untyped list of hyperparameters for typed key-value metadata, so new fields can be added without breaking older files. According to the specification, the format aims for:

• Single-file deployment. Architecture, tokenizer, chat template and tensors all live in one file.
• Memory mapping (mmap). Tensor data is aligned so it can be loaded quickly.
• Extensibility. New metadata keys don't break existing readers.

A file starts with the magic bytes GGUF and a version number, followed by the metadata, a tensor index (name, shape, type, offset), and the aligned tensor data. Each tensor carries its own type, so a single file usually mixes several. Norm weights stay in F32, the output layer is often Q6_K, and most other weights use the headline type. The spec's naming convention includes the encoding as a filename component, which is why most files end in something like -Q4_K_M.gguf.

Legacy quants: Q4_0, Q4_1, Q5_0, Q5_1, Q8_0

The original types split each tensor into blocks of 32 weights and round every weight to the nearest level within its block. "_0" types store one FP16 scale per block (w = q × scale). "_1" types also store a minimum (w = q × scale + min), which fits lopsided ranges better. The bits per weight follow directly from the block layout:

Hugging Face's GGUF table labels all of these as legacy methods that are no longer widely used. Q8_0 is the exception in practice. It is the near-lossless option, and convert_hf_to_gguf.py can write it directly. The 4- and 5-bit legacy types have been overtaken: in llama.cpp's Llama 3 8B measurements, Q4_K_S is about the same size as Q4_0 but has clearly lower KL divergence (0.043 vs 0.072).

K-quants: Q2_K to Q6_K and the _S/_M/_L suffix

K-quants, added to llama.cpp in mid-2023, group weights into super-blocks of 256, each split into sub-blocks of 16 or 32. Each sub-block's scale (and, for some types, its minimum) is itself quantized to 4–8 bits, with an FP16 scale (and min) stored once per super-block. Quantizing the scales lets the format use fine-grained ranges without much overhead. The base layouts are:

• Q2_K: 16 × 16 sub-blocks, 4-bit scales and mins, 2.625 bpw.
• Q3_K: 16 × 16, 6-bit scales, 3.4375 bpw.
• Q4_K: 8 × 32, 6-bit scales and mins, 4.5 bpw.
• Q5_K: 8 × 32, 6-bit scales and mins, 5.5 bpw.
• Q6_K: 16 × 16, 8-bit scales, 6.5625 bpw.

The suffix names a mix, not a different block layout. llama.cpp knows that some tensors are more sensitive than others, especially the attention value projection (attn_v) and the feed-forward down projection (ffn_down), and promotes them to a higher type:

• _S (small) leaves almost every tensor at the base type.
• _M (medium) promotes part of the sensitive tensors. In Q4_K_M, the attn_v and ffn_down tensors in the first and last eighth of the layers, plus every third layer in between, use Q6_K. For a 32-layer model that is half of them.
• _L (large), offered only for Q3_K, promotes these tensors further (to Q5_K).

When a k-quant name has no suffix, it means _M: Q3_K, Q4_K and Q5_K are aliases for Q3_K_M, Q4_K_M and Q5_K_M. Q6_K has a single variant.

I-quants and the importance matrix

I-quants (IQ1_S up to IQ4_XS) aim for the best quality at very low bit rates. Instead of evenly spaced levels, most of them encode small groups of weights as entries in a codebook. IQ2_XXS, for example, stores groups of 8 weights as an index into 256 points chosen from the E8 lattice, an idea adapted from the QuIP# research. Signs are stored separately. IQ4_NL and IQ4_XS use 16 non-uniformly spaced levels, clustered closer together near zero where most weights sit.

Almost all low-bit types benefit from an importance matrix (imatrix). To build one, llama-imatrix runs calibration text through the model and records, for each weight column, the average squared activation it receives. llama-quantize then minimises a weighted rounding error, so weights that feed large activations are rounded more carefully. The imatrix also improves k-quants and legacy quants. For the lowest-bit types (the IQ1 and IQ2 families and the smallest IQ3 types) and for Q2_K_S, llama-quantize refuses to run without one.

Bits per weight, file size and how to pick

Each type has two bit counts. The nominal one comes from the block layout. The effective one is the whole file divided by the parameter count, which includes the mixes, the output layer and the unquantized norms. The table below lists effective values that llama.cpp's quantize README reports for Llama 3.1 8B. They are approximate and vary slightly between models.

Plug any of these types into the VRAM calculator to include the KV cache at your context length.

How to pick a quant type

llama.cpp's perplexity README publishes a KL-divergence scoreboard for Llama 3 8B, comparing each quant's output distribution with FP16 (lower is better). Mean KLD was about 0.0014 for Q8_0, 0.0055 for Q6_K, 0.011 for Q5_K_M, 0.031 for Q4_K_M (0.028 with an imatrix), 0.072 for Q4_0 and 0.10 for Q3_K_M. Other models will differ, but the ranking is a useful guide:

• Q4_K_M is the common default. It gives good quality at about 4.9 bpw, and llama.cpp's -hf download option picks it automatically when a repository has it.
• Q5_K_M or Q6_K are for when you have room and care about code, maths or long reasoning.
• Q8_0 is effectively lossless and a good reference to compare other quants against.
• IQ4_XS is about 5% smaller than Q4_K_S. With an imatrix, its KL divergence in the scoreboard was slightly higher (0.036 vs 0.032).
• IQ3 and IQ2 types are for fitting a much larger model into limited memory. A big model at about 3 bpw can beat a small model at 6 bpw, but test it. Below about 3 bpw, quality drops quickly.

Speed depends on the backend. The README's own throughput tables show some low-bit types running slower than their size suggests, so benchmark on your hardware with llama-bench. For comparisons with GPU-native formats, see GPTQ vs AWQ vs GGUF and the LLM quantization guide.

Making your own GGUF with llama.cpp

Build llama.cpp, install the Python requirements, convert the Hugging Face checkpoint to a 16-bit GGUF, then quantize it:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release
python3 -m pip install -r requirements.txt

# 1. Convert a local Hugging Face model folder to BF16 GGUF
python convert_hf_to_gguf.py ./Llama-3.1-8B-Instruct \
    --outfile llama-3.1-8b-instruct-bf16.gguf --outtype bf16

# 2. Quantize to Q4_K_M
./build/bin/llama-quantize llama-3.1-8b-instruct-bf16.gguf \
    llama-3.1-8b-instruct-Q4_K_M.gguf Q4_K_M

For i-quants, or to improve any type, build an importance matrix first:

./build/bin/llama-imatrix -m llama-3.1-8b-instruct-bf16.gguf \
    -f calibration.txt -o imatrix.gguf -ngl 99

./build/bin/llama-quantize --imatrix imatrix.gguf \
    llama-3.1-8b-instruct-bf16.gguf llama-3.1-8b-instruct-IQ3_XXS.gguf IQ3_XXS

convert_hf_to_gguf.py accepts --outtype values f32, f16, bf16, q8_0, tq1_0, tq2_0 and auto. With --remote, it reads safetensors directly from a Hugging Face repo ID. Useful llama-quantize options include --output-tensor-type, --token-embedding-type, --tensor-type for per-tensor overrides, and --pure to turn the k-quant mixes off. Check quality with llama-perplexity using --kl-divergence-base and --kl-divergence. The full walkthrough is in how to quantize a model.