How to quantize a model yourself, step by step

You can quantize most open-weight language models on your own hardware with free, open-source tools. The route depends on where the model will run. Use llama.cpp's GGUF tools for laptops, Macs and Ollama. Use bitsandbytes for quick 4-bit loading inside Python. Use llm-compressor, or GPTQModel, to produce GPTQ or AWQ checkpoints for vLLM. This guide walks through all three, shows how to run the result, and explains how to check that quality held up.

Choose a route

All three routes start from the original high-precision weights, usually BF16 safetensors from Hugging Face. Gated models need an accepted licence and a logged-in token (hf auth login). Make sure you have enough disk space and RAM. llama.cpp's quantize documentation notes that its tools currently load the whole model into memory. If you are unsure which format you need, read GPTQ vs AWQ vs GGUF first.

Route A: GGUF with llama.cpp

1. Build the tools and install the converter's requirements

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build                  # add -DGGML_CUDA=ON for NVIDIA; Metal is on by default on macOS
cmake --build build --config Release
python3 -m pip install -r requirements.txt

The binaries end up in build/bin/, including llama-quantize, llama-imatrix, llama-perplexity, llama-cli and llama-server.

2. Convert to a 16-bit GGUF

hf download Qwen/Qwen3-8B --local-dir ./Qwen3-8B
python convert_hf_to_gguf.py ./Qwen3-8B --outtype bf16 --outfile qwen3-8b-bf16.gguf

Common --outtype values are f32, f16, bf16, q8_0 and auto, which picks the highest-fidelity 16-bit type. Convert at full 16-bit precision so that every later quant is made from the original values. The converter can also read a Hub repo directly with --remote, which its help text labels experimental.

3. (Optional) Compute an importance matrix

An importance matrix (imatrix) records which weights matter most on representative text. llama-quantize uses it to spend precision where it counts. This matters most for 4-bit and lower types. llama.cpp's perplexity scoreboard for Llama 3 8B shows Q4_K_M with an imatrix scoring lower perplexity and KL divergence than Q4_K_M without one.

./build/bin/llama-imatrix -m qwen3-8b-bf16.gguf -f calibration.txt -o imatrix.gguf -ngl 99

Use a plain-text file that resembles your real workload: prose, code or chat in the languages you use. -ngl 99 offloads the computation to the GPU.

4. Quantize

./build/bin/llama-quantize --imatrix imatrix.gguf qwen3-8b-bf16.gguf qwen3-8b-Q4_K_M.gguf Q4_K_M

Leave out --imatrix for a plain quantization. The type name sets the trade-off. Q8_0 is close to lossless, Q5_K_M and Q4_K_M are the common local defaults, and IQ types cover roughly 2–4.5 bits per weight (see GGUF quant types). Running llama-quantize with no arguments prints its usage, including every type and, for many, a reference size and perplexity change. Avoid --allow-requantize from an already-quantized file, because the tool itself warns this can severely reduce quality.

Route B: 4-bit loading with bitsandbytes

bitsandbytes quantizes weights while Transformers loads them, with no calibration and no separate conversion step. It supports NVIDIA GPUs (NF4/FP4 from Pascal onward), Intel XPU and Gaudi, and CPU.

pip install --upgrade transformers accelerate bitsandbytes

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "Qwen/Qwen3-8B"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # NormalFloat4 from the QLoRA paper
    bnb_4bit_compute_dtype=torch.bfloat16,  # dtype used for the matmuls
    bnb_4bit_use_double_quant=True,         # also quantize the scales
)
model = AutoModelForCausalLM.from_pretrained(
    model_id, quantization_config=bnb_config, device_map="auto", dtype="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
print(f"{model.get_memory_footprint() / 1e9:.2f} GB")

Here is what each setting does:

• bnb_4bit_quant_type="nf4" uses a 4-bit grid shaped for normally distributed weights. Transformers recommends it for training 4-bit base models.
• bnb_4bit_compute_dtype sets the type that weights are dequantized to for each matrix multiply. The default is float32, and bfloat16 is faster.
• bnb_4bit_use_double_quant quantizes the per-block scales themselves, saving about 0.4 bits per parameter.

With a recent bitsandbytes, you can store the result with model.save_pretrained("qwen3-8b-bnb-nf4") or push_to_hub. vLLM can load bitsandbytes checkpoints through its separate vllm-bnb-plugin. In practice, this route is mainly for experiments and QLoRA. GPTQ and AWQ checkpoints are the more common choice for production GPU serving.

Route C: GPTQ or AWQ with llm-compressor

Which tool is current? AutoAWQ's README declares the project officially deprecated, and its GitHub repository was archived in May 2025. vLLM's AutoAWQ page points to llm-compressor's AWQ examples instead, and vLLM's quantization overview suggests llm-compressor as the starting point. On the GPTQ side, Transformers states that AutoGPTQ is no longer supported and directs users to GPT-QModel (gptqmodel), which is actively maintained and also supports AWQ. Use llm-compressor if your target is vLLM, or GPTQModel if you want its API or one of its other backends.

The vLLM docs advise installing llm-compressor and vLLM in separate Python environments. llm-compressor's INT4 documentation lists Ampere, Ada Lovelace and Hopper NVIDIA GPUs for running INT4 models.

pip install llmcompressor

GPTQ (W4A16, group size 128)

from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
oneshot(
    model=model,
    dataset="perfectblend",       # built-in calibration set; your own data works too
    splits="train[:512]",
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=512,
)

save_dir = "Meta-Llama-3-8B-Instruct-W4A16-G128"
model.save_pretrained(save_dir, save_compressed=True)
tokenizer.save_pretrained(save_dir)

AWQ (4-bit asymmetric)

AWQ in llm-compressor is a two-part recipe. AWQModifier computes activation-aware scales, then QuantizationModifier applies the 4-bit weights:

from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.transform.awq import AWQModifier

recipe = [
    AWQModifier(),
    QuantizationModifier(ignore=["lm_head"], scheme="W4A16_ASYM", targets=["Linear"]),
]

Pass this recipe to the same oneshot call. llm-compressor's INT4 guide suggests starting at 512 samples of 2,048 tokens (its AWQ example uses 256 samples of 512 tokens) and formatting calibration text with the model's own chat template. If accuracy drops, increase the sample count or use data closer to your use case. The documented examples leave lm_head unquantized.

Run the quantized model

llama.cpp: llama-cli and llama-server

./build/bin/llama-cli -m qwen3-8b-Q4_K_M.gguf -ngl 99 -c 8192
./build/bin/llama-server -m qwen3-8b-Q4_K_M.gguf -ngl 99 -c 8192 --port 8080

llama-cli opens an interactive chat using the chat template stored in the GGUF. llama-server serves an OpenAI-compatible API, with /v1/chat/completions among its endpoints, plus a built-in web UI. -ngl sets how many layers go to the GPU. In current builds it defaults to auto and accepts all. -c sets the context length. The current server documentation lists 9931 as the default port, so this example sets --port 8080 explicitly to match the evaluation command below.

Ollama: import the GGUF with a Modelfile

Ollama does not quantize GGUF files during import, so point it at the file you just made. Create a file named Modelfile:

FROM ./qwen3-8b-Q4_K_M.gguf
PARAMETER num_ctx 8192

ollama create qwen3-8b-q4km -f Modelfile
ollama run qwen3-8b-q4km

For split GGUF files, use a wildcard such as FROM ./model-*.gguf. Run ollama ps to confirm whether the model loaded fully on the GPU or partly on the CPU. If replies look malformed, add a TEMPLATE block that matches the model's chat format.

vLLM: serve the compressed checkpoint

pip install vllm
vllm serve ./Meta-Llama-3-8B-Instruct-W4A16-G128 --max-model-len 8192

vLLM reads the quantization method from the checkpoint's config, so llm-compressor and GPTQModel outputs need no extra flag. The server listens on port 8000 by default. vLLM can also load GGUF files, but its documentation calls that support highly experimental and recommends passing the original model's tokenizer with --tokenizer.

Evaluate your quant

A quantized model that loads and chats is not necessarily a good one. Run three checks and compare each against the unquantized model.

1. Perplexity and KL divergence (GGUF)

./scripts/get-wikitext-2.sh
./build/bin/llama-perplexity -m qwen3-8b-bf16.gguf   -f wikitext-2-raw/wiki.test.raw -ngl 99
./build/bin/llama-perplexity -m qwen3-8b-Q4_K_M.gguf -f wikitext-2-raw/wiki.test.raw -ngl 99

Lower perplexity is better. Only compare runs that use the same model, tokenizer, text and settings. A small increase over BF16 is normal for 4-bit quants. A large jump means something went wrong. For a sharper measure, save the BF16 logits once with --kl-divergence-base base.kld. Then run the quant with --kl-divergence-base base.kld --kl-divergence to get KL divergence and the percentage of tokens where both models make the same top prediction. The logits file is large: llama.cpp cites 37 GiB for Llama 3 on Wikitext-2. Use --chunks to limit the run.

2. A small task evaluation

EleutherAI's lm-evaluation-harness runs standard benchmarks against Transformers, vLLM or a running llama-server:

pip install "lm_eval[vllm,api]"
lm_eval --model vllm \
  --model_args pretrained=./Meta-Llama-3-8B-Instruct-W4A16-G128,add_bos_token=true \
  --tasks gsm8k --num_fewshot 5 --limit 250 --batch_size auto

lm_eval --model gguf --model_args base_url=http://127.0.0.1:8080 --tasks hellaswag --limit 500

Run the same command against the original model and compare the scores, taking the reported standard error into account. --limit keeps the run short. Small samples are noisy, so treat a difference within the error bars as no difference. Quantized models can be sensitive to the beginning-of-sequence token, which is why the llm-compressor docs pass add_bos_token=true.

3. Side-by-side prompts

Benchmarks miss failures that matter to you. Keep 20–30 prompts from your real workload: long instructions, code, structured output and non-English text if you use it. Run them through both models with greedy decoding (for example --temp 0 in llama-cli, or temperature=0 through the API) and read the outputs side by side. Watch for broken formatting, lost instructions, repetition loops and factual drift. Quantization damage often shows up first in long or precise tasks.