KV cache quantization: the memory math and how to do it

The KV cache holds the key and value vectors of every token a model has already processed, in every attention layer. It grows linearly with context length and with the number of sequences you serve at once, so at long contexts it can take more memory than the quantized weights. Storing it in 8 bits instead of 16 halves that cost and usually has little effect on output quality. Going down to 4 bits saves more memory, but you need to test the result.

What the KV cache stores

In a decoder-only transformer, each new token's query is compared against the keys of all earlier tokens, and the resulting weights are applied to their values. Those keys and values never change once computed, so engines compute them once per token and keep them. That store is the KV cache.

During prefill, the engine processes the prompt in parallel and writes one entry per token. During decode, each new token appends one entry and reads the entire cache back. So:

• Cache size depends on how many tokens are held, meaning prompt plus generated output, multiplied by the number of concurrent sequences.
• Every decode step streams the whole cache through the memory bus. A large cache costs bandwidth as well as capacity.

The cache holds no weights and no queries. Weight quantization (see the LLM quantization guide) and KV cache quantization are separate settings, and you can combine them freely.

The KV cache memory formula

Every full-attention layer stores one key vector and one value vector per KV head per token. Each vector holds head_dim elements. Multiplying everything out gives:

The leading 2 covers keys plus values. n_{kv_heads} means key/value heads, which can be fewer than query heads (see below). head_dim is usually hidden_size ÷ n_{attention_heads}. bytes_per_element is 2 for FP16/BF16 and 1 for FP8. For llama.cpp's block formats it is about 1.0625 for q8_0 (32 values plus a 16-bit scale = 34 bytes) and 0.5625 for q4_0 (18 bytes per 32 values).

Only tokens and batch vary at run time, so compute the per-token cost first. Both examples use the published Llama 3.1 configurations.

Batch multiplies everything. Eight concurrent 32K-token conversations on the 8B model need 128 KiB × 32,768 × 8 = 32 GiB of FP16 cache, about seven times its 4-bit weights.

How GQA and MQA shrink the cache

Multi-head attention (MHA) gives every query head its own key and value heads. Multi-query attention (MQA) shares one K/V head across all query heads. Grouped-query attention (GQA) shares one K/V head per group of query heads. The cache stores only K/V heads, so their count sets the memory cost.

• Llama 3.1 8B has 32 query heads and 8 KV heads, which is groups of 4. Its cache is one quarter of what MHA would need. With MHA, the 128K figure would be 64 GiB instead of 16 GiB.
• Llama 3.1 70B has 64 query heads and 8 KV heads, which is groups of 8. With MHA, 128K of FP16 cache would take 320 GiB instead of 40 GiB.

The KV head count is fixed in training. At inference, you control context length, batch and bytes per element. Sliding-window layers and latent-attention designs change the formula, so for those models check the engine's startup log instead of applying it blindly.

When context length, not weights, dominates memory

Divide the weight size by the per-token cache cost to find the context length at which the cache equals the model:

• 8B at Q4_K_M: llama.cpp's quantize documentation lists the Llama 3.1 8B Q4_K_M file at 4.58 GiB. 4.58 GiB ÷ 128 KiB ≈ 37,500 tokens. Past about 37K tokens of FP16 cache, the cache is larger than the weights. At 128K, it is 16 GiB against 4.58 GiB.
• 70B at Q4_K_M: the same documentation lists about 43.1 GB (≈ 40.1 GiB). At 128K, the 40 GiB FP16 cache roughly equals the weights.

Speed shows the same effect. Each decode step reads the weights and the whole cache, so at 128K on the 8B model the step reads 16 GiB of cache against under 5 GiB of weights, and generation slows as the conversation grows.

When you hit the memory wall, work through these options in order:

1. Allocate only the context you use. In llama.cpp, set -c explicitly. By default -c is taken from the model, and the default --fit behaviour adjusts unset arguments, including context size, to fit device memory. That can leave you with a smaller window than you expected.
2. Reduce parallel slots or batch size. Each concurrent sequence needs its own cache.
3. Quantize the KV cache to 8 bits.
4. Only then consider a smaller weight quant, or 4-bit KV.

KV cache quantization in practice

llama.cpp

The flags are -ctk/--cache-type-k and -ctv/--cache-type-v. Accepted types are f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1, and both default to f16. A quantized V cache requires Flash Attention. -fa defaults to auto, and in that mode llama.cpp turns Flash Attention on when V is quantized. If you pass -fa off, it refuses to create the context. The requirement applies to V. A quantized K cache on its own is accepted without Flash Attention.

llama-server -m model-Q4_K_M.gguf -c 32768 -fa on -ctk q8_0 -ctv q8_0

Ollama

Ollama exposes one server-wide setting, OLLAMA_KV_CACHE_TYPE, with the values f16 (default), q8_0 and q4_0. It takes effect only when Flash Attention is active. Ollama enables Flash Attention automatically where the backend supports it, and OLLAMA_FLASH_ATTENTION=1 forces it on.

OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

vLLM

vLLM sets the cache type with --kv-cache-dtype (or kv_cache_dtype in Python). The default is auto, which uses the model's own dtype. The documented quantized options are fp8 (the same as fp8_e4m3, on CUDA and ROCm) and fp8_e5m2 (CUDA).

vllm serve meta-llama/Llama-3.1-8B-Instruct --kv-cache-dtype fp8 --max-model-len 32768

Without calibration, vLLM sets every scale to 1.0. Its documentation recommends calibrating scales with llm-compressor, which stores them in the checkpoint. --kv-cache-dtype-skip-layers sliding_window keeps sensitive sliding-window layers at native precision. Because vLLM pre-allocates cache blocks from a fixed share of GPU memory (--gpu-memory-utilization, default 0.92), FP8 roughly doubles the tokens it can hold.

Hugging Face Transformers

Transformers offers a quantized cache per generate call. The default quanto backend supports int2 and int4, and hqq adds int8. The docs warn it can slow short-context generation.

out = model.generate(**inputs, max_new_tokens=256,
                     cache_implementation="quantized",
                     cache_config={"backend": "quanto", "nbits": 4})

Quality considerations

8-bit is the safe default. llama.cpp's q8_0 stores a separate scale for every 32 values, so its rounding error is small compared with FP16. FP8 E4M3 has a narrow dynamic range, so its accuracy depends on good scales, and calibrated scales beat the default of 1.0. Ollama's documentation says q8_0 usually has no noticeable effect on quality.

4-bit is a trade-off you must measure. Errors in the keys change attention scores. With tens of thousands of cached tokens competing for attention, small score errors are more likely to reorder close candidates. Ollama's documentation likewise notes that q4_0 losses may be more noticeable at larger context sizes.

Keys are harder than values. KIVI and KVQuant both quantize keys per channel, because key magnitudes vary strongly between channels. KIVI quantizes values per token. llama.cpp's block formats quantize runs of 32 values within each token's vectors, a per-token grouping that the KIVI analysis found suits values better than keys. If you need to go below 8 bits, try keeping K at q8_0 and dropping only V to q4_0, after checking the CUDA build caveat above.

High GQA ratios amplify error. Each cached KV head serves its whole group of query heads, so its error spreads further. Ollama's documentation notes that high-GQA models may be more sensitive.