Learn quantization, one field at a time
Practical guides with the formulas worked out. Start with the foundations, then follow the track you need: AI models, signals and audio, music timing or physics.
- Foundations
What is quantization? One idea behind AI, audio, music and physics
How mapping values to discrete levels works, the math of its error, and where the same idea shows up, from 4-bit LLMs to atoms.
- AI models
LLM quantization: the complete practical guide
Memory math, method families from GPTQ to NVFP4, quality metrics, and a hardware-first checklist for quantizing LLMs.
- AI models
GGUF quant types explained: legacy, k-quants and i-quants
Decode GGUF names like Q4_K_M and IQ3_XXS, compare bits per weight, and learn to pick and build the right llama.cpp quant.
- AI models
GPTQ vs AWQ vs GGUF vs EXL2: LLM quantization methods compared
How GPTQ, AWQ, GGUF, EXL2/EXL3, bitsandbytes, HQQ and FP8 work, where each runs best, and a scenario-by-scenario guide to picking one.
- AI models
Number formats for AI: from FP32 to FP8, NF4, MXFP4 and ternary
Bit layouts, range and precision of FP32, BF16, FP16, FP8, INT8/INT4, NF4, MX formats, NVFP4 and 1.58-bit ternary, plus which hardware runs them.
- AI models
PTQ vs QAT: post-training quantization and quantization-aware training
How post-training quantization and quantization-aware training differ, how calibration and fake quantization work, and when QAT is worth it.
- AI models
KV cache quantization: the memory math and how to do it
The KV cache formula, worked Llama 3.1 examples at 8K and 128K context, and how to quantize the cache in llama.cpp, Ollama and vLLM.
- AI models
Choosing hardware for local LLMs: memory first, bandwidth second
What fits in 8–80 GB at Q4, Q5 and Q8, why memory bandwidth caps tokens per second, and how GPUs, Apple Silicon and CPUs compare.
- AI models
How to quantize a model yourself, step by step
Quantize an LLM with llama.cpp, bitsandbytes or llm-compressor, run it in llama.cpp, Ollama or vLLM, and check quality before you trust it.
- Signals & audio
ADC quantization error, SQNR and ENOB explained
LSB size, the ±½ LSB error bound, the Δ²/12 noise model, the 6.02N + 1.76 dB derivation, ENOB, oversampling and dither in one place.
- Signals & audio
Audio bit depth and dither: 16-bit, 24-bit, 32-bit float
What 16, 24 and 32-bit float really give you, why dither beats truncation, which dither to use, and how to calculate audio file size.
- Music timing
Quantize in music production: grid, strength, swing and groove
Grid values in milliseconds, quantize strength, swing as a ratio, groove templates, audio quantize and how to tighten timing without losing feel.
- Physics
Quantization in physics: why energy comes in steps
Planck's E = hf, the photoelectric effect, hydrogen levels, particle in a box, quantum numbers and the harmonic oscillator, with worked numbers.
Looking for a term? The glossary defines more than 80 of them. Prefer one page? Print the quantization cheat sheet.