Quantization playground
Choose a signal or a set of neural-network weights, set the bit width and scheme, and watch the quantized values, the error and the signal-to-quantization-noise ratio respond instantly.
Input
Scheme
| Original x | Code q | Dequantized | Error |
|---|
Five experiments worth two minutes
- The 6 dB rule. With the full-scale sine, step from 4 to 8 bits. Measured SQNR climbs about 6 dB per bit and tracks the ideal 6.02N + 1.76 dB line.
- Why outliers hurt LLMs. Choose "LLM-style weights with outliers", 4 bits, per-tensor. Three large weights set the scale, so most weights collapse onto a couple of levels and SQNR drops. Switch to per-group with group size 64: each block gets its own scale and the error falls sharply. This is the idea behind the group-wise scales in GPTQ, AWQ and GGUF.
- Clipping as a trade. Keep the outlier weights, per-tensor, and clip at the 99th percentile. A few weights are now badly wrong, but the rest get finer steps. Whether that is a good trade depends on how important the clipped values are.
- Dither. Pick the quiet sine at 4 bits. Without dither, the error repeats with the signal (distortion). With TPDF dither the error turns into steady noise that no longer tracks the signal — the reason audio engineers dither when reducing bit depth.
- Truncation bias. Switch rounding to "Truncate". The error now sits mostly on one side of zero, a systematic bias that round-to-nearest avoids.
Frequently asked questions
What is SQNR?
Signal-to-quantization-noise ratio: the power of the original signal divided by the power of the quantization error, in decibels. Higher is better. For an ideal quantizer and a full-scale sine wave it is about 6.02 dB per bit plus 1.76 dB.
Why is symmetric quantization common for weights?
Neural-network weights are roughly centred on zero, so a symmetric range wastes little and keeps the zero-point at 0, which makes integer matrix multiplication simpler and faster. Activations after functions such as ReLU are often one-sided, where asymmetric ranges use the levels better.
Does more clipping always help?
No. Clipping reduces rounding error for typical values but creates large errors for the clipped ones. Calibration methods choose the clipping point that minimises a total error measure such as mean squared error or KL divergence.