What is quantization? A cross-domain guide

Quantization replaces a value that could be anything within a range with the nearest member of a small, fixed set of allowed values, and digital audio, 4-bit language models and snapped MIDI notes all depend on it. Physics uses the word for something deeper: some systems can only have certain energies. Here is the core math, a worked example, and how each field applies it.

What quantization means

A quantizer takes an input that is continuous, or far finer than you want to store, and outputs one of L allowed levels. You store only the integer index of the level, which costs log₂(L) bits: 16 levels need 4 bits, 256 need 8, and 65,536 need 16.

The loss is permanent. Many inputs map to the same level, and the index can't tell you which one it came from. You accept that in exchange for a compact, fixed-size representation. The design question is how many levels you need, and where to put them, so the loss doesn't matter for the task.

Sampling vs quantization. Sampling makes time discrete. Quantization makes the value discrete. Digital audio needs both, while a neural network's weights need only quantization.
Uniform vs non-uniform. Uniform quantizers space their levels equally. Floating-point formats and QLoRA's NF4 put levels where values are most common. See number formats.

The math of uniform quantization

Step size, rounding and clipping

A uniform quantizer has a step size Δ (the gap between levels) and a range. With b bits there are 2^b levels. To quantize a value, divide by Δ, round to the nearest integer, and clamp to the representable range:

q = clamp(round(x / Δ), q_min, q_max) → x̂ = Δ · q

There are two kinds of error. Granular (rounding) error is never more than Δ/2. Clipping (overload) error happens when an input lies outside the range and is pinned to the end level, and it has no bound. A wider range avoids clipping but enlarges Δ for every other value. Choosing the range is called calibration in machine learning and gain staging in audio, and the trade-off is the same.

Scale and zero-point

Machine-learning frameworks call Δ the scale S and add an integer zero-point Z, so that unsigned codes can cover an interval that isn't centred on zero. This is the affine scheme r = S(q − Z) used by Jacob et al. for integer-only inference:

S = (x_max − x_min) / (2^b − 1) Z = round(−x_min / S)

q = clamp(round(x / S) + Z, 0, 2^b − 1) x̂ = S · (q − Z)

Because Z is an integer, real zero always comes back as exactly 0. Padding, ReLU outputs and pruned weights depend on that. A symmetric quantizer sets Z = 0 and uses a range of ±max|x|, which is simpler to compute but wastes levels when the data is lopsided.

The error model: why Δ²/12

If the input moves across many levels and isn't locked to the grid, the rounding error e = x̂ − x behaves like a variable spread evenly over [−Δ/2, +Δ/2]:

σ_e² = (1/Δ) ∫_−Δ/2^+Δ/2 e² de = Δ² / 12 RMS error = Δ / √12

Each extra bit halves Δ, which cuts noise power to a quarter, a drop of about 6.02 dB. For a full-scale sine wave, an ideal N-bit quantizer achieves:

SQNR ≈ 6.02 · N + 1.76 dB

This assumes a full-scale sine input, with noise measured from DC up to half the sample rate. Gray and Neuhoff call the per-bit gain the "6-dB-per-bit rule."

The model breaks down when the signal is small relative to Δ, nearly constant, or repeats in step with the sampling. The error then tracks the signal and shows up as distortion instead of benign noise.

Worked example: seven numbers to 4 bits

Take −1.37, −0.42, 0.00, 0.58, 1.91, 2.74 and an outlier, 4.20. Calibrate the range to [−1.5, 3.0], deliberately leaving the outlier outside it. Four bits give codes 0–15.

S = 4.5 / 15 = 0.3. Z = round(1.5 / 0.3) = 5.

4-bit affine quantization with S = 0.3, Z = 5
x	x / S	Rounded	q (0–15)	x̂ = S(q − Z)	Error
−1.37	−4.567	−5	0	−1.50	−0.13
−0.42	−1.400	−1	4	−0.30	+0.12
0.00	0.000	0	5	0.00	0.00
0.58	1.933	2	7	0.60	+0.02
1.91	6.367	6	11	1.80	−0.11
2.74	9.133	9	14	2.70	−0.04
4.20	14.000	14	19 → 15	3.00	−1.20

All six in-range errors are within Δ/2 = 0.15. Their RMS is about 0.087, close to the theoretical Δ/√12 ≈ 0.0866. With so few samples, the closeness is partly luck.
Zero maps to code 5 and comes back exactly.
The outlier is clipped to 3.00, an error of −1.20, eight times the worst rounding error.
Widening the range to [−1.5, 4.2] removes the clipping, but S rises to 0.38 and the expected RMS error on the other values rises to about 0.11. One extreme value costs every other value some accuracy. LLM methods such as LLM.int8(), SmoothQuant and AWQ are built around this outlier problem.

The scales take storage too, so real formats share one scale across a block. In llama.cpp's Q4_0 format, 32 four-bit codes share one 16-bit scale: (32 × 4 + 16) / 32 = 4.5 bits per weight.

Try it: Quantization playground

Quantize a signal or a weight distribution to 1–16 bits and watch the error, histogram and SQNR change live.

Open the tool

The quantization playground shows the same trade-off on a waveform, with live error and SQNR.

Quantization in AI models

A model with 8 billion parameters stored in 16-bit format takes 8 × 10⁹ × 2 bytes = 16 GB. At 4 bits it takes about 4 GB, plus the scales. Token generation is usually limited by memory bandwidth, so smaller weights often mean faster output as well. The VRAM calculator does this arithmetic for real models.

Model quantization changes the textbook recipe in four ways:

Granularity. Scales are kept per output channel or per block of 32–256 weights, not one per tensor.
Outliers. The LLM.int8() paper found that a few feature dimensions have systematically huge activations. It computes those dimensions in 16-bit and everything else in 8-bit.
Objective. The aim is to keep the model's outputs unchanged, not to minimise per-weight error. GPTQ chooses its roundings to minimise each layer's output error on calibration data.
Timing. Post-training quantization (PTQ) converts a finished model. Quantization-aware training (QAT) simulates rounding during training. See PTQ vs QAT.

Read the LLM quantization guide for the full picture, GGUF quant types to decode names like Q4_K_M, and KV cache quantization for the attention cache.

One clear explainer a week

New tools, guides and formats across AI, signals, audio and physics. No spam; unsubscribe any time.

Signals, ADCs and audio bit depth

An analog-to-digital converter (ADC) samples a voltage and quantizes each sample to an N-bit code. Its step is one least significant bit (LSB), about the full-scale range divided by 2^N. Here the Δ/√12 model is physical: it sets the noise floor of an ideal converter.

16-bit (the CD format, 44.1 kHz stereo): 6.02 × 16 + 1.76 ≈ 98 dB.
24-bit: ≈ 146 dB, more than any real analog circuit can deliver. That's why converters are rated by effective number of bits, ENOB = (SINAD − 1.76 dB) / 6.02, which uses the measured noise and distortion.
Uncompressed CD audio runs at 44,100 × 16 × 2 = 1,411,200 bits/s.

When you reduce bit depth, for example from a 24-bit mix to a 16-bit master, you add a little noise first. This is called dither. It breaks the link between error and signal, so quiet fades end in smooth hiss instead of grit. Use the ADC calculator for SNR, ENOB and file sizes. Read ADC quantization error and bit depth and dither for more.

Music: quantizing time to a grid

In a DAW, quantize moves MIDI notes to the nearest line of a rhythmic grid. It's the same rounding, applied to time, with the grid spacing as Δ. At 120 BPM a quarter note lasts 0.5 s, so sixteenth-note lines fall every 0.125 s. A note played at 1.070 s is 0.070 s from the line at 1.000 s and 0.055 s from the line at 1.125 s, so it snaps to 1.125 s. No note moves more than half a step, 62.5 ms, which is the Δ/2 bound again.

Musicians rarely want full snapping, because it removes human feel. DAWs therefore offer a strength or amount control, which moves each note only part of the way:

t′ = t + a · (g − t)

t is the played time, g is the nearest grid line, and a runs from 0 to 1. With a = 0.5, notes move halfway.

Swing delays every second grid line, which reshapes the grid itself. Quantizing music saves no storage. It's purely a timing tool. Try the music quantizer and the music quantize guide.

Physics: when nature does the quantizing

In quantum physics, quantization isn't an engineering choice. It's how bound systems behave: an electron in an atom can only have certain energies. Light carries energy in packets of E = hf, and the SI fixes the Planck constant h at exactly 6.626 070 15 × 10⁻³⁴ J s. In the Bohr model of hydrogen:

E_n = −13.6 eV / n², n = 1, 2, 3, …

A drop from n = 3 to n = 2 releases 13.6 × (1/4 − 1/9) ≈ 1.89 eV. Converting with hc ≈ 1239.84 eV·nm gives λ ≈ 1239.84 / 1.889 ≈ 656 nm, hydrogen's red line. A particle confined to a box of width L has levels E_n = n²h² / (8mL²). Its lowest level is not zero, and the gaps grow as n increases.

Both meanings share an image: a ladder of allowed values. The differences matter more:

Engineering quantizers round away values that really exist, and that creates error. A bound quantum system simply has no in-between energies, so nothing is rounded.
Engineers choose their levels. Physical levels follow from the equations and boundary conditions, and they are rarely evenly spaced.

Compute levels and wavelengths with the energy levels tool, or read quantization in physics.

Key takeaways

Quantization maps values onto a finite set of levels, and the loss can't be reversed. b bits give 2^b levels.
Rounding error stays within ±Δ/2. Clipping error has no bound, so choosing the range is a trade-off.
Under the usual assumptions, error variance is Δ²/12, so each bit adds about 6 dB of SQNR.
Scale and zero-point let unsigned integers cover any range while keeping zero exact.
Outliers drive most of the design of modern LLM quantization.
Music quantizes time, often at partial strength. In physics, quantization is a property of nature and carries no error.

Frequently asked questions

Is quantization the same as sampling?

No. Sampling makes time discrete. Quantization makes values discrete. Digital audio uses both, while model weights only need quantization.

Why is quantization noise Δ²/12?

If the rounding error is spread evenly between −Δ/2 and +Δ/2, the mean of its square works out to Δ²/12. That holds for busy signals that cross many levels. It fails for tiny, constant or grid-locked signals.

What does "4-bit" mean for an AI model?

Each weight is stored as one of 16 integer codes, and the codes in a block share a higher-precision scale. With the scales included, practical 4-bit formats use roughly 4.1–5 bits per weight, depending on block size and scale format.

How many decibels does each extra bit add?

About 6.02 dB. An ideal N-bit quantizer with a full-scale sine input reaches about 6.02N + 1.76 dB: roughly 98 dB at 16 bits.

Sources and further reading

Gray & Neuhoff, "Quantization," IEEE Trans. Information Theory 44(6), 1998 — the standard survey, including the high-resolution error model.
Analog Devices MT-001: the formula SNR = 6.02N + 1.76 dB — derivation of q/√12 and the assumptions behind it.
Analog Devices MT-003: SINAD, ENOB, SNR, THD and SFDR — the definition of ENOB.
Jacob et al., integer-arithmetic-only inference (arXiv 1712.05877) — the scale and zero-point scheme.
Gholami et al., A Survey of Quantization Methods for Efficient Neural Network Inference (arXiv 2103.13630) — an overview of neural-network quantization.
Dettmers et al., LLM.int8() (arXiv 2208.07339) — outlier features in large transformers.
OpenStax University Physics Vol. 3: Bohr's model of the hydrogen atom — energy levels and photon emission.
OpenStax University Physics Vol. 3: The quantum particle in a box — box energy levels.
BIPM: SI defining constants — the exact Planck constant.

About this guide. Written and fact-checked by the Quantization.com editorial team against the primary sources listed above. Spotted an error or an outdated command? Send a correction — we fix verified issues within a few days. See our editorial policy.