Why Low-Bit Quantization Is Hard
Three first-principle problems: quantization creates rounding/clipping error, scale controls the error tradeoff, and nearest rounding can be locally correct but globally bad for LLM layers.
q = clamp(round(x / s), qmin, qmax)
xdq = q × s
e = xdq − x
Rounding error and clipping error
INT4 only has a small set of grid points. Values inside the range are rounded; values outside the range are clipped.
Rounding: value is inside range
Clipping: value is outside range
Scale selection is a tradeoff
The scale chooses both the grid spacing and the representable range. Large scale protects outliers; small scale preserves normal values.
Large scale: protect outlier
| Effect | Result |
|---|---|
| Outlier 2.0 | Preserved |
| Small values 0.05, 0.08, 0.10 | Collapse to 0 |
| Dominant error | Rounding error for normal values |
Small scale: preserve normal values
| Effect | Result |
|---|---|
| Small values 0.05, 0.08, 0.10 | Preserved well |
| Outlier 2.0 | Clipped to 0.21 |
| Dominant error | Clipping error for outliers |
Rounding is not always harmless
Nearest rounding minimizes scalar error, but LLM layers care about output error after GEMM: Y = XW.
Nearest rounding: locally correct
[1, 1]
[0.49, 0.49]
0.98
0.98
[1, 1]
[0, 0]
0
0.98
Each 0.49 is closer to 0 than 1, so nearest rounding chooses [0, 0]. But the layer output becomes very wrong.
Non-nearest rounding: globally better
[1, 1]
[1, 0]
1
0.02
locally worse
The model does not optimize each weight alone. It cares about whether XW is preserved after quantization.