Model Quantization: INT8, INT4, and Binary Neural Networks

Suppose an Indian edtech startup wants to ship an offline doubt-solving assistant — Hindi and Tamil, no internet required — on a ₹8,000 Android phone with 4 GB of RAM and no dedicated NPU. The language model behind it has 7 billion parameters. Trained and stored the normal way, in 32-bit floating point (FP32), each parameter costs 4 bytes, so the weights alone occupy 28 GB. Even the more modest FP16 format used for most training runs still needs 14 GB — more RAM than the entire phone has, before the operating system, the camera app, or WhatsApp get a share. The model literally does not fit. Cut every weight down to an 8-bit integer and the same model needs 7 GB; cut it to 4 bits and it needs 3.5 GB — tight, but plausible with careful memory management; go all the way to a single bit per weight and the raw parameter storage drops below 900 MB. This is not a rounding trick or a compression codec run at load time — it is a change to the actual numeric format the network computes in, and it is the difference between "runs only in a data center" and "runs in your pocket." That is what this chapter is about: how a network trained in floating point gets mapped onto integers, why 4 bits is dramatically harder than 8, and why some networks go to the extreme of a single bit per weight.

Why compress a trained network at all

Two costs matter for running a neural network at inference time, and they are not the same cost. The first is storage: how many bytes does the model occupy on disk and in RAM. The second, and usually the larger one in practice, is memory bandwidth. A large language model generates text one token at a time, and to produce each token it must read every weight in the network from memory at least once. GPU compute (FLOPs) has grown far faster over the last decade than GPU memory bandwidth has, so for the token-by-token decoding that dominates LLM inference, the GPU frequently sits idle waiting for weights to arrive from HBM rather than being limited by how fast it can multiply numbers — a well-known bottleneck engineers call being "memory-bound." Halving the number of bytes per weight roughly halves the time spent waiting, independent of anything else. A third, smaller effect is that integer arithmetic units are cheaper to build and run faster per watt than floating-point units of the same bit width, so INT8 matrix multiplication on the same silicon can be several times faster than FP16. Quantization attacks all three costs at once by changing the representation, not the architecture.

From floating point to a discrete grid: affine quantization

A trained weight tensor is a set of real numbers scattered inside some bounded range — say, between −0.91 and 0.91. Floating point represents each of those numbers independently, spending 32 (or 16) bits per value regardless of how many distinct values are actually needed. Quantization instead builds one shared, coarse integer grid for the whole tensor (or a chunk of it) and stores only which grid point each weight is closest to. The grid is defined by a single floating-point number called the scale, and the mapping is an affine transformation: each real value is divided by the shared step size and rounded to the nearest integer, then optionally shifted by an offset, and finally clipped to the smallest and largest integer the bit width allows — q = round(w / scale) + zero_point, clipped to [q_min, q_max]. Recovering an approximate real value later — dequantizing — simply reverses that arithmetic: ŵ = (q − zero_point) × scale.

When a tensor's values are roughly symmetric around zero — true for most trained weight matrices — the zero_point is set to 0 and the scheme is called symmetric quantization, with scale = max(|w|) / q_max. When a tensor is one-sided — activations after a ReLU, for instance, which are never negative — a nonzero zero_point lets the integer grid shift so the available codes aren't wasted on a half-range that never occurs; this is asymmetric quantization. Either way, the crucial fact — and the one that separates real quantization from "just rounding off decimal digits" — is that the scale is a single shared floating-point number covering the whole tensor (or group), not something stored per weight. Each weight itself becomes a small integer with no exponent or fraction of its own; all the numeric range information lives in that one shared scale. This is why the scheme is sometimes called block floating point: one exponent, many small mantissas.

Worked example: quantizing a weight vector to INT8

Take a small weight vector from some layer: w = [−0.87, 0.23, 0.91, −0.45]. For symmetric INT8, the integer codes must fit in [−127, 127] (127, not 128, is used as the positive bound so the grid stays exactly symmetric about zero — the code −128 is simply never used). The scale is set by the largest-magnitude weight: scale = max(|w|) / 127 = 0.91 / 127 ≈ 0.0071654.

The same computation, as code — this is real, runnable Python and its output below is what actually executing it produces:

def quantize_symmetric(w, bits=8):
    q_max = 2 ** (bits - 1) - 1              # 127 for INT8, 7 for INT4
    scale = max(abs(x) for x in w) / q_max
    q = [round(x / scale) for x in w]        # integer codes
    dequant = [qi * scale for qi in q]       # ŵ, what inference actually uses
    return scale, q, dequant

w = [-0.87, 0.23, 0.91, -0.45]
scale, q, dequant = quantize_symmetric(w, bits=8)
print(scale)
print(q)
print(dequant)

# output:
# 0.0071653543307086615
# [-121, 32, 127, -63]
# [-0.8670078740157481, 0.22929133858267717, 0.91, -0.4514173228346457]

The mean absolute error here is about 0.00128 — small, because the grid spacing (0.00717) is small relative to the weight magnitudes. Note that 0.91 lands on code 127 exactly, using the full range, while the smallest-magnitude weight (0.23) only reaches code 32: the single largest weight in the tensor determines how finely everything else gets resolved. That fact becomes important later.

Down to INT4: the same math, sixteen times coarser

Nothing about the formula changes for 4-bit quantization — only q_max shrinks, from 127 to 7 (again reserving one code, −8, for exact symmetry). On the same weight vector, scale = 0.91 / 7 = 0.13:

Mean absolute error is now 0.0325 — about 25 times larger than the INT8 case, from a grid that is only 16 times coarser (16 levels instead of 256). Errors grow faster than the level count shrinks because rounding error and grid spacing are directly proportional, and a 4-bit grid simply cannot get close to most real values. In a toy 4-element example this is a curiosity; across a 7-billion-parameter transformer it is enough to noticeably degrade output quality if applied naively, per tensor, uniformly.

Two refinements make 4-bit weights usable in practice, and both are real published methods, not folklore. GPTQ (Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, ICLR 2023) is a one-shot, post-training method that quantizes a weight matrix one column at a time; after each column is rounded to its integer grid, the remaining not-yet-quantized columns are adjusted slightly, using second-order curvature information (an approximate inverse Hessian computed from a small calibration set), to compensate for the error the rounding just introduced. This lets it push weights of models with well over a hundred billion parameters down to 3–4 bits in a few GPU-hours, with no retraining. AWQ (Ji Lin, Jiaming Tang, Haotian Tang, Song Han, and colleagues, MLSys 2024) takes a different angle: it observes that a small fraction of weight channels — those multiplied by activations with unusually large magnitude — matter disproportionately for output quality. Uniform quantization gives every value roughly the same absolute rounding error (about half a grid step), so for a channel whose weights are large, that fixed absolute error is a small relative error, while for a channel whose weights are small it is a large one. AWQ exploits this directly: before quantizing, it multiplies each salient channel's weights by a per-channel factor s > 1 and divides the corresponding activations by the same s — mathematically w·x = (w·s)·(x/s), so exact arithmetic is unchanged — which makes the salient weights larger relative to the fixed rounding step, shrinking their relative error, at the cost of very slightly worse precision on the less important channels. No weight is skipped or kept at higher precision; the scaling alone rebalances where the error goes. A cheaper, complementary trick used by both methods and by most production INT4 kernels is per-group quantization: instead of one scale for an entire weight matrix, use one scale per contiguous block of (commonly) 64 or 128 weights, so a single outlier in one group cannot flatten the precision of every other weight in the matrix — a phenomenon Active Recall Question 6 below makes concrete.

The extreme case: binary neural networks

Push the bit width to its logical floor — one bit per weight — and the affine formula degenerates into a sign function. BinaryConnect (Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David, NeurIPS 2015) first showed a network could be trained with weights constrained to exactly ±1. XNOR-Net (Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi, ECCV 2016) refined this by pairing the binary sign with one floating-point scaling factor per output channel, α = mean(|w|), so the binarized weight is ŵ = α · sign(w) rather than a bare ±1 — a single float recovers some of the magnitude information lost by binarizing. On the same vector, α = (0.87+0.23+0.91+0.45)/4 = 0.615:

Mean absolute error is 0.275 — over 200 times the INT8 error on the same four numbers. Individually, binary weights are a poor approximation of anything. The reason binary networks are still used at all is not accuracy per weight; it is what binarization does to the arithmetic. When both weights and activations are constrained to ±1 and encoded as single bits (1 for +1, 0 for −1), a dot product over N elements — normally N multiplications and N−1 additions — becomes two bitwise instructions: XNOR the two bit-vectors, then count the set bits (popcount). Two matching bits (both +1 or both −1) contribute +1 to the true bipolar dot product and produce an XNOR output of 1; two mismatched bits contribute −1 and produce XNOR output 0, so dot(a, b) = 2 · popcount(XNOR(a_bits, b_bits)) − N.

XNOR of the two bit strings is 1, 0, 0, 1, so popcount = 2 and the formula gives 2×2 − 4 = 0. Checking by direct multiplication: (1)(1) + (−1)(1) + (1)(−1) + (1)(1) = 1 − 1 − 1 + 1 = 0 — the two methods agree, as they must, since XNOR-plus-popcount is not an approximation of the bipolar dot product, it is an exact bit-level implementation of it. XNOR and popcount are among the cheapest operations a CPU or custom accelerator can execute — orders of magnitude cheaper than a floating-point multiply-add. The XNOR-Net paper reports roughly 58× faster convolutions and roughly 32× lower memory use for fully binarized (weights and activations) convolutional layers versus a full-precision baseline on their CPU implementation. Binary networks trade a large, fixed per-weight error for an enormous constant-factor speedup — a trade that only makes sense where the accuracy hit is affordable (small vision/audio models, wake-word detectors, extreme-edge microcontrollers) and where every joule and every byte of memory is scarce, such as an onboard image-triage model on a power- and radiation-constrained satellite subsystem rather than a general-purpose language model.

Teaching a network to live with rounding: PTQ, QAT, and the straight-through estimator

There are two moments at which quantization can be introduced. Post-training quantization (PTQ) takes an already-trained FP32/FP16 model, runs a small calibration set through it to measure the actual min/max (or a percentile) of each tensor, computes scales from those statistics, and converts weights (and often activations) to integers — no gradient descent involved. This is cheap and fast, which is why GPTQ and AWQ are both PTQ methods, but it has no mechanism to compensate for the accuracy it loses beyond the layer-wise correction tricks described above. Quantization-aware training (QAT) instead simulates the rounding during the forward pass of training or fine-tuning — weights and activations are quantized and immediately dequantized ("fake quantized") before every layer's computation — so the loss function sees quantization error and the optimizer can adjust the real-valued weights to be more forgiving of it. The obstacle is that round() has zero gradient almost everywhere (it is a staircase function), so backpropagation through it would kill every gradient. BinaryConnect's contribution was applying the straight-through estimator (STE) — introduced two years earlier by Bengio, Léonard, and Courbariaux (2013) — to binarize weights at scale: use the quantized (rounded, or binarized) weights for the forward pass, but during the backward pass, pretend the rounding function was the identity function and pass the gradient through unchanged (clipped to zero outside the representable range). It is mathematically inexact — the true gradient of a step function is zero or — but it works well enough in practice that it remains the standard trick behind essentially all QAT and binary-network training a decade later.

The quantization grid, side by side

The diagram below places FP32, INT8, INT4, and binary representations on the same number line so the shrinking grid is visible directly, using the scale and α values computed in the worked examples above.

Common misconception: "if the inputs are 8-bit, the accumulator must be 8-bit too"

Students meeting INT8 inference for the first time often assume that because the weights and activations are stored as 8-bit integers, the running sum inside a dot product must also stay within 8-bit range. It cannot, and real hardware never does it that way. Take the INT8-quantized weight vector from the worked example, q_w = [−121, 32, 127, −63], and quantize an activation vector a = [0.5, −0.2, 0.33, 0.81] the same way (symmetric, scale_a = 0.81/127 ≈ 0.006378), giving q_a = [78, −31, 52, 127]. The integer dot product these two vectors actually need is the sum of four products:

−11827 does not fit in a signed 8-bit range of [−128, 127] — it overflows by nearly a hundredfold. If the hardware actually accumulated in 8 bits, this computation would silently wrap around and produce garbage on almost every real matrix multiplication, not just pathological ones. Real INT8 GEMM kernels (and the tensor cores that run them) always accumulate in a wider register — INT32 is standard — precisely so that a sum of many 8-bit products cannot overflow. Only once the full dot product is finished does the wide integer result get rescaled back to something meaningful, by multiplying by scale_w × scale_a and, at that point, optionally re-quantized to 8 bits for the next layer: rescaled ≈ −11827 × 0.0071654 × 0.006378 ≈ −0.5405, versus the true FP32 dot product (−0.87)(0.5) + (0.23)(−0.2) + (0.91)(0.33) + (−0.45)(0.81) ≈ −0.5452 — an absolute error of about 0.0047, small enough to be usable, but only because the accumulation itself was never bottlenecked to 8 bits. The two multiplication operands are narrow (8 bits each, which is exactly what makes the multiply fast and cheap), but the accumulator that sums many such products must always be wide enough that no realistic sum overflows it — this is precisely why INT8 matrix-multiply kernels report an INT32 output tensor, and why the rescale-and-round step happens exactly once per layer, not once per multiplication.

Active recall

Attempt each question before reading its answer.

1. A model has 7 billion parameters. What is its weight storage in FP16, INT8, and INT4, and what compression factor does each give relative to FP32?
2. Why must the accumulator inside a quantized matrix multiply be wider than the bit width of the two operands? Use the −11827 example to justify your answer numerically, not just conceptually.
3. Quantize w = [0.62, −0.15, −0.98, 0.40] to symmetric INT8: give the scale, the four integer codes, the dequantized values, and the per-element absolute errors.
4. A single weight matrix uses one shared scale for all 4096 columns (per-tensor quantization). Explain, using the idea of a shared scale set by the maximum magnitude, why splitting the matrix into groups of 128 columns and giving each group its own scale (per-group quantization) usually improves accuracy at 4-bit precision, and what it costs.
5. Using XNOR + popcount, compute the bipolar dot product of a = [+1, +1, −1, −1] and b = [−1, +1, −1, +1]. Verify your answer by direct multiplication.
6. Ripple-effect question. Take the original worked vector w = [−0.87, 0.23, 0.91, −0.45] and add one more weight to it: 3.5 (an outlier — this can genuinely happen in trained transformer weight/activation tensors). Recompute the INT8 symmetric scale and the integer codes for all five weights, including the original four. What happens to the precision available to the original four weights, and why does this matter for large language models specifically?

Answers

1. FP16: 7×10⁹ × 2 bytes = 14.0 GB (2× smaller than FP32's 28.0 GB). INT8: 7×10⁹ × 1 byte = 7.0 GB (4× smaller than FP32). INT4: 7×10⁹ × 0.5 byte = 3.5 GB (8× smaller than FP32).

2. Each 8-bit operand ranges over [−128, 127], but a dot product sums many products of such operands, and each product can itself already reach roughly ±16,256 (128×127). Summing four such products, as in the worked example, produced −11827 — far outside 8-bit range, and a real weight matrix multiply sums over hundreds or thousands of terms, not four. An 8-bit accumulator would overflow (wrap around) on essentially every nontrivial dot product, so hardware always accumulates in a wider register (INT32 in practice) and only rescales to a smaller format once, after the full sum is complete.

3. max(|w|) = 0.98, so scale = 0.98/127 ≈ 0.0077165. Codes: round(0.62/0.0077165) = 80, round(−0.15/0.0077165) = −19, round(−0.98/0.0077165) = −127, round(0.40/0.0077165) = 52. Dequantized: 0.6173, −0.1466, −0.9800, 0.4013. Absolute errors: 0.0027, 0.0034, 0.0000, 0.0013.

4. Per-tensor scale is fixed by the single largest-magnitude weight anywhere in the matrix. If most columns hold small weights but a handful of columns (or one column) contain a much larger value, that one value stretches the scale for the whole tensor, and every small weight elsewhere gets crushed toward the low end of the integer range, where relative rounding error is largest — exactly the mechanism demonstrated in question 6. Giving each group of 128 columns its own scale means each group's grid is set by the maximum within that smaller group, so an outlier in one group no longer degrades the precision of unrelated columns. The cost is metadata: one extra float scale per group instead of one per tensor, plus slightly more complex kernel logic to track which scale applies to which columns — a small storage overhead traded for a large accuracy gain at low bit widths.

5. Bit encoding (1 = +1, 0 = −1): a_bits = 1100, b_bits = 0101. XNOR = 0110, popcount = 2. Formula: 2×2 − 4 = 0. Direct check: (1)(−1) + (1)(1) + (−1)(−1) + (−1)(1) = −1 + 1 + 1 − 1 = 0. Matches.

6. New max(|w|) = 3.5, so the new scale = 3.5/127 ≈ 0.0275591 — about 3.85× larger than the original 0.0071654 (exactly the ratio 3.5/0.91). Re-quantizing all five weights with this new scale: the outlier itself lands at code 127 (using the full range, as any current max always will), but the original four weights, which previously used codes [−121, 32, 127, −63] — one of them using the entire available range — now round to only [−32, 8, 33, −16]. The largest-magnitude original weight (0.91) no longer reaches code 127; it barely reaches 33, using about 26% of the available integer codes instead of 100%. Every one of the original four weights lost roughly three quarters of its representable precision, purely because one unrelated value joined the tensor. This is not a toy artifact: real transformer activation tensors (and some weight tensors) develop exactly this kind of outlier once model scale passes a few billion parameters, and naive per-tensor INT8 quantization applied across such a tensor destroys precision everywhere else in it. This is precisely the problem the LLM.int8() method (Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, NeurIPS 2022) addresses for transformer inference, by detecting the small number of outlier feature dimensions and computing just those in FP16 while quantizing the rest to INT8 — and it is the same reason per-group and per-channel scaling (question 4) matter more as models get larger, not less.

Think About It

Think about this: How would you explain model quantization: int8, int4, and binary neural networks to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind model quantization: int8, int4, and binary neural networks, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.