A crop-disease detection startup operating out of Hubballi flies a fixed-wing drone over cotton fields in interior Karnataka. The drone carries a small convolutional network trained in the cloud to classify leaf images into five categories: healthy, and four common diseases (bacterial blight, leaf curl virus, grey mildew, boll rot). The field has no mobile network. The drone cannot phone home mid-flight to ask a cloud GPU "is this leaf diseased?" — it has to decide onboard, in real time, from a companion computer that also has to run flight control, GPS logging, and camera capture, all on a battery that has to last the return flight too. The model the researchers trained is stored the way every deep learning framework stores it by default: 32-bit floating point, one 4-byte number per weight. For a modest 3.4-million-parameter classifier, that is 13.6 MB of weights alone, before you count the activation memory needed to run a forward pass, before you count the RAM the flight stack needs, before you count anything left over for image buffers. The company's engineers do not have a bigger drone budget. What they have is quantization: a way to re-represent every one of those 3.4 million weights using one byte instead of four, cutting the model to 3.4 MB, and — as this chapter derives precisely rather than asserts — making the arithmetic itself faster and cheaper on the silicon the drone actually carries.
Why 32-bit floats are the wrong currency for an edge chip
An IEEE-754 float32 number spends its 32 bits as 1 sign bit, 8 exponent bits, and 23 mantissa bits, giving it a dynamic range from roughly 1.2 × 10⁻³⁸ to 3.4 × 10³⁸ and about 7 decimal digits of precision anywhere in that range. That range is built for scientific computing where a single variable might need to represent both a atom's mass and a galaxy's mass in the same program. A trained neural network's weights do not need anything close to that. After training converges, the weights in a given layer typically cluster in a narrow band — often within a few units of zero, with a standard deviation you can measure directly from the tensor. Spending 32 bits, with an exponent field that can represent 2⁻¹²⁶ and 2¹²⁷ alike, on a number that in practice never leaves the interval [−4, 4] is wasted precision the drone's power budget cannot afford.
Quantization replaces that floating representation with a fixed-point one: every weight in a tensor is mapped onto one of 256 evenly spaced integers (for 8-bit quantization), using a single scale factor computed from the tensor's own actual range. This is not "keeping fewer decimal places" of each number independently — it is a wholesale change of number system, from a per-number floating exponent to a per-tensor shared linear grid. That distinction matters enough that it is worth deriving exactly, because it is also the source of a common and costly misconception.
The affine quantization mapping, precisely
The standard scheme is affine (linear) quantization. For a weight tensor w, choose a scale s and, for asymmetric ranges, a zero-point z, such that:
quantize: q = clip( round(w / s) + z , qmin, qmax )
dequantize: ŵ = s * (q − z)For weights, which are typically distributed roughly symmetrically around zero, the simplest and most common variant is symmetric quantization: fix z = 0, use the signed int8 range [−127, 127] (127 rather than 128 so the representable range stays symmetric — one code point at −128 is simply left unused), and set the scale from the largest-magnitude weight in the tensor:
s = max(|w|) / 127
q = round(w / s), clipped to [−127, 127]
ŵ = s * qActivations after a ReLU, by contrast, are never negative — using a symmetric signed range on them would waste half the grid on codes that never occur. For those, asymmetric quantization is used instead: measure the true [min, max] of the activation (via a calibration pass, described later), set s = (max − min) / 255 and z = round(−min / s), and use the full unsigned range [0, 255]. The zero-point z is exactly the integer code that represents real-valued zero — necessary because after a ReLU, "zero" is the single most common activation value, and it needs an exact, error-free representation in the integer grid.
Worked example: quantizing a real weight vector to INT8
Take a small slice of a trained weight tensor: w = [−2.4, −1.1, 0.0, 0.8, 3.6, 1.9]. Applying symmetric int8 quantization:
max(|w|) = 3.6, so s = 3.6 / 127 = 0.028346456692913385.
Now compute w / s for each entry and round to the nearest integer:
−2.4 / s = −2.4 × 127/3.6 = −84.6667 → round → −85
−1.1 / s = −1.1 × 127/3.6 = −38.8056 → round → −39
0.0 / s = 0.0 → round → 0
0.8 / s = 0.8 × 127/3.6 = 28.2222 → round → 28
3.6 / s = 3.6 × 127/3.6 = 127.0000 → round → 127
1.9 / s = 1.9 × 127/3.6 = 67.0278 → round → 67So the quantized tensor is q = [−85, −39, 0, 28, 127, 67], storable in single signed bytes. Dequantizing (ŵ = s·q) to check the round-trip error:
−85 × s = −306/127 = −2.409449 (error 0.009449)
−39 × s = −702/635 = −1.105512 (error 0.005512)
0 × s = 0.000000 (error 0)
28 × s = 504/635 = 0.793701 (error 0.006299)
127 × s = 18/5 = 3.600000 (error 0)
67 × s = 1206/635 = 1.899213 (error 0.000787)Notice two things. First, the two extreme values (−2.4 and 3.6, which set the scale) round-trip with essentially zero error — quantization is always exact at the values that define the range. Second, every error here is below s/2 = 0.014173. That is not a coincidence: round-to-nearest quantization guarantees the error on any single value is at most half the scale, because rounding never moves a value by more than half a grid step. This gives you a design lever — a smaller scale (finer grid, or fewer outlier values stretching the range) directly bounds your worst-case per-weight error.
The same computation in code, self-contained and traceable line by line against the arithmetic above:
import numpy as np
def quantize_int8_symmetric(weights):
w = np.array(weights, dtype=np.float64)
scale = np.max(np.abs(w)) / 127
q = np.round(w / scale).astype(np.int8)
return q, scale
def dequantize(q, scale):
return q.astype(np.float64) * scale
weights = [-2.4, -1.1, 0.0, 0.8, 3.6, 1.9]
q, scale = quantize_int8_symmetric(weights)
print("scale:", round(scale, 6))
print("quantized:", list(q))
print("dequantized:", [round(float(v), 4) for v in dequantize(q, scale)])
# scale: 0.028346
# quantized: [-85, -39, 0, 28, 127, 67]
# dequantized: [-2.4094, -1.1055, 0.0, 0.7937, 3.6, 1.8992]The mapping, drawn
The diagram below shows exactly this example: the continuous FP32 line on top, the discrete 256-point INT8 grid on the bottom, and the rounding that connects each original weight to its nearest grid code.
Per-channel quantization — and the misconception it corrects
A student's first guess at what quantization does is usually: "it just rounds each number to fewer decimal places, like writing 3.14159 as 3.14." That is wrong in a way that matters. Quantization does not touch each number in isolation — it applies one shared scale, derived from the whole tensor's range, to every value in it. That shared dependency is exactly what makes quantization fragile to outliers, and exactly what the worked example above hid, because its six numbers happened to be well-behaved.
Consider a convolutional layer with two output channels. Channel A's weights all sit in [−1.0, 1.0]. Channel B has one unusually large weight (a common occurrence in trained networks, often near batch-norm-fused layers) reaching 10.0. If you quantize the whole weight tensor with a single per-tensor scale, that scale is set by the global maximum: s = 10.0 / 127 = 0.078740. Now quantize a modest channel-A weight, w = 0.9:
q = round(0.9 / 0.078740) = round(11.43) = 11
ŵ = 11 × 0.078740 = 0.866142
error = |0.9 − 0.866142| = 0.033858Channel A never needed a grid that stretches out to 10 — its own natural scale, computed per-channel, would be s_A = 1.0/127 = 0.007874:
q = round(0.9 / 0.007874) = round(114.3) = 114
ŵ = 114 × 0.007874 = 0.897638
error = |0.9 − 0.897638| = 0.002362The per-channel error (0.0024) is roughly 14× smaller than the per-tensor error (0.0339) for that same weight, purely because one outlier elsewhere in the tensor was allowed to stretch the shared grid. This is why every production mobile inference engine (TensorFlow Lite, ONNX Runtime Mobile, PyTorch's quantization toolkit) quantizes convolution and linear-layer weights per output channel: each output filter gets its own scale, computed from its own weight range, so one noisy channel cannot degrade every other channel's precision. Activations, by contrast, are usually quantized per-tensor (per-layer), because doing per-channel calibration on activations that change with every input would need to be recomputed at inference time, which defeats the purpose.
Post-training quantization vs. quantization-aware training
There are two ways to get from an FP32-trained model to an INT8 deployment, and they trade off effort against accuracy.
Post-training quantization (PTQ) takes an already-trained FP32 model and quantizes it directly — no retraining. Weight scales come straight from the weight tensors, as in the worked example. Activation scales need a short calibration pass: run a few hundred representative inputs (for the drone, a few hundred field-captured leaf images) through the FP32 model, record the min/max (or a percentile-clipped range, to avoid one freak outlier activation ruining the scale) that each activation tensor actually reaches, and fix those as the quantization ranges. PTQ is fast — minutes, not a training run — and for CNNs with reasonably well-behaved weight distributions it typically costs 1–2 percentage points of accuracy.
Quantization-aware training (QAT) is used when that accuracy loss is unacceptable. It inserts "fake quantization" nodes into the forward pass during a short fine-tuning run: each weight and activation is quantized and immediately dequantized (ŵ = s·round(w/s)) before being used, so the network experiences quantization noise while it is still updating its weights, and learns weights that are robust to it. The obstacle is that round() has zero gradient almost everywhere (it is a staircase function), so ordinary backpropagation cannot flow through it. QAT uses the straight-through estimator (STE): on the backward pass, treat round() as if it were the identity function inside the valid range (gradient = 1) and zero outside it. It is an approximation, not a true derivative, but empirically it lets the network's weights drift toward values whose rounding error costs little accuracy — closing most of the gap PTQ leaves on the table, at the price of an actual training run.
What the drone actually gains
Back to the field. The 3.4M-parameter classifier at FP32 is 13.6 MB of weights; at INT8 it is 3.4 MB — exactly 4× smaller, because that ratio is fixed by 4 bytes vs. 1 byte and holds regardless of how many parameters the model has. On a companion computer built around an ARM Cortex-A55-class SoC with roughly 1 GB of RAM shared across the flight controller's telemetry buffers, GPS logging, and live camera frames, a budget of perhaps 20 MB for the vision model and its runtime is realistic. The FP32 weights alone, plus the intermediate activation buffers a convolutional forward pass needs (which are themselves 4× larger at FP32 than at INT8, for the same reason), leave little headroom before something else on the drone starves for memory. The INT8 model fits with room to spare.
Memory is not the only win. ARM's NEON SIMD unit uses 128-bit registers. A 128-bit register holds four 32-bit float lanes, but sixteen 8-bit integer lanes — four times as many values processed per instruction. Because a convolution is fundamentally a long sequence of multiply-accumulate operations, and INT8 dot-product instructions (such as ARMv8.2's SDOT) exploit exactly this lane packing, the theoretical throughput ceiling for the same convolution is roughly 4× higher in INT8 than in FP32 on this class of chip — though real speedups land lower than that ceiling once you account for memory bandwidth, the layers that still run in float, and the small dequantization overhead at a network's output. Energy tells a similar story: integer multiply-accumulate circuits are simpler and consume markedly less energy per operation than floating-point ones on the same process node — a difference researchers in efficient deep learning have measured at roughly an order of magnitude for multiplies specifically, at older process nodes. Exact figures shift with process node and design, but the qualitative point does not: fewer bits moved, integer arithmetic instead of floating-point, and a model that is both smaller and cheaper to run per inference — which is the entire reason a battery-powered drone can classify a diseased cotton leaf without ever touching a network connection.
Active recall
Attempt each question before reading its answer.
- Symmetric-quantize
w = [4.0, −1.0, 2.5, −4.0, 0.5]to INT8. Give the scale, the five quantized integers, and the dequantized values. - A weight tensor is quantized per-tensor with a scale set by one outlier channel. What specifically goes wrong for the other channels, and how does per-channel quantization fix it?
- Why is asymmetric quantization (with a nonzero zero-point) the natural choice for post-ReLU activations, while symmetric quantization is the natural choice for weights?
- The FP32→INT8 memory reduction is always exactly 4×. Is the inference speedup also always exactly 4×? Why or why not?
- What breaks when you try to backpropagate through the
round()function during quantization-aware training, and what technique works around it? - A student says: "Quantization is just rounding each float to fewer decimal digits, like 3.14159 → 3.14." What is wrong with that statement?
Answers.
1. max(|w|) = 4.0, so s = 4.0/127 = 0.0314961. Dividing each weight by s: 4.0/s = 127.000 → 127; −1.0/s = −31.75 → −32; 2.5/s = 79.375 → 79; −4.0/s = −127.000 → −127; 0.5/s = 15.875 → 16. Quantized: [127, −32, 79, −127, 16]. Dequantized: 127×s = 4.0; −32×s = −1.007874 (error 0.007874); 79×s = 2.488189 (error 0.011811); −127×s = −4.0; 16×s = 0.503937 (error 0.003937). Every error is below s/2 = 0.015748, as guaranteed.
2. The shared scale is dragged out to cover the outlier channel's range, so every other channel — which never needed that wide a grid — gets coarser resolution than its own values warrant, inflating its rounding error (the worked comparison above showed a ~14× error increase for a channel whose true range was 10× smaller than the tensor maximum). Per-channel quantization gives each output channel its own scale computed from its own weights, so one channel's outlier no longer taxes every other channel's precision.
3. Post-ReLU activations are bounded below by exactly zero and can range arbitrarily high above it — an inherently one-sided, asymmetric distribution, so mapping them onto the full unsigned [0,255] range with a zero-point that lands exactly on the integer representing real zero uses the grid efficiently and represents the very common "exactly zero" activation with no error. Weights are typically distributed roughly symmetrically around zero, so a signed [−127,127] range with zero-point fixed at 0 wastes nothing and needs no separate zero-point parameter to store or compute with.
4. No. The 4× memory ratio is fixed purely by byte width (4 bytes vs. 1 byte) and always holds. Speedup depends on the hardware actually exploiting the narrower type: how many int8 SIMD lanes fit per instruction relative to float32 lanes (up to 4× on 128-bit NEON, as derived above), memory-bandwidth limits that may already be the bottleneck rather than compute, layers that remain in float (e.g. the final softmax), and the overhead of quantizing inputs and dequantizing outputs at the model's boundary.
5. round() is a staircase function: its true derivative is 0 almost everywhere and at the jumps, so ordinary backpropagation would report zero gradient for every quantized weight and stop learning. The straight-through estimator (STE) works around this by treating round() as the identity function on the backward pass (gradient ≈ 1 within the valid range, 0 outside it) — an approximation that is mathematically inexact but lets useful gradient signal flow through, letting the network adapt its weights to tolerate the quantization noise it will face at deployment.
6. Two things: quantization does not round each number independently — it applies one scale derived from the whole tensor's range to every value in the tensor, so a single outlier weight or activation can distort the precision available to every other value (as shown in the per-channel comparison). And it is not decimal rounding at all — it re-represents the numbers in an entirely different number system, a fixed-point integer grid with an explicit scale (and often a zero-point) computed by calibration, not a cosmetic truncation of a float's printed digits.
Think About It
Think about this: How would you explain quantization: running models on edge devices to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind quantization: running models on edge devices, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.