A Translation App That Has to Work on a Rs. 7,000 Phone
Picture a shopkeeper in a small town in Bihar who has just received a carton of medicine with dosage instructions printed only in English. She opens Google Lens on her phone, points the camera at the label, and within a second sees the text translated into Hindi. Her phone is not a flagship device — it is an entry-level Android phone worth somewhere around Rs. 7,000, with 3 GB of RAM, a modest processor, and an internet connection that drops to patchy 2G the moment she steps behind her own shop counter. Yet the translation still happens, right there on the phone, in under a second.
The neural network that recognizes the English text and turns it into Hindi was originally trained on powerful data-center GPUs, using numbers stored with 32 bits of precision for every single one of its millions of weights. Shipped exactly as trained, a network like that would be too large to download over a weak connection, too slow to run on a budget chip, and would drain the phone's battery in minutes. Somewhere between "trained on a GPU cluster" and "running instantly in her palm," someone had to make that network dramatically smaller and faster — without making it noticeably worse at its job.
The technique that makes this possible is called quantization: taking the numbers inside a trained neural network and representing them with fewer bits than they were trained with. It sounds like it should only save storage space. As this chapter will show, it also makes the model run faster — often two to four times faster on the right hardware — which turns out to be the more surprising and more important half of the story.
This is not a one-off trick for a single app. Every India-facing AI feature that has to work offline or on inexpensive hardware — camera translation, voice typing in a regional language, a keyboard app predicting the next word, spam detection running inside a messaging app — leans on this exact same idea. Understanding quantization is understanding how AI stops being something that only runs in a data centre and becomes something that fits in every pocket.
What a Number Actually Costs Inside a Computer
Every number a computer stores takes up a fixed number of bits, and every one of those bits has to be moved through memory, cache, and processor registers each time it is used. The default format for numbers during neural network training is the 32-bit floating-point number, usually written as FP32. Following the IEEE 754 standard, one FP32 number occupies 32 bits — 4 bytes — arranged as 1 sign bit, 8 exponent bits, and 23 mantissa (fraction) bits. The exponent bits let the number represent anything from a tiny fraction to an enormous value; the mantissa bits give it fine-grained precision at whatever scale the exponent selects.
That precision genuinely matters during training. Gradient descent nudges each weight by a very small amount on every step — often a change in just the fourth, fifth, or sixth decimal place — and millions of such tiny nudges accumulate over the course of training. With only a couple of decimal digits of precision available, most of those nudges would simply round away to nothing, and the network would never learn properly. FP32 exists to make that slow, delicate accumulation possible.
But once training is finished and the network is frozen for deployment, that same precision starts to look like overkill. Consider a simpler version of the same idea: if you are laying floor tiles in a room that measures 3.85 by 4.2 metres, you do not need a laser reading of 3.847293817 metres — the extra digits change nothing about how many tiles you buy. Two decimal places carry all the information the task actually needs. A trained network's weights are similar: they are useful, meaningful numbers, but once training has stopped, most of them no longer need 23 bits of mantissa to keep doing their job correctly.
Lower-precision formats exist for exactly this reason:
- FP16 (half precision) uses 16 bits — 1 sign, 5 exponent, 10 mantissa bits — half the size of FP32.
- INT8 uses 8 bits to store a plain integer, typically ranging from -128 to 127 — a quarter the size of FP32.
- INT4 uses just 4 bits, ranging from -8 to 7 — an eighth the size of FP32, increasingly used for today's large language models.
If you have ever browsed AI models on Hugging Face or run a local chatbot with a tool like Ollama, you may already have seen filenames such as Llama-3-8B-Q4 or a model labelled "INT8" — that label is telling you exactly which of these formats the model's numbers were squeezed into.
What Is Quantization, Exactly?
Quantization is the process of mapping a wide range of precise values onto a small, fixed set of lower-precision values, using a simple and consistent formula — and then computing directly with that smaller set of values. For neural networks, quantization is usually applied to the weights (the numbers the network learned during training that decide how much influence each input has) and, in a full deployment, to the activations too — the intermediate values that flow from one layer to the next while the network is actually running.
The most common approach is called linear (or affine) quantization, and it works in three steps. First, look at the actual range of values you need to represent — say, all the weights in one layer — and find the largest value by magnitude. Second, compute a scale factor: divide that maximum magnitude by the largest integer your target format can hold (127 for INT8). Third, convert every value by dividing it by the scale and rounding to the nearest integer. To recover an approximate version of the original number later, multiply the integer back by the same scale — a step called dequantization.
This works well specifically because trained neural network weights are not scattered evenly across some enormous range — they are almost always small numbers clustered close to zero, a natural side effect of how networks are initialized and regularized during training. A narrow, predictable range is exactly what a scale-and-round scheme compresses with very little loss.
Weights are easy to quantize because they are fixed the moment training ends — the same scale factor can be computed once and reused forever after. Activations are trickier, because their actual range depends on whatever input is fed into the network at that moment. Two strategies handle this. Static quantization runs the finished FP32 model over a small batch of representative example inputs beforehand, records the typical range each layer's activations fall into, and fixes a scale factor from that calibration data before the model ever ships. Dynamic quantization instead measures the actual minimum and maximum of each layer's activations in real time, on every single inference, and computes the scale on the fly. Static quantization is faster at inference time, since no range-finding happens during the real run, but dynamic quantization adapts automatically to inputs that look nothing like the calibration data used ahead of time.
Worked Example: Quantizing Five Weights by Hand
Suppose one small slice of a trained layer has produced these five FP32 weights:
weights = [-0.82, -0.35, 0.10, 0.47, 0.91]We will quantize them to INT8 using symmetric linear quantization — "symmetric" because positive and negative values are treated identically, with no separate offset.
Step 1 — find the scale. The largest magnitude among the five weights is 0.91. INT8 is kept symmetric around zero by using the range -127 to 127, so:
scale = 0.91 / 127 ≈ 0.00716535Step 2 — divide each weight by the scale and round to the nearest integer.
-0.82 / 0.00716535 = -114.44 → -114
-0.35 / 0.00716535 = -48.85 → -49
0.10 / 0.00716535 = 13.96 → 14
0.47 / 0.00716535 = 65.59 → 66
0.91 / 0.00716535 = 127.00 → 127The quantized weights stored on the phone are now [-114, -49, 14, 66, 127] — five plain 8-bit integers, one byte each, instead of five 32-bit floats. That alone is a 4x reduction in storage for this slice.
Step 3 — check what was lost. To see the value the hardware effectively uses, dequantize by multiplying each integer back by the scale:
-114 * 0.00716535 = -0.81685
-49 * 0.00716535 = -0.35110
14 * 0.00716535 = 0.10031
66 * 0.00716535 = 0.47291
127 * 0.00716535 = 0.91000Compare these to the originals: -0.82 became -0.81685, 0.47 became 0.47291, and so on. Every value shifted by less than 1% of its own size, and 0.91 — the value that defined the scale — came back exact. This is not luck: because rounding always moves a value to the nearest representable point, the error on any single number can never exceed half of the scale, which here is about 0.00358. Every error above sits comfortably inside that bound. This is the entire trade of quantization in miniature — a small, mathematically bounded amount of rounding error, exchanged for a quarter of the storage and, as the next section shows, a large jump in speed.
The Same Idea, in Code
The exact calculation above is only a few lines of Python with NumPy:
import numpy as np
weights = np.array([-0.82, -0.35, 0.10, 0.47, 0.91], dtype=np.float32)
def quantize_int8(x):
scale = np.max(np.abs(x)) / 127
q = np.round(x / scale).astype(np.int8)
return q, scale
def dequantize(q, scale):
return q.astype(np.float32) * scale
q_weights, scale = quantize_int8(weights)
recovered = dequantize(q_weights, scale)
print("Original :", weights)
print("Scale :", scale)
print("Quantized:", q_weights)
print("Recovered:", recovered)
print("Max error:", np.max(np.abs(weights - recovered)))Running this prints the same integers and recovered values worked out by hand above, with a maximum error of roughly 0.003 — safely under the 0.00358 bound. Production frameworks such as PyTorch's torch.quantization module and TensorFlow Lite perform this same core arithmetic, just applied across millions of weights at once, layer by layer, with extra bookkeeping for details like unusually large outlier values and, commonly, a separate scale for each output channel rather than one scale for an entire layer — a refinement called per-channel quantization that keeps the "narrow, predictable range" assumption from the previous section true even when a few channels happen to have larger weights than the rest.
Why Fewer Bits Also Means More Speed
Shrinking the model is the easy half to picture. The speedup is the half worth understanding properly, and it comes from three separate sources.
The first is memory bandwidth. Running a neural network is not only about doing arithmetic — it is about constantly moving weights from memory into the processor to be used. For most models, this movement, not the arithmetic itself, is the real bottleneck. An INT8 weight is a quarter the size of an FP32 weight, so four times as many weights can be pulled from memory in the same amount of time, and far more of a layer's weights fit into the processor's small, fast cache instead of slower main memory.
The second is parallel width. Processors compute many numbers at once using wide registers and SIMD (Single Instruction, Multiple Data) instructions — ARM NEON, found in virtually every phone processor, is one example. A 128-bit-wide register holds four 32-bit floats, but the same 128 bits hold sixteen 8-bit integers. One SIMD instruction can therefore perform sixteen INT8 multiplications in roughly the time it takes to perform four FP32 multiplications — up to four times the throughput per instruction, simply by switching formats.
The third is dedicated hardware. The Snapdragon and MediaTek Dimensity chip families that power the large majority of Android phones sold in India, from budget models to flagships, include a separate NPU (Neural Processing Unit) or DSP block built specifically to run neural network arithmetic in low precision, alongside the general-purpose CPU and GPU. These blocks are not built to accelerate FP32 math the way they accelerate INT8 math — an integer arithmetic circuit needs far fewer logic gates than a floating-point circuit capable of the same throughput, so it is smaller, faster, and uses less energy per operation. When the shopkeeper's translation model runs in INT8, the phone can hand the work to this specialized silicon instead of the comparatively slow, power-hungry main processor. During the actual matrix multiplications, the hardware multiplies pairs of 8-bit integers and accumulates the results in a wider 32-bit integer register, large enough to avoid overflow, so the entire core computation runs in integer arithmetic from start to finish without ever touching the more expensive floating-point circuitry.
Put together, these three effects are why practical INT8 deployments commonly report inference two to four times faster than FP32 on hardware with dedicated integer support, alongside noticeably longer battery life per session — a genuine speedup, not just a smaller download.
Two Ways to Get There
There are two standard strategies for producing a quantized model, and choosing between them is a straightforward trade-off between effort and accuracy.
Post-training quantization (PTQ) takes a model that has already finished training in FP32 and converts its weights to INT8 afterward, exactly as in the worked example above. It needs no retraining, takes minutes to run, and typically needs only a small sample of representative data to calibrate scale factors properly. For most well-trained models, this is enough.
Quantization-aware training (QAT) is more involved: the effect of quantization is simulated during training itself, so the network's own gradient descent process learns weights that are already robust to the rounding they will experience later. This costs extra training time and data, but typically recovers more of the accuracy lost to quantization, and is generally preferred when a model is being pushed to very low precision, such as INT4, where the rounding error is larger and harder to absorb after the fact.
Quantization is only one member of a family of techniques for making trained networks smaller and faster. Pruning removes entire weights or connections that contribute little to the output, rather than shrinking every number's precision. Knowledge distillation trains a smaller "student" network from scratch to imitate a larger "teacher" network's behaviour. All three are frequently combined in the same deployed model, but quantization is usually the first and cheapest one to try, because — as the worked example showed — it can be applied to an existing trained model in minutes, without touching its architecture at all.
How Much Accuracy Do You Actually Give Up?
The worked example showed individual weight errors under 1%. Does that translate into a noticeably worse model? Usually, remarkably little. Trained neural networks tend to be heavily over-parameterized and redundant — a single prediction is typically the combined result of thousands of weights acting together, and a small, evenly distributed rounding error in each one rarely changes which output ends up largest.
There is a specific reason this robustness holds even though every weight has shifted a little. In a classification task, a network's final layer produces one score, called a logit, for each possible output class, and the winning class is simply whichever logit is largest. Quantization shifts every logit by a small amount, because it shifts every weight feeding into it by a small amount — but this rarely flips the ranking between the top two logits, because a well-trained network usually settles on a clear winner with a comfortable margin over the runner-up. Rounding only becomes dangerous for the rare input where the network was already uncertain, with two logits nearly tied — precisely the kind of input a full-precision model would also have struggled with.
Benchmarks reported by organizations such as Google, NVIDIA, and Qualcomm consistently show that well-implemented INT8 post-training quantization costs well under one percentage point of accuracy on standard image-classification and language benchmarks, in exchange for a quarter of the memory footprint and a substantial speed gain. Before any of this ships to a real phone, engineers routinely confirm this directly: they run both the original and the quantized model over the same held-out validation data and compare the outputs, rather than trusting the trade-off blindly.
The trade-off gets sharper at more extreme precisions. Pushing all the way to INT4 or below can measurably hurt accuracy if done carelessly, particularly for the occasional unusually large "outlier" weight or activation that a single shared scale factor represents poorly — a large outlier forces the scale factor to be large too, which coarsens the rounding for every ordinary, small-valued weight sharing that scale. This is precisely why techniques built specifically for aggressive low-bit quantization of large language models, such as GPTQ and AWQ — both designed to let multi-billion-parameter models run on a single consumer GPU — go to extra lengths to protect these outliers or calibrate scales more carefully, rather than applying one scale factor to an entire layer and hoping for the best.
Back to the Shop Counter
Return to the shopkeeper in Bihar, phone raised over the medicine label. The translation model she is using was very likely quantized to INT8, or lower, somewhere between the data centre where it was trained and the app store where she downloaded it — the same rounding-and-scaling process worked through by hand earlier in this chapter, applied across millions of weights instead of five. The result is a model roughly a quarter of its original size, so it fit inside a modest app download over a weak connection; a model that runs on her phone's dedicated NPU instead of grinding through the CPU; a model that returns an answer in under a second instead of several; and a model that barely touches the battery, instead of draining it — all while translating the label just as correctly as the full-precision original would have.
Nothing about quantization is exotic once it is seen this way. It is the same instinct as writing "3.85 metres" instead of "3.847293817 metres" for a room that is about to be tiled: work out how much precision a task genuinely needs, discard the rest in a controlled and mathematically bounded way, and spend the space and time saved on making the whole system faster, cheaper, and usable by more people — including someone whose only computer is a Rs. 7,000 phone with a weak internet connection.
Think About It
Think about this: How would you explain quantization: lower precision = speedup to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where quantization: lower precision = speedup is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting quantization: lower precision = speedup to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind quantization: lower precision = speedup, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.