Model Compression: Shrinking Giant Networks

It takes under half a second. You lift your phone, glance at the screen, and it unlocks: no PIN, no fingerprint, just your face. You have probably done this hundreds of times without a second thought, even inside a lift with zero signal bars or on a metro platform where your data connection has dropped completely. In that half-second, a neural network somewhere inside your phone's processor looked at your face, compared it with what it remembers, and decided: yes, this is the same person who set up this device.

Now think about the neural networks you have studied so far in this course. The ones that recognise handwritten digits, classify photographs, or predict the next word in a sentence are usually trained on powerful machines in a data centre: racks of GPUs, tens of gigabytes of memory, a stable power supply, and as much time as the training run needs. A well-known image-classification network called ResNet-50, for example, has roughly 25.6 million parameters, the individual numbers the network learned during training. If each of those numbers is stored the standard way, as a 32-bit floating-point value taking 4 bytes, the model's weights alone add up to:

25.6 million × 4 bytes = 102.4 million bytes ≈ 102.4 MB

That is a real, working figure for one moderately sized image classifier, and networks built for tasks like translation or speech recognition are often several times larger. Your phone's face-unlock system, by contrast, has to respond in well under a second, has to work with no network connection at all (nobody wants their face photographed and uploaded to a server just to check the time), and has to share the device's limited memory and battery with dozens of other apps, without noticeably draining your charge after the fiftieth unlock of the day.

A 100+ MB network that takes a full second to run will not do. So how do engineers take a large, highly accurate network and shrink it down to something that fits comfortably inside a budget smartphone, without throwing away the intelligence it took days of GPU training to build? Three ideas do most of the work: cutting away the parts of a network that barely matter, storing what remains more cheaply, and training smaller networks to imitate bigger ones. Their names are pruning, quantization, and knowledge distillation.

Two Numbers That Decide Everything

Every technique in this chapter attacks one simple relationship:

model size ≈ (number of parameters) × (bytes used to store each parameter)

For the ResNet-50 example above, that was 25.6 million parameters times 4 bytes each. Shrink either number on the right, and the model gets smaller. Pruning reduces the first number by removing parameters that barely matter. Quantization reduces the second number by storing each surviving parameter using fewer bytes. Knowledge distillation takes a more radical approach: rather than trimming the giant network at all, it trains a brand-new, much smaller network to imitate the giant one's behaviour from scratch.

Why not just train a small network directly and skip the giant one entirely? Because bigger networks are usually easier to train well. Extra parameters give the network more capacity to discover useful patterns during training, even if much of that capacity turns out to be redundant once training is finished. It is often easier to train a large, accurate network and then compress it than to train a small network from scratch and hope it reaches the same accuracy on its own. Compression gives you both the accuracy that comes from a large network's capacity and the footprint of a small one.

That footprint matters in concrete ways on a phone. Storage and download size matter because every app update travels over a network connection that, outside major Indian cities, can still be slow or capped by a limited data plan: a 100 MB model bundled into an app is 100 MB someone has to download before the feature even works. Memory matters because a phone's RAM is shared across the operating system, the camera, your messaging apps, and everything else running at the same moment; a feature that needs hundreds of megabytes just to make one prediction will get that memory reclaimed by the operating system before long. Battery matters because every multiplication a neural network performs costs a small amount of energy, and a face-unlock model running fifty times a day cannot afford to be wasteful. Latency, the delay between input and output, matters because a face-unlock that takes three seconds feels broken, and because computing on the device instead of sending a photo to a server removes the round trip entirely, which is also better for privacy.

Pruning: Cutting the Deadwood

A trained neural network almost always contains connections that do very little work. During training, a weight's value shifts based on how much it helps reduce the network's error; some weights end up mattering a great deal, and others end up hovering close to zero, contributing almost nothing to the final prediction no matter what the input is. Pruning finds these near-useless weights and removes them: the neural-network equivalent of a gardener cutting away deadwood so a tree keeps its shape and its fruit while carrying less weight.

The simplest and most common version is magnitude pruning: after training finishes, look at the absolute value of every weight, and set any weight below some threshold to exactly zero. A weight of zero contributes nothing to a neuron's output, so removing it barely changes what the network computes, provided the threshold is chosen carefully and the network is fine-tuned afterward to adjust for the loss.

Let's trace this by hand on a tiny slice of a network: eight weights feeding into a single neuron.

[1.42, -0.03, 0.91, 0.02, -1.85, 0.01, -0.88, -0.04]

Suppose we choose a pruning threshold of 0.1: any weight whose absolute value is smaller than 0.1 gets zeroed out.

• 1.42 → |1.42| = 1.42, which is ≥ 0.1, so it survives.
• -0.03 → |-0.03| = 0.03, which is < 0.1, so it is pruned to 0.
• 0.91 → survives (0.91 ≥ 0.1).
• 0.02 → pruned (0.02 < 0.1).
• -1.85 → survives (1.85 ≥ 0.1).
• 0.01 → pruned (0.01 < 0.1).
• -0.88 → survives (0.88 ≥ 0.1).
• -0.04 → pruned (0.04 < 0.1).

Four of the eight weights survive: [1.42, 0, 0.91, 0, -1.85, 0, -0.88, 0]. That is 50% sparsity, half the connections in this slice are now zero. Do this across an entire network, and it is common to remove anywhere from a third to well over three-quarters of the weights with only a small, recoverable dip in accuracy, especially if the network is fine-tuned for a few more epochs afterward so the surviving weights adjust to carrying the extra responsibility.

There is an important catch. Zeroing out individual weights scattered across a dense matrix, called unstructured pruning, does not automatically make a model smaller or faster, because a regular array still reserves space for every zero unless the software specifically stores the network in a sparse format, recording only the nonzero values and their positions. The alternative is structured pruning: removing entire neurons, channels, or filters at once, so what remains is simply a smaller ordinary network that runs faster and takes less memory on any hardware, with no special support required. Production systems that need guaranteed speedups on ordinary phone chips generally lean toward structured pruning for this reason.

Quantization: Fewer Bits, Almost the Same Brain

Every weight in a neural network is a number, and the format used to store that number matters as much as how many of them there are. By default, most networks are trained using 32-bit floating-point numbers, or FP32, a format that offers a wide range of values and fine-grained precision at a cost of 4 bytes per number. Quantization asks a simple question: does a trained network actually need that much precision to make correct predictions, or would a cheaper, lower-precision format work almost as well?

For most networks, a much cheaper format works almost as well. A common choice is INT8: 8-bit signed integers, taking just 1 byte per number and covering the range -128 to 127. Going from FP32 to INT8 shrinks storage for every weight by a factor of four, simply by changing the format, with no change to the number of parameters at all.

The trick is converting real-valued weights, which can be any decimal number like 1.23 or -0.87, into one of only 256 available integers, while losing as little information as possible. The standard approach is affine quantization. First, find the range of values you need to represent: say, a layer's weights all fall between -2.0 and 2.0. Then compute a scale that maps that continuous range onto the 256 integers from -128 to 127:

scale = (max_value − min_value) / (max_int − min_int) = (2.0 − (−2.0)) / (127 − (−128)) = 4.0 / 255 ≈ 0.0157

To quantize any weight w, divide by the scale and round to the nearest whole number: q = round(w / scale). To use the weight later during inference, reverse the process and dequantize by multiplying back: w' = q × scale.

Let's trace one real weight through the whole pipeline. Take w = 1.23:

• Divide by the scale: 1.23 / 0.0157 ≈ 78.34
• Round to the nearest integer: q = 78
• Store just the number 78 in a single byte, instead of storing 1.23 across four bytes.
• When the network runs, dequantize: w' = 78 × 0.0157 ≈ 1.2246
• Compare to the original: |1.23 − 1.2246| ≈ 0.0054, an error of well under half a percent.

That tiny rounding error is the entire cost of quantization for this one weight. Multiply this saving across 25.6 million weights, as in the ResNet-50 example, and the model's footprint drops from roughly 102.4 MB to roughly 25.6 MB, a quarter of the original size, while every individual weight is off from its true value by, typically, less than a percent. Errors of that size, spread across millions of weights feeding into thousands of neurons, tend to average out rather than compound, which is why quantized networks usually lose only a small fraction of a percentage point of accuracy, if any.

Two practical notes are worth knowing. Quantization can be applied purely after training, called post-training quantization, which is fast and simple, or the network can be trained (or fine-tuned) while simulating the rounding effects of low precision, called quantization-aware training, which usually recovers whatever tiny accuracy gap remains. INT8 is not the only option, either: some deployments use 16-bit floating point as a gentler halfway step, and researchers continue to push into 4-bit and even lower precision for specific use cases.

Knowledge Distillation: Learning from a Teacher's Doubts

Pruning and quantization both start with a trained giant network and cut it down. Knowledge distillation takes a different route: train a small "student" network from scratch, but instead of teaching it using only the correct answers, teach it using the full output of a large, already-trained "teacher" network.

Here is why that matters. Suppose the teacher network is classifying a handwritten digit, and the true answer is "2". A standard training label is a hard label: a single correct answer, with 100% of the "credit" going to the digit 2 and 0% everywhere else. A well-trained teacher network's raw output, before it commits to a final answer, is usually a full probability spread across all ten digits, a soft label. For one particular handwritten "2", it might look something like this:

• digit 2: 0.82
• digit 3: 0.11
• digit 8: 0.04
• digit 7: 0.02
• all other digits combined: 0.01

That distribution says something a hard label never could: this particular "2" was drawn in a way that also looks a little like a "3" and faintly like an "8", real information about which digits get confused with each other, learned by the teacher across its entire training run. Geoffrey Hinton and his colleagues at Google, who formalised this idea in a 2015 paper, called this extra information "dark knowledge": knowledge the teacher has learned that a hard label alone throws away.

During distillation, the student network is trained to match the teacher's soft output as closely as possible, usually alongside the true hard labels too. The student is learning from a richer signal: not "the answer is 2" but "the answer is 2, and here is how confident to be about every alternative." That signal lets it reach an accuracy that would be very difficult for a network of its small size to reach on hard labels alone. The result is a compact network, architected to be small from the very beginning, that has effectively absorbed a distilled version of everything the giant teacher learned.

Distillation is often combined with the other two techniques rather than used alone: train a giant teacher, distil it into a smaller student architecture, then prune and quantize that student further before shipping it to a phone.

Putting It Together: A Compression Pipeline in Code

Every major deep learning framework builds these ideas in as working code you can run today. Here is a small, complete example using PyTorch, applying dynamic quantization to a simple fully connected network of the kind you have built in earlier chapters:

import os
import torch
import torch.nn as nn

class DigitClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)  # e.g. a flattened 28x28 image
        self.fc2 = nn.Linear(256, 10)   # 10 digit classes

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        return self.fc2(x)

def print_size(model, label):
    torch.save(model.state_dict(), "temp_model.p")
    size_mb = os.path.getsize("temp_model.p") / 1e6
    print(f"{label}: {size_mb:.3f} MB")
    os.remove("temp_model.p")

model = DigitClassifier()
model.eval()
print_size(model, "Original FP32 model")

# Convert the Linear layers' weights to INT8
quantized_model = torch.quantization.quantize_dynamic(
    model, {nn.Linear}, dtype=torch.qint8
)
print_size(quantized_model, "Quantized INT8 model")

print_size saves the model's learned parameters to disk and checks the actual file size: an honest, real-world measurement rather than a theoretical one. DigitClassifier has two linear layers: fc1 with 784 × 256 + 256 = 200,960 parameters, and fc2 with 256 × 10 + 10 = 2,570 parameters, for 203,530 parameters in total, so the original FP32 file comes out close to the 0.8 MB mark (203,530 × 4 bytes ≈ 0.81 MB). The single call to torch.quantization.quantize_dynamic replaces every nn.Linear layer's weight tensor with an INT8 version, while leaving the much smaller bias terms in full precision. Running this script shows the saved file shrink to roughly a quarter of its original size, with no retraining, no change to the network's architecture, and no change to how you call the model afterward.

Now trace what a combined pipeline would do to the eight-weight example from earlier. Pruning first removed four of the eight weights, leaving [1.42, 0, 0.91, 0, -1.85, 0, -0.88, 0]. Quantize just the four surviving nonzero weights from FP32 to INT8, and the numbers work out like this:

• Before any compression: 8 weights × 4 bytes (FP32) = 32 bytes.
• After pruning and quantization: 4 surviving weights × 1 byte (INT8) = 4 bytes for the values themselves.

That is an 8x reduction on this tiny slice, from two independent techniques stacking on top of each other. A production system would also need a few extra bits to record which four positions out of the original eight survived pruning, a small, worthwhile overhead when scaled up to millions of weights, and one that specialised sparse-storage formats are built to minimise. Pruning, quantization, and distillation are not competing choices but complementary tools, and real deployment pipelines, the ones that get a face-unlock model small enough to fit comfortably on a budget phone, typically use two or three of them together.

The Trade-off Nobody Skips

None of these techniques are free. Each one, in some sense, throws away information that the original giant network learned: some weights, some precision, or the sheer number of parameters available to represent complex patterns. Push pruning or quantization too aggressively, say, pruning 95% of a network's weights, or quantizing down to 2 bits, and accuracy eventually falls off a cliff rather than degrading gently.

This is why compression in practice is rarely a single step applied blindly. Engineers typically compress in stages, measuring accuracy after each one, and often fine-tune the compressed network afterward: running a few more, cheap, rounds of training so the surviving weights adjust to their new roles, recovering most or all of the accuracy that compression cost. The choice of how far to compress is a genuine trade-off. A face-unlock model might tolerate almost no accuracy loss, since a false unlock is a security problem, while a keyboard model that suggests the next word as you type can tolerate being wrong occasionally, since the user just keeps typing. There is no single correct amount of compression, only the amount appropriate for a given accuracy budget, memory budget, and battery budget.

This is also why model compression is treated as a core part of the deep learning workflow rather than an optional step bolted on at the end. A model that is 99% accurate but never ships because it does not fit on the target device is, in a very real sense, less useful than a 98% accurate model that runs instantly in every user's pocket.

Back to the Locked Screen

Return to that half-second face unlock from the start of this chapter. Whatever network is running behind the scenes almost certainly is nowhere close to 100 MB, does not take a full second to respond, and does not need to phone home to a server before deciding whether to let you in, because somewhere in its development, engineers applied the ideas in this chapter. Perhaps unimportant weights were pruned away. Perhaps every remaining weight was quantized down from 32-bit precision to something far cheaper to store and compute. Perhaps the model that actually ships was distilled from a much larger, more accurate research model, trained to imitate its soft, richly detailed judgments rather than just its final yes-or-no answer.

The giant networks you have learned to build and train in this course are genuinely powerful, and that power is real. But power that only exists on a data-centre GPU is power very few people in India, or anywhere, can actually use in daily life, especially outside the largest cities, on affordable devices, over uneven networks. Model compression is the bridge between a network that works in a research lab and a network that works in someone's pocket: on a train platform with no signal, on a phone that cost a fraction of the machine it was trained on, in well under a second, every single time.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind model compression: shrinking giant networks, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.