Convolutional Neural Networks: From LeNet to ResNet

The cheque that taught a machine to read

In 1998, Yann LeCun's group at AT&T Bell Labs built a network to solve a narrow, unglamorous problem: read the handwritten digits on a bank cheque — the courtesy amount, the account number — accurately enough that a bank could process it without a human retyping it. The network was called LeNet-5, and it was deployed commercially, reading a meaningful share of cheques handled by US banks through the late 1990s. It is the direct ancestor of every convolutional network used today.

India has its own version of this exact problem. The Reserve Bank of India's Cheque Truncation System, rolled out through the 2010s, stopped physically moving cheques between banks and instead moves scanned images — the cheque is truncated at the first bank, and every downstream step works off a picture. Recognising the digits, the signature region, and the MICR line on that scanned image is the same pattern-recognition task LeNet-5 solved in 1998, run now at national scale, on GPUs, by networks that are architecturally LeNet's grandchildren. This chapter traces that lineage — LeNet to AlexNet to VGG to ResNet — and at each step asks a specific engineering question: what broke, and what fixed it.

Why not just flatten the image into a dense network?

You already know the fully connected (dense) network from earlier deep learning chapters: every input unit connects to every unit in the next layer, each with its own weight. Nothing stops you from flattening an image into a long vector and feeding it to one. The reason nobody does this at scale is arithmetic, not principle.

Take a modest 224×224 RGB image. Flattened, that is 224 × 224 × 3 = 150,528 numbers. A single dense layer connecting this to just 1,000 hidden units needs 150,528 × 1,000 = 150,528,000 weights — before you've processed a single meaningful feature. Compare that to one convolutional layer with 64 filters, each 3×3×3 (three input channels): parameters = (3×3×3 + 1 bias) × 64 = 1,792. The dense layer needs more than 84,000 times as many weights to do a much cruder job, because it learns a completely separate weight for "bright pixel in the top-left corner" and "bright pixel in the bottom-right corner," when what you actually want is one detector — an edge, a curve, a blob of colour — applied everywhere the image might contain it.

Convolution encodes two structural assumptions about images that a dense layer ignores entirely: locality (a pixel's meaning depends mostly on its immediate neighbours, not on a pixel 200 rows away) and translation invariance (a cat's ear looks like a cat's ear whether it's in the top-left or bottom-right of the frame, so the same detector should fire in both places). A convolutional layer bakes in both assumptions by using one small kernel and sliding it — weight sharing — across the entire input, rather than learning an independent weight for every pixel position.

The convolution operation, worked by hand

Take a 5×5 grayscale patch and a 3×3 kernel:

img5x5 = [
    [1, 2, 3, 0, 1],
    [0, 1, 2, 3, 1],
    [1, 0, 1, 2, 0],
    [2, 1, 0, 1, 3],
    [0, 2, 1, 0, 1],
]
kernel3x3 = [
    [1, 0, -1],
    [1, 0, -1],
    [1, 0, -1],
]

This kernel is a classic vertical-edge detector: it weights the left column of any 3×3 window positively, ignores the middle column, and weights the right column negatively. A window where the left side is brighter than the right produces a large positive number; a window that is flat produces something near zero.

With a 5×5 input, a 3×3 kernel, stride 1, and no padding, the kernel has exactly 3 valid horizontal positions and 3 valid vertical positions, so the output is 3×3. Sliding it across every position and taking the elementwise product-and-sum (this is technically cross-correlation, not true flipped convolution, but every deep learning framework calls it "convolution" and so will this chapter) at position (0,0):

def convolve(img, k):
    kh, kw = len(k), len(k[0])
    oh, ow = len(img) - kh + 1, len(img[0]) - kw + 1
    out = [[0] * ow for _ in range(oh)]
    for i in range(oh):
        for j in range(ow):
            total = 0
            for u in range(kh):
                for v in range(kw):
                    total += img[i + u][j + v] * k[u][v]
            out[i][j] = total
    return out

print(convolve(img5x5, kernel3x3))
# [[-4, -2, 4], [0, -4, -1], [1, 0, -2]]

Trace the top-left cell by hand to see where −4 comes from. The 3×3 window at (0,0) is rows 0–2, columns 0–2: [[1,2,3],[0,1,2],[1,0,1]]. Multiplying elementwise by the kernel and summing:

(1·1)+(2·0)+(3·−1) + (0·1)+(1·0)+(2·−1) + (1·1)+(0·0)+(1·−1)
= (1+0−3) + (0+0−2) + (1+0−1) = −2 + −2 + 0 = −4

The output feature map that results, [[-4,-2,4],[0,-4,-1],[1,0,-2]], is smaller than the input (5×5 shrank to 3×3) and encodes where vertical intensity drops are strongest — the +4 at position (0,2) marks the window with the sharpest left-bright-to-right-dark transition in the patch. This one filter finds one pattern. A real convolutional layer runs dozens of independently learned filters over the same input and stacks their outputs as channels, so the layer learns a whole vocabulary of local patterns simultaneously.

Padding, stride, and pooling

Three more choices define a convolutional layer beyond the kernel itself. Stride is how far the kernel jumps between positions — stride 1 checks every position, stride 2 skips every other one and roughly halves each spatial dimension. Padding adds a border of zeros around the input so the kernel can center on edge pixels and the output can stay the same size as the input if desired ("same" padding) rather than shrinking ("valid" padding, what the worked example above used). The general formula for output size along one dimension is:

output = floor((W − K + 2P) / S) + 1

where W is input width, K is kernel size, P is padding, S is stride. Check it against the worked example: W=5, K=3, P=0, S=1 → floor((5−3+0)/1)+1 = floor(2)+1 = 3. Matches the 3×3 output above.

Pooling is a separate, parameter-free downsampling step — typically 2×2 max-pooling or average-pooling with stride 2 — that reduces the spatial resolution of a feature map while keeping the most important activations, cutting computation for subsequent layers and giving the network a small amount of translation tolerance (an edge shifted by one pixel still survives a 2×2 max-pool).

LeNet-5: the architecture, layer by layer

LeNet-5 takes a 32×32 grayscale digit (MNIST's 28×28 images are zero-padded to 32×32) and alternates convolution with average pooling before ending in dense layers:

Every shape and parameter count here is derivable, not asserted. C1: (5×5×1 + 1 bias) × 6 filters = 156. Spatial size after C1: floor((32−5)/1)+1 = 28. After S2's 2×2 pool: 14. C3: (5×5×6 + 1) × 16 = 2,416, spatial floor((14−5)/1)+1 = 10. After S4: 5. C5 is the interesting one: the input to C5 is exactly 5×5×16, and the kernel is exactly 5×5, so each of the 120 filters produces a single 1×1 output — C5 is written as a convolution for architectural uniformity, but mathematically it is identical to a dense layer connecting a flattened 5×5×16 = 400-value input to 120 outputs: (5×5×16 + 1) × 120 = 48,120 parameters. F6: (120+1)×84 = 10,164. Output: (84+1)×10 = 850. Total: 61,706 trainable parameters — small enough to train on 1990s hardware in reasonable time, which was the whole point.

# Input: 32x32x1 (MNIST digits, zero-padded from 28x28)
Conv2d(1, 6, kernel_size=5)     # -> 6 x 28 x 28   (32-5+1=28)
AvgPool2d(2, stride=2)          # -> 6 x 14 x 14
Conv2d(6, 16, kernel_size=5)    # -> 16 x 10 x 10  (14-5+1=10)
AvgPool2d(2, stride=2)          # -> 16 x 5 x 5
Conv2d(16, 120, kernel_size=5)  # -> 120 x 1 x 1   (5-5+1=1, acts as Dense)
Flatten()                       # -> 120
Linear(120, 84)                 # -> 84
Linear(84, 10)                  # -> 10   (digit class scores)

AlexNet and VGGNet: going deeper, going smarter

LeNet-5 worked on 32×32 digits. It did not scale to the 224×224 natural photographs of ImageNet — not because convolution stopped working, but because training deep networks on large images with 1990s tools (sigmoid/tanh activations, CPU training, no regularisation for millions of parameters) was infeasible. AlexNet (2012) removed those specific bottlenecks: ReLU activations instead of saturating sigmoids (gradients don't vanish for positive inputs), dropout to regularise a much larger parameter count, and training split across GPUs. It won the 2012 ImageNet competition by a large margin and triggered the deep learning boom in computer vision.

VGGNet (2014) asked a sharper architectural question: instead of one large kernel, what if you stack several small ones? Two consecutive 3×3 convolutions (stride 1, no padding change) have a combined receptive field of 5×5 — each output pixel depends on a 5×5 neighbourhood of the original input, the same reach as a single 5×5 kernel. But the parameter cost differs: for C input and output channels, two stacked 3×3 layers cost 2 × 9C² = 18C² parameters, while one 5×5 layer costs 25C². Stacking three 3×3 layers gives a 7×7 receptive field (general rule for stride-1 stacks: RF = n(K−1) + 1, so 3×(3−1)+1 = 7) at a cost of 27C², versus 49C² for a single 7×7 kernel.

Same receptive field, 28% fewer parameters, and — because each 3×3 layer is followed by its own ReLU — more nonlinear decision boundaries packed into the same reach. VGGNet used nothing but stacked 3×3 convolutions and 2×2 pooling, all the way to 16 or 19 weight layers, and this "small kernel, more depth" recipe became the default template for every architecture after it.

The degradation problem — the misconception depth creates

A natural assumption at this point: if going from LeNet's 5 layers to VGG's 19 improved accuracy, stacking still more layers should keep improving it. This is the specific misconception worth naming and correcting: depth does not monotonically help, and the failure mode of very deep plain networks is not overfitting. If it were overfitting, a deeper network would fit its own training data extremely well while generalising poorly to test data — training error would be low, test error high. What researchers observed instead, when they kept stacking plain convolution-ReLU blocks past roughly 20 layers, was that both training error and test error got worse as depth increased. A 56-layer plain network could have higher error on the very data it was trained on than an 18-layer plain network. That is not a generalisation problem — the optimiser is failing to find a good solution even on the training set, which is a symptom of the optimisation landscape becoming harder to search as gradients propagate through more and more layers, each of which can shrink (or, less commonly, amplify) the signal on the way back. This is the degradation problem, and it is what ResNet was built to fix.

ResNet: skip connections and the gradient highway

He et al.'s insight in 2015 was procedural, not about representational power: a network with residual connections can, in the worst case, learn to behave exactly like a shallower network — so adding layers should never make things worse, provided those extra layers are easy to optimise toward doing nothing. The mechanism is the residual block. Instead of asking a stack of layers to learn the desired mapping H(x) directly, you reparameterise it to learn the residual F(x) = H(x) − x, and reconstruct the output as H(x) = F(x) + x by adding the original input back in via a shortcut that skips the convolutions entirely.

class BasicBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn1   = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn2   = nn.BatchNorm2d(channels)
        self.relu  = nn.ReLU()

    def forward(self, x):
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + identity     # the shortcut
        return self.relu(out)

Why does this fix the degradation problem specifically, rather than just relabelling it? Consider backpropagation through one block. Without the shortcut, the gradient flowing back through the block is dL/dx = dL/dy · dF/dx — a pure product. Stack twenty such blocks and the gradient is a product of twenty Jacobians; if each has magnitude even slightly below 1, the product shrinks toward zero exponentially with depth, and early layers receive almost no learning signal. With the shortcut, y = F(x) + x, so dy/dx = dF/dx + 1, and the gradient becomes:

∂L/∂x = ∂L/∂y · (∂F/∂x + 1) = ∂L/∂y · ∂F/∂x + ∂L/∂y

The second term, ∂L/∂y, passes through completely unchanged regardless of how small ∂F/∂x becomes. Even if a particular block has learned almost nothing useful yet and its local gradient is tiny, the identity path still carries a full-strength gradient signal straight back to earlier layers. Stack a hundred of these blocks and the gradient still has a direct, unattenuated path to the input — the "+1" is a highway that the vanishing-gradient product cannot shut down. This is also why the residual formulation is easier to optimise toward doing nothing: driving F(x) toward zero (push the block's weights toward zero) is a trivial optimisation target, whereas forcing a plain stack of nonlinear conv-ReLU layers to approximate the identity function is not trivial at all, since ReLU and convolution don't naturally compose into "pass the input through unchanged." ResNet, using blocks like this stacked over 34, 50, and even 152 layers, won the 2015 ImageNet competition and made networks of that depth trainable for the first time.

Active recall

Attempt these before reading the worked answers below.

1. Why does a convolutional layer use far fewer parameters than a dense layer processing the same image, even though both eventually need information from every pixel?
2. A 28×28×1 input passes through a conv layer with 8 filters of size 5×5, stride 1, no padding. What is the output shape, and how many parameters does the layer have?
3. In LeNet-5, layer C5 is implemented as a convolution, yet it behaves exactly like a dense layer. Why?
4. What is the degradation problem, and why is it evidence against "the network is just overfitting"?
5. Compute the receptive field after three stacked 3×3, stride-1 convolutional layers, and state the equivalent single kernel size.
6. Why is learning the residual F(x) = H(x) − x an easier optimisation target than learning H(x) directly, when the ideal mapping is close to the identity?

Worked answers

1. A dense layer learns an independent weight for every (input pixel, output unit) pair, so it must relearn "detect an edge" separately for every position an edge could appear. A convolutional layer learns one small kernel and slides it across the whole image (weight sharing), so the same edge detector is reused at every location — for a 224×224×3 input compared against a 3×3×3, 64-filter conv layer, the dense first layer needs over 84,000 times as many parameters (150,528,000 versus 1,792) to represent far less structure.
2. Output spatial size = floor((28−5)/1)+1 = 24, giving 24×24×8. Parameters = (5×5×1 + 1 bias) × 8 filters = 26 × 8 = 208.
3. C5's input is 5×5×16 and its kernel is exactly 5×5, so each filter has only one valid position and produces a single 1×1 output value per filter — every output touches every input value exactly once, which is precisely what a dense layer connecting a flattened 400-value input (5×5×16) to 120 outputs would do. It's kept in convolution notation purely so the whole network can be described and implemented with one operation type.
4. The degradation problem is the empirical observation that once plain (non-residual) convolutional networks get sufficiently deep (tens of layers), adding more layers increases both training error and test error. If the problem were overfitting, training error would stay low while only test error rose — the network would be memorising training data at the cost of generalisation. Instead training error itself worsens, meaning the optimiser is failing to find a good solution even on data it has direct access to, which points to an optimisation-landscape/gradient-flow problem rather than a capacity or generalisation problem.
5. Using RF after n stride-1 layers of kernel size K: RF = n(K−1) + 1. For n=3, K=3: RF = 3×2+1 = 7. Three stacked 3×3 convolutions see the same 7×7 neighbourhood of the input as a single 7×7 kernel would, at roughly 27C² parameters instead of 49C².
6. If the ideal mapping is close to identity, a plain stack of conv-BN-ReLU layers must learn weights that make several nonlinear operations compose into "pass the input through almost unchanged" — nothing about convolution or ReLU makes that combination easy to reach or stable to hold. The residual formulation only needs F(x) to approach zero, which is trivially achieved by driving the block's weights toward zero; the optimiser has to find a small perturbation rather than reconstruct an identity function out of nonlinear parts.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where convolutional neural networks: from lenet to resnet is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting convolutional neural networks: from lenet to resnet to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind convolutional neural networks: from lenet to resnet, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.