Residual Connections: Skip and Learn

The Balance That Doesn't Start From Zero

Open a banking app and look at the account balance. Say it reads ₹8,240. At the end of the month, the bank credits a small interest payment, and the number becomes ₹8,265. Notice what the bank does not do. It does not walk back through every transaction in the account's history, every UPI payment for chai, every mobile recharge, every salary credit since the account was opened, and recompute the balance from the very first rupee. It takes the number that is already sitting there and adds a small correction on top of it.

New balance = Old balance + Change.

That looks almost too simple to matter. But this exact pattern, carry forward what already works and compute only the small change on top of it, turns out to be one of the most consequential ideas in the history of deep learning. It is called a residual connection, also called a skip connection, and it is the reason engineers can train neural networks that are dozens or even hundreds of layers deep without the network falling apart during training. Before this idea was introduced in 2015, deep learning had a strange and frustrating problem: making a network deeper often made it perform worse, not better. This chapter explains why that happened, and how one addition sign fixed it.

Why Deeper Should Mean Better

A neural network learns by stacking layers, each one transforming its input a little before passing it to the next. In image recognition, early layers might detect edges and colours, middle layers might combine these into textures and shapes, and later layers might combine those into whole objects: a cricket bat, an autorickshaw, a dosa. It seems obvious that a network with more layers should be at least as capable as a shallower one, since it has more room to represent complicated ideas.

There is a simple argument for why a deeper network should never be worse than a shallower one. Suppose a network already works well, with some number of layers. Now stack a few extra layers on top of it. In principle, those extra layers could just learn to copy their input straight through to their output, doing nothing at all. Mathematically, this is called an identity mapping. If the extra layers simply pass their input through unchanged, the deeper network behaves exactly like the shallower one and loses nothing. So a deeper network should, at worst, match a shallower one, and at best, do better by learning something extra.

When researchers actually built and trained very deep plain networks, plain meaning ordinary stacks of layers with no special tricks, they found the opposite. Networks with more layers frequently had higher training error than their shallower counterparts. Not testing error, which could be blamed on overfitting. Training error: the error measured on the very data the network was learning from. A 34-layer plain network could end up worse at fitting its own training data than an 18-layer version of the same design. Something was preventing the deeper network from even learning the easy option of "do what the shallow network does, plus nothing." Kaiming He and his colleagues at Microsoft Research, who documented this clearly in 2015, named it the degradation problem: accuracy gets saturated as depth increases and then degrades rapidly, and this degradation is not caused by overfitting.

The Trouble With Long Chains of Multiplication

To see why depth causes trouble, look at what happens during backpropagation, when a network works out how much each weight contributed to the final error and adjusts it. This adjustment signal, the gradient, has to travel backward from the output layer all the way to the first layer, passing through every layer in between. By the chain rule, the gradient that reaches an early layer is the product of the local derivatives of every layer it passes through.

Products of many numbers smaller than 1 shrink quickly. If each layer's local derivative is around 0.6, then after five layers the signal reaching an earlier layer is roughly 0.6⁵ ≈ 0.078, under 8% of its original strength. After ten layers it is roughly 0.6¹⁰ ≈ 0.006, about half a percent. Push this out to fifty or a hundred layers, typical for a serious image-recognition network, and the gradient reaching the earliest layers becomes so small that, in floating-point arithmetic, it is effectively zero. Those early layers stop receiving any useful signal about how to improve and effectively stop learning. This is the classic vanishing gradient problem.

It is worth being precise here, because deep learning researchers already had tools for fighting vanishing gradients in 2015: careful weight initialisation, and a technique called batch normalisation, which rescales the activations flowing through a network so they do not shrink or blow up layer after layer. Kaiming He's team used batch normalisation in their plain networks and still saw the degradation problem appear. That told them something more general than simple gradient shrinkage was making very deep plain networks hard to optimise; the error surface itself seemed to become harder to navigate as depth increased, even when gradients were reasonably well-behaved at initialisation. Their fix did not just patch the gradient-shrinking symptom. It changed what each layer was being asked to learn in the first place.

Learn the Difference, Not the Whole Thing

Call the function a stack of layers is supposed to compute H(x), the ideal transformation from input x to output. In a plain network, each block of layers tries to learn H(x) directly. Residual connections change the question. Instead of asking a block to learn H(x) from scratch, they ask it to learn the difference between the output and the input:

F(x) = H(x) − x

and then reconstruct the real output by adding the input back:

H(x) = F(x) + x

F(x) is called the residual function. It is only responsible for the leftover part, the correction, not the whole answer. The path that carries x forward unchanged and adds it back in is the shortcut connection (or skip connection). In a real network, a block computes F(x) using a couple of convolutional layers, and a separate wire carries the original x around those layers to be added at the end.

This is exactly the passbook idea from the start of the chapter, with one important refinement. A bank's interest credit is computed from the existing balance, a percentage of what is already there, not attached independently the way a fixed transaction amount is. In the same way, a residual block's correction F(x) is computed by looking at x itself; it is the layer's own judgment, based on the input it received, about what small adjustment is worth making. The output is never asked to reinvent x from nothing. It only has to describe what should change.

Why does this reframing help? Return to the identity-mapping argument: if the best thing a block could do is nothing at all, pass its input straight through, a plain block has to learn a stack of weighted, non-linear layers that exactly reproduces the identity function, which is a surprisingly awkward target to hit with gradient descent. A residual block, by contrast, only has to learn to output zero. If F(x) = 0, then H(x) = 0 + x = x, the identity, automatically. Pushing a set of weights toward zero is one of the easiest things gradient-based training does; it is what regularisation techniques like weight decay already encourage. Residual connections turn "please reproduce the input exactly" into "feel free to do nothing," and doing nothing is easy to learn.

Tracing the Numbers, Layer by Layer

The clearest way to see the effect is to trace actual numbers through a tiny stack of layers, once without a shortcut and once with one. To keep the arithmetic simple, imagine each "layer" here has just a single weight, rather than a full matrix of weights. The underlying mechanism scales up unchanged to real networks with thousands of weights per layer.

Let the input be x = 2.0, and let three consecutive layers each use the small weight w = 0.1. Each plain layer computes h = ReLU(w × h_previous), where ReLU keeps positive values unchanged and zeroes out negative ones. Since every value here stays positive, ReLU behaves as a simple pass-through, so the local derivative of each layer is just w.

Plain network, forward pass:

• h0 = 2.0 (the input)
• h1 = ReLU(0.1 × 2.0) = 0.2
• h2 = ReLU(0.1 × 0.2) = 0.02
• h3 = ReLU(0.1 × 0.02) = 0.002

The signal has shrunk from 2.0 to 0.002 in three layers, down to a tenth of a percent of its starting strength. Now trace the gradient backward through the same network. Suppose the gradient arriving at the last layer, ∂L/∂h3, is 1: a clean unit of "how much the loss cares about this output." Since each layer's local derivative is w = 0.1, the chain rule multiplies this local derivative in at every step going backward.

Plain network, backward pass:

• ∂L/∂h2 = 1 × 0.1 = 0.1
• ∂L/∂h1 = 0.1 × 0.1 = 0.01
• ∂L/∂h0 = 0.01 × 0.1 = 0.001

Only 0.1% of the original gradient reaches the input. Now redo the exact same three layers with a shortcut connection added to each one, so every layer adds a correction to its input instead of replacing it: h = h_previous + ReLU(w × h_previous).

Residual network, forward pass:

• h0 = 2.0
• h1 = 2.0 + ReLU(0.1 × 2.0) = 2.0 + 0.2 = 2.2
• h2 = 2.2 + ReLU(0.1 × 2.2) = 2.2 + 0.22 = 2.42
• h3 = 2.42 + ReLU(0.1 × 2.42) = 2.42 + 0.242 = 2.662

The local derivative of a residual layer is 1 + w instead of just w, because differentiating h_previous + F(h_previous) gives 1 + F′(h_previous). The shortcut path contributes a clean, unconditional 1 alongside whatever the residual branch contributes.

Residual network, backward pass:

• ∂L/∂h2 = 1 × (1 + 0.1) = 1.1
• ∂L/∂h1 = 1.1 × 1.1 = 1.21
• ∂L/∂h0 = 1.21 × 1.1 = 1.331

The gradient reaching the input is now 1.331, above its starting value, instead of 0.001. Extend this pattern out to ten layers using the same weight, and the gap becomes dramatic: a plain network's gradient shrinks to roughly 0.1¹⁰, one ten-billionth of its original size, indistinguishable from zero on any real computer. A residual network's gradient, following 1.1¹⁰, comes out to about 2.59, still a healthy, informative signal. The shortcut path guarantees a floor under the gradient that no amount of depth can fully erase, because that path always contributes an unconditional 1 to the local derivative at every layer, regardless of how small or poorly tuned the residual branch's own contribution happens to be.

Checking the Arithmetic in Code

The hand calculation above can be verified directly by running it.

def relu(z):
    return max(0, z)

w = [0.1, 0.1, 0.1]   # same three weights, used by both networks
x = 2.0

# Forward pass: plain network, no shortcut
h_plain = [x]
for wi in w:
    h_plain.append(relu(wi * h_plain[-1]))

# Forward pass: residual network, shortcut at every layer
h_res = [x]
for wi in w:
    h_res.append(h_res[-1] + relu(wi * h_res[-1]))

print("Plain activations:   ", [round(v, 6) for v in h_plain])
print("Residual activations:", [round(v, 6) for v in h_res])

# Backward pass: gradient reaching the input, starting from dL/dh3 = 1
grad_plain = 1.0
for wi in reversed(w):
    grad_plain *= wi          # plain layer's local derivative is w

grad_res = 1.0
for wi in reversed(w):
    grad_res *= (1 + wi)      # residual layer's local derivative is 1 + w

print("Gradient at input, plain:   ", round(grad_plain, 6))
print("Gradient at input, residual:", round(grad_res, 6))

Running this prints [2.0, 0.2, 0.02, 0.002] for the plain activations and [2.0, 2.2, 2.42, 2.662] for the residual ones, followed by a gradient of 0.001 at the input for the plain network and 1.331 for the residual one, exactly matching the hand trace above. Turning the same idea into a real deep learning framework takes very little extra code.

What a Residual Block Looks Like in Practice

A real residual block, as used in a convolutional network for image recognition, replaces the toy single-weight layers above with actual convolutional layers, but the shortcut idea is identical:

            x
            │
            ├─────────────────┐
            │                 │
       [ Conv 3×3 ]           │
       [ BatchNorm ]          │   shortcut
       [    ReLU   ]          │  (skip connection)
            │                 │
       [ Conv 3×3 ]           │
       [ BatchNorm ]          │
            │                 │
            └────────(+)◄─────┘
                      │
                  [  ReLU  ]
                      │
                      y

In PyTorch, this block is only a few lines:

import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
        self.bn1   = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
        self.bn2   = nn.BatchNorm2d(channels)
        self.relu  = nn.ReLU()

    def forward(self, x):
        identity = x                     # keep the input for the shortcut
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + identity             # the skip connection itself
        return self.relu(out)

Everything before out = out + identity is an ordinary pair of convolutional layers; this is the residual function F(x) being computed. That single addition is the entire architectural change that separates a residual block from a plain one. When the input and output of a block have different shapes, for instance when a network reduces the image's height and width or changes the number of channels partway through, the shortcut cannot add x directly, because the shapes will not match. In that case, the shortcut passes x through a small 1×1 convolution first, just enough to reshape it, before adding. This is called a projection shortcut, as opposed to the plain identity shortcut used when shapes already match.

152 Layers, One Competition

This idea was introduced by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun at Microsoft Research in their 2015 paper "Deep Residual Learning for Image Recognition." The networks built this way are called ResNets, typically named after their depth: ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152, where the number counts the layers with learnable weights. For the deeper versions, each block is redesigned as a bottleneck block, a 1×1 convolution to reduce the number of channels, a 3×3 convolution to do the main work cheaply on that reduced representation, and another 1×1 convolution to expand the channels back, which keeps the computational cost manageable even at great depth.

The result was not a modest improvement. An ensemble of residual networks, including a model 152 layers deep, eight times the depth of the 19-layer VGG network that had been near the practical limit just a year earlier, won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2015 classification task with a top-5 error rate of just 3.57%, meaning the correct label was missing from the model's five most confident guesses less than four times in every hundred images. Residual networks were also the foundation of winning entries that year in ImageNet object detection, ImageNet localisation, and COCO object detection and segmentation. Depth, once a liability, had become a straightforward source of accuracy, because the shortcut connections kept those extra layers trainable.

The same paper tested how far this could be pushed on a smaller dataset, CIFAR-10, training a residual network with over a thousand layers. It trained successfully; its training error stayed low even at that extreme depth, confirming that the optimisation problem really had been fixed. Its test accuracy, however, was slightly worse than a 110-layer version of the same design, most likely because a network with that many parameters was simply too large for CIFAR-10's relatively small training set and began to overfit. That result is a useful reminder: residual connections solve the problem of making very deep networks trainable. Whether more depth actually helps a given task still depends on having enough data to justify it.

The Idea Outgrew ResNet

Residual connections turned out to be far bigger than one image-recognition architecture. Earlier that same year, a related architecture called Highway Networks had explored a similar idea using a learned gate to control how much of a layer's input should pass through unchanged. ResNet's contribution was showing that a much simpler, fixed shortcut, with no gate to learn at all, worked just as well or better, while being easier to train at extreme depth.

The idea then spread well beyond convolutional networks. The Transformer architecture, introduced in 2017 and now the basis of large language models, wraps a residual connection around every one of its attention and feed-forward sub-layers: the output of each sub-layer is added back to its own input before being passed on, the same x + F(x) pattern traced earlier in this chapter. Every time a modern language model processes a sentence, it is leaning on the same shortcut-and-correction idea introduced for recognising images a decade earlier. In India, this same building block sits inside systems used for tasks like screening chest X-rays, reading handwritten text in Indian scripts, and detecting crop disease from a photograph taken on a farmer's phone: anywhere a network needs real depth to notice fine detail without becoming impossible to train.

Back to Your Balance

Return to the banking app from the start of the chapter. Every balance update on that screen follows the same shape: keep what was already correct, and add only the small, computed change. A residual network asks every one of its layers to behave the same way. Instead of demanding that each layer rebuild a good representation of the input from nothing, it lets the representation carry forward untouched through the shortcut, and asks the layer only to contribute whatever small correction it can confidently compute. Multiply that across fifty, a hundred, or a hundred and fifty-two layers, and the difference between recomputing everything and adding a correction is the difference between a network that cannot be trained at all and one that could recognise a photograph better than any network that came before it.

The next time a number on a screen updates by a small amount instead of being recalculated from a long history, that is worth noticing. It is the same principle, in different clothing, that let deep learning stop being limited by depth.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind residual connections: skip and learn, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.