Transfer Learning: Standing on Giants' Shoulders

The 200-photo problem

A startup building a crop-disease app for Indian farmers — the kind of tool a smallholder in Nashik points at a tomato leaf to find out if it has early blight — has a data problem before it has a modelling problem. The team can collect perhaps 200 labelled photographs: 100 healthy leaves, 100 blighted ones, shot on farmer phones in real light, real dirt, real motion blur. That is the entire training set.

Suppose they build a small convolutional network from scratch — four convolutional blocks feeding a dense classifier, the standard shape every G11 student has already implemented — and train it end to end on those 200 images. Training accuracy climbs to 100% within a few dozen epochs. Validation accuracy sits at 52%, barely above a coin flip. This is not a bug in the code. It is what happens when a network with far more free parameters than training examples is given the freedom to memorise instead of generalise. Section 3 below turns that sentence into an exact number.

The fix is not "collect a million tomato-leaf photos" — no Indian agri-tech startup has that budget. The fix is transfer learning: take a network someone else already trained on 1.28 million photos of a thousand unrelated categories (dogs, chairs, umbrellas — ImageNet has no "early blight" class at all), keep almost everything that network learned, and retrain only a small piece of it on the 200 leaf photos. This chapter derives why that works, exactly how much it saves in trainable parameters, when it fails, and how to decide how much of a pretrained network to keep frozen versus retrain.

Why a network trained on dogs and umbrellas helps you classify leaves

A convolutional network is a composition of functions. Write a trained classifier as

f(x) = h( g(x) )

where g is the stack of convolutional blocks — the "backbone" — and h is the final classification layer. The backbone's job is to turn a raw pixel grid into a compact, informative representation; the head's job is to draw a decision boundary through that representation for a specific set of classes.

What does each convolutional block inside g actually learn? Visualising the filters of a trained CNN layer by layer (a standard exercise since the original AlexNet paper, and studied systematically by Yosinski et al., "How transferable are features in deep neural networks?", NeurIPS 2014) shows a consistent hierarchy:

• Layer 1 learns oriented edge detectors and colour-opponent blobs — filters that look almost identical whether the network was trained on ImageNet, on faces, or on satellite tiles. Edges and colour gradients are a universal property of natural images, not a property of "dogs" or "leaves."
• Layer 2 combines edges into textures and corners — cross-hatching, grain, simple contours. Still domain-general: a blighted leaf and a rusted metal sheet share texture statistics.
• Layer 3 combines textures into parts and localised patterns — the kind of mid-level shape that starts to be somewhat class-relevant, but is still reusable across very different final tasks.
• The final layers combine parts into whole-object, class-specific responses — "this activation pattern means Labrador," which is genuinely tied to the source task and does not transfer as directly.

Yosinski's key experimental result was to freeze the first n layers of a network trained on one half of ImageNet's classes, transplant them into a network for the other half, and measure accuracy as n increases. Performance stayed close to a fully-trained baseline through the early and middle layers, then degraded as more of the task-specific final layers were forced to stay frozen — confirming the hierarchy above is not just a visual impression but a measurable transferability gradient. That gradient is the entire theoretical basis of transfer learning: general-purpose visual structure (edges, textures, parts) is learned once at enormous scale on ImageNet and reused for free; only the last, task-specific mapping needs to be learned again, on your 200 images.

Two transfer strategies, precisely

Given a pretrained g (backbone) and a new task, there are two disciplined ways to reuse it:

Feature extraction. Freeze every weight in g — set requires_grad = False, or equivalently exclude those parameters from the optimiser entirely. Replace h with a fresh, randomly initialised head sized for your classes, and train only h. The backbone becomes a fixed function that turns an image into a feature vector; you are training a (possibly quite shallow) classifier on top of features someone else spent thousands of GPU-hours computing.

Fine-tuning. Initialise g with the pretrained weights, but continue updating some or all of them, jointly with the new head, using a small learning rate. This lets the backbone adapt its representations toward the new domain instead of using ImageNet's representations unchanged.

Which one to use depends on two independent variables: how much labelled target data you have, and how similar the target domain is to the source domain (ImageNet, in the usual case). A widely used heuristic, standard in transfer-learning practice, lays out the four combinations:

Notice the strategy is never "always freeze everything" and never "always fine-tune everything." Both are wrong in different quadrants of this table — the misconception section below names exactly this error.

Worked example 1 — the parameter count that explains the overfitting

Return to the from-scratch network that scored 52% on 200 leaf photos. Give it a concrete architecture — four 3×3 convolutional blocks over RGB input, followed by global average pooling and a two-class dense head:

Block 1: Conv3x3,  in=3,   out=32   -> params = (3*3*3   + 1) * 32  =    896
Block 2: Conv3x3,  in=32,  out=64   -> params = (3*3*32  + 1) * 64  = 18,496
Block 3: Conv3x3,  in=64,  out=128  -> params = (3*3*64  + 1) * 128 = 73,856
Block 4: Conv3x3,  in=128, out=256  -> params = (3*3*128 + 1) * 256 = 295,168
GlobalAvgPool -> Dense(256 -> 2)    -> params = (256 + 1) * 2       =    514
                                                Total trainable = 388,930

The parameter formula for a convolutional layer is (kernel_h × kernel_w × in_channels + 1) × out_channels — the "+1" per output channel is its bias term, and each output channel needs its own full 3×3×in_channels filter. Trained from scratch, this network has 388,930 free parameters looking at 200 training images: 1,944.65 parameters per example. A model that flexible, with that little data to constrain it, has more than enough capacity to assign an essentially arbitrary label to every training photo — which is precisely what 100% training accuracy alongside 52% validation accuracy means.

Now freeze Blocks 1–4 at their pretrained (ImageNet) values and train only the head:

Trainable parameters = 514  (just the head)
Frozen parameters     = 388,416  (Blocks 1-4, reused unchanged)
Parameters per training example = 514 / 200 = 2.57

That is a reduction from 388,930 to 514 trainable parameters — roughly 757× fewer (388,930 / 514 ≈ 756.67, i.e. about 757 times fewer). Going from 1,944.65 down to 2.57 parameters per training example is the difference between a model with almost no constraint on what it can memorise and one with barely enough freedom to draw a sensible boundary through 512-dimensional feature space. The features themselves — edges, textures, leaf-vein patterns — were never re-derived from the 200 photos at all; they came pretrained, from 1.28 million unrelated images.

Worked example 2 — freezing a real pretrained network in code

The toy network above makes the arithmetic transparent; production code does the identical thing to a real, much larger pretrained network. ResNet-18, a standard image classifier pretrained on ImageNet, has 11,689,512 total parameters, and its final layer (fc) maps a 512-dimensional feature vector to ImageNet's 1,000 classes:

import torch
import torch.nn as nn
from torchvision import models

# Load a CNN pretrained on ImageNet (1.28M images, 1000 classes)
backbone = models.resnet18(weights="IMAGENET1K_V1")

# Freeze every parameter -- these are the general-purpose
# edge / texture / part filters we reuse exactly as-is
for param in backbone.parameters():
    param.requires_grad = False

# Replace the 1000-way ImageNet head with a fresh 2-way head
# for "healthy" vs "blighted" -- only this layer trains
num_features = backbone.fc.in_features        # 512
backbone.fc = nn.Linear(num_features, 2)

trainable = sum(p.numel() for p in backbone.parameters() if p.requires_grad)
total     = sum(p.numel() for p in backbone.parameters())
print(f"{trainable:,} trainable out of {total:,} total")
# 1,026 trainable out of 11,177,538 total

Tracing the arithmetic: the original fc layer had 512 × 1000 + 1000 = 513,000 parameters, all discarded. The new fc layer has 512 × 2 + 2 = 1,026 parameters, all trainable. Total parameters after the swap: 11,689,512 − 513,000 + 1,026 = 11,177,538. Only the new head's 1,026 parameters have requires_grad = True — a reduction of roughly 10,894× relative to training the whole network (11,177,538 / 1,026 ≈ 10,894.3), an even larger saving than the toy example because a real pretrained backbone is so much bigger relative to a two-class head.

Once the head has learned something sensible (its loss has stopped falling sharply), a second stage can cautiously unfreeze the last block for domain adaptation, using a much smaller learning rate for the backbone than for the head:

# Stage 2: gently adapt the last block to leaf photos specifically
for param in backbone.layer4.parameters():
    param.requires_grad = True

optimizer = torch.optim.Adam([
    {"params": backbone.fc.parameters(),     "lr": 1e-3},
    {"params": backbone.layer4.parameters(), "lr": 1e-5},
])

The 100× gap between the two learning rates is deliberate, not cosmetic. layer4's weights already sit in a region of parameter space that produces useful, ImageNet-quality features; a large gradient step computed from a head that starts out badly wrong (it is freshly initialised) would drag those weights far from that good region before the head has learned anything useful — destroying the very representation you paid to keep, an effect generally called catastrophic forgetting. A tiny learning rate lets the backbone drift toward the target domain slowly, while the head, which has nothing to lose, is allowed to move fast.

This two-stage recipe — freeze-and-train-head, then selectively fine-tune with a small learning rate — is the outline behind crop-disease identification apps used across Indian agriculture today. The engineering reason such tools work with training sets of a few hundred to a few thousand field photos, rather than the millions ImageNet required, is exactly worked example 1 and 2 above: the 388,930 (or 11.2 million) parameters needed to learn "what edges, textures, and shapes look like in general" were paid for once, by someone else, at ImageNet scale, and never need to be paid for again.

The mechanism, end to end

The diagram below traces the toy four-block network from worked example 1: a photo flows through four convolutional blocks whose weights are frozen at pretrained values, then through a newly attached head trained on the 200 leaf photos.

The misconception to unlearn

The error nearly every student makes on first meeting transfer learning is believing that freezing more of the backbone is always the safer, more conservative choice — that since freezing worked so well in the small-and-similar quadrant of the decision matrix, freezing everything must be the universally cautious default. It is not. Freezing is only safe when the source and target domains are visually similar enough that ImageNet's features are actually relevant to the target task.

Consider a target task on a genuinely different kind of image — a chest X-ray, or a Sentinel satellite tile. These images share almost none of ImageNet's visual statistics: no natural-photo lighting, no RGB colour semantics (X-rays are single-channel intensity, not colour), no everyday object silhouettes. A backbone frozen at ImageNet's values will hand this task features tuned to detect "fur texture" and "sky-versus-ground colour gradients" — largely irrelevant to detecting a hairline fracture or a flooded field. Freezing the whole backbone here is not the safe choice; it silently caps accuracy at whatever a fixed, wrong-domain feature extractor can support, no matter how well the head is trained on top of it. The correct response in that quadrant, exactly as the decision matrix states, is to fine-tune more of the network — accepting the higher overfitting risk that comes with more trainable parameters, because the alternative (frozen but irrelevant features) has a worse ceiling, not a lower risk. "Freeze more" is not a synonym for "safer"; it is a synonym for "assumes the two domains are similar," and that assumption has to be checked, not defaulted to.

Active recall

Attempt these before reading the answers.

1. The from-scratch toy CNN has 388,930 trainable parameters and is shown 200 training images. State this ratio as "parameters per training example," and explain in one sentence why that ratio predicts the 100%-train / 52%-validation gap observed.
2. In the ResNet-18 code example, trainable parameters drop from 11,689,512 to 1,026 after freezing the backbone and replacing the head. Compute the reduction factor and explain, in terms of what requires_grad = False does to the optimiser, why it is exactly this number and not some other.
3. A hospital in Coimbatore has 50,000 labelled chest X-rays and wants to detect a specific lung condition. X-rays look nothing like ImageNet's natural photos. Using the decision matrix, which quadrant does this fall into, and what strategy does it recommend? Justify why "freeze everything" would be a poor choice here even with 50,000 examples.
4. During Stage 2 fine-tuning, the head uses learning rate 1e-3 and layer4 uses 1e-5 — a 100× gap. Explain, using the idea of catastrophic forgetting, what would go wrong if both were trained at 1e-3 from the start of Stage 2.
5. A classmate says: "Transfer learning means the pretrained model already understands leaf disease before you even train it, so you barely need labelled data at all." What is wrong with this claim, precisely?
6. In the toy CNN, instead of freezing all four blocks, unfreeze only Block 4 and the head, keeping Blocks 1–3 frozen. Compute the new trainable parameter count.

Answers

1. 388,930 / 200 = 1,944.65 parameters per training example. With roughly two thousand degrees of freedom available for every single labelled example, the network has more than enough capacity to fit an arbitrary function that reproduces every training label exactly (hence 100% training accuracy) without that function bearing any necessary relationship to the true healthy/blighted decision boundary — hence near-chance validation accuracy.
2. 11,177,538 / 1,026 ≈ 10,894×. Setting requires_grad = False excludes those parameters' gradients from being computed and from being passed to the optimiser, so no update ever reaches them — the count of parameters that remain True after that loop, plus the freshly created head, is exactly 1,026: the head's own 512 × 2 + 2 parameters, with every other one of the network's 11.68 million weights left untouched at its pretrained value.
3. Large dataset (50,000, "tens of thousands+") crossed with "very different domain" — the recommendation is to fine-tune the entire network, using the pretrained ImageNet weights only as an initialisation rather than a fixed feature source. Freezing everything here would cap the model at ImageNet-style features (natural-photo edges and textures) that are a poor match for X-ray intensity patterns, and with 50,000 labelled examples there is more than enough data to safely re-shape even the earliest filters toward what actually distinguishes the target condition — so freezing would be needlessly conservative and would leave accuracy on the table.
4. The head starts with random weights and therefore an initially large, noisy loss and correspondingly large gradients. If layer4 shared the head's 1e-3 learning rate, those large early gradients would immediately push layer4's pretrained weights a long distance from the good representation ImageNet training found — overwriting useful structure before the head has learned anything sensible to backpropagate. That destructive overwriting of previously learned representations is catastrophic forgetting; the 100× smaller learning rate keeps the backbone's drift slow enough that it only adapts, rather than being overwritten.
5. Wrong on two counts. First, ImageNet contains no leaf-disease class at all, so nothing in the pretrained network has ever produced a "blighted leaf" decision — the network has learned generic visual structure (edges, textures, parts), not the target labels. Second, the classification head is initialised randomly and has learned nothing about any class until it is trained on your labelled data — "zero training on your data" would produce an untrained, effectively random 2-way classifier bolted onto good features, not a working diagnosis. You still need labelled examples; transfer learning reduces how many you need, from millions to hundreds, not to zero.
6. Block 4 (295,168) + head (514) = 295,682 trainable parameters, with Blocks 1–3's combined 93,248 parameters (896 + 18,496 + 73,856) staying frozen. This sits between the two extremes computed earlier (514 when only the head trains, 388,930 when everything trains) — exactly the kind of intermediate point the "small, moderately different domain" cell of the decision matrix recommends.

Think About It

Think about this: How would you explain transfer learning: standing on giants' shoulders to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind transfer learning: standing on giants' shoulders, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.