Semantic Segmentation in Deep Learning

The crop-insurance problem a bounding box cannot solve

Under the Pradhan Mantri Fasal Bima Yojana, an insurer settling a drought or flood claim needs to know how many hectares of a specific crop were actually damaged inside a farmer's plot. The National Remote Sensing Centre feeds this pipeline with satellite imagery, and the underlying computer-vision task is not "is there a rice field somewhere in this tile" and it is not "draw a box around the rice field." A field boundary in Punjab or the Krishna delta is an irregular polygon that follows a canal, a bund, or a property line inherited from a partition decades ago. A rectangular box drawn around such a field either clips off a damaged corner or drags in a neighbour's undamaged plot, and either error changes a payout. What the claim actually needs is a decision, made independently at every pixel of the image, about whether that pixel is "this crop," "a different crop," "bare soil," or "water" — a label map with the exact same height and width as the input image, not a handful of rectangles.

That task — assign every pixel in an image a class label — is semantic segmentation. It is a different output contract from image classification (one label for the whole image) and from object detection with bounding boxes (a short list of rectangles, each with a class and a confidence score, covering only the objects the detector chose to localize). This chapter builds semantic segmentation from first principles: why ordinary classification CNNs cannot produce a pixel-dense output, how the encoder-decoder design with skip connections (U-Net) fixes that, and how to read the loss functions and evaluation metrics that are unique to dense prediction. A companion chapter on object detection covers bounding-box architectures in depth; this chapter does not repeat that material, because segmentation is solving a structurally different problem even though both tasks start from the same convolutional backbones.

Dense prediction: the output has the same shape as the input

Formally, semantic segmentation learns a function f that maps an image tensor of shape H×W×3 to a label map of shape H×W, where every entry is one of K class indices. Compare the three vision tasks by their output shape alone:

A classification network compresses an image down to a single vector — the last operation in a ResNet or VGG backbone is a global average pool that deliberately destroys spatial information, because for "is this a cat" the location of the whiskers doesn't matter. Segmentation needs the opposite: it needs to keep, or reconstruct, spatial information at the resolution of the original image, because the entire point is to say where. Naively, a classification backbone applied to a full image after removing its final pooling and fully-connected layers still shrinks the feature map: five stages of stride-2 convolution or max-pooling turn a 512×512 image into a 16×16 feature map, a 32× reduction in each spatial dimension. That coarse map has excellent semantic content (each cell "knows" a lot about the 32×32 patch of the original image it summarizes) but has completely lost the ability to say which of those 1,024 original pixels is the field boundary and which is two metres to its left.

Jonathan Long, Evan Shelhamer, and Trevor Darrell's 2015 CVPR paper Fully Convolutional Networks for Semantic Segmentation is the origin point for solving this with a single trainable network: replace the final fully-connected classification layers with 1×1 convolutions (turning the whole network "fully convolutional," so it can accept any input size and emit a spatial map instead of a single vector), then learn an upsampling stage that expands the coarse, semantically rich bottleneck map back up to the input resolution. Olaf Ronneberger, Philipp Fischer, and Thomas Brox's 2015 MICCAI paper U-Net: Convolutional Networks for Biomedical Image Segmentation refined this into the architecture nearly every modern segmentation network still descends from, by making the upsampling path a mirror image of the downsampling path and, critically, wiring direct connections between matching resolutions on the two sides.

Encoder, decoder, bottleneck: the two things a segmentation network must not lose

Split the network into two halves. The encoder (also called the contracting path) is an ordinary CNN stack: repeated 3×3 convolutions with ReLU, interleaved with 2×2 max-pooling that halves the spatial resolution and typically doubles the channel count. Each pooling stage trades spatial precision for a larger receptive field and richer semantic context — by the bottleneck, one "pixel" of the feature map has seen a large patch of the original image and encodes a confident answer to "what kind of region is this," but has no memory of exactly where inside that patch the boundary sits. The decoder (expanding path) reverses the spatial shrinkage using transposed convolution (sometimes called "deconvolution," though it is not a true mathematical inverse of convolution) or learned upsampling, doubling resolution at each stage until the output map matches the input size.

If a decoder only had the bottleneck to work from, it would be reconstructing fine pixel boundaries out of information that was already discarded — like trying to redraw a field's exact edge from a memory that only recorded "there is a green field somewhere around here." This is the core design problem U-Net solves: at every encoder stage, before pooling throws away resolution, the full-resolution feature map at that stage is cached. When the decoder reaches the matching stage on the way back up, that cached map is concatenated, channel-wise, onto the decoder's upsampled feature map. This is the skip connection. It gives the decoder simultaneous access to two different things at every level: coarse semantic context, carried up from the bottleneck through the decoder's own path, and fine spatial detail, carried directly across from the encoder without ever passing through a bottleneck. A convolution immediately after the concatenation lets the network learn how to combine the two.

Tracing a mini U-Net by hand

The arithmetic below is not illustrative pseudo-math — every number was computed and verified with NumPy so that each step is independently checkable. Take a 4×4 single-channel patch representing NDVI-like brightness values, where low values mean bare soil and high values mean healthy crop. The left two columns are non-crop, the right two columns are crop, with a bit of sensor noise:

X =
[0.1  0.1  0.8  0.9]
[0.1  0.2  0.9  0.9]
[0.0  0.1  0.8  0.8]
[0.0  0.0  0.7  0.9]

ground truth Y (1 = crop, 0 = non-crop):
[0  0  1  1]
[0  0  1  1]
[0  0  1  1]
[0  0  1  1]

Step 1 — encoder convolution. Apply one 3×3 kernel (a real network would learn dozens of these per layer in parallel; tracing one filter by hand keeps the arithmetic tractable) that responds to left-to-right gradients, with zero-padding so the output stays 4×4:

K =
[-1  0  1]
[-1  0  1]
[-1  0  1]

Convolving X with K (stride 1, pad 1) and applying ReLU gives the encoder's pre-pooling feature map A:

A (post-ReLU) =
[0.3  1.5  1.5  0.0]
[0.4  2.3  2.2  0.0]
[0.3  2.3  2.3  0.0]
[0.1  1.5  1.6  0.0]

Notice column 3 is entirely zero. That is not an error — it is a real, checkable property of this configuration: column 2 (the true boundary, where soil meets crop) has a strong local gradient and lights up, exactly as a left-right gradient kernel should. Column 3 sits at the image's right edge, so the kernel's rightmost tap reads the zero-padding beyond the image and produces a large negative pre-activation (-1.5 to -2.5 across the four rows) that ReLU clips to zero — not because there is no gradient, but because the padding-boundary effect combined with ReLU erases it, even though those pixels are unambiguously crop. This one line of arithmetic is the whole justification for having a bottleneck at all: edge-sensitive encoder features are excellent at finding boundaries but blind to "what is the interior of this region," which is exactly the kind of coarse, whole-region judgment the pooled, deeper features are good at.

A is cached here — this is the tensor the skip connection will carry forward.

Step 2 — max-pool to the bottleneck. A 2×2, stride-2 max-pool over A gives the bottleneck P:

P =
[2.3  2.2]
[2.3  2.3]

Resolution has dropped from 4×4 to 2×2 — a 4× reduction in pixel count, and the exact spatial position each surviving value came from inside its 2×2 window is now unrecoverable from P alone.

Step 3 — decoder: transposed convolution. Upsample P back to 4×4 with a learned 2×2, stride-2 transposed-convolution kernel Kt = [[1.0, 0.5], [0.5, 0.25]]. Each input value is scaled by every entry of Kt and placed into its own non-overlapping 2×2 block of the output (kernel_size equals stride here, so adjacent blocks do not overlap and there is nothing to sum):

U =
[2.300  1.150  2.200  1.100]
[1.150  0.575  1.100  0.550]
[2.300  1.150  2.300  1.150]
[1.150  0.575  1.150  0.575]

Look at the pattern inside each 2×2 block: the top-left cell always carries the full source value, the diagonal cell always carries a quarter of it. That repeating pattern is a block-tiling artifact specific to this kernel/stride combination: because kernel_size equals stride (2 and 2), there is no overlap between adjacent output blocks, so each non-overlapping 2×2 block is simply one bottleneck scalar times the four fixed weights of Kt — any non-uniform kernel will reproduce its own weight-ratio pattern as a tiled ripple across the output, with nothing to do with the input image. This is a close cousin of, but mechanistically distinct from, the classic checkerboard artifact that Odena, Dumoulin, and Olah documented in their 2016 Distill article Deconvolution and Checkerboard Artifacts: that failure mode arises when the kernel size is not evenly divisible by the stride, so adjacent kernel footprints overlap unevenly — some output positions are covered by two overlapping applications of the kernel and others by only one. The example above has zero overlap by construction, yet still shows a regular, non-physical intensity ripple, for the tiling reason just described. Both failure modes point to the same fix: many production segmentation networks replace transposed convolution with a plain bilinear upsample followed by an ordinary convolution — the upsampling step becomes fixed and artifact-free, and all the learning happens in the convolution afterward.

Step 4 — skip connection: concatenate. The decoder now has U (4×4, carries bottleneck context) and the cached A (4×4, carries the raw edge response). Stack them as two channels: [U ‖ A], shape 4×4×2. A 1×1 convolution — weights w1 = 0.6 on the U channel, w2 = 0.4 on the A channel, bias b = −0.5 — reduces the two channels to one logit per pixel, L = 0.6·U + 0.4·A − 0.5:

L =
[ 1.00   0.79   1.42   0.16]
[ 0.35   0.77   1.04  -0.17]
[ 1.00   1.11   1.80   0.19]
[ 0.23   0.45   0.83  -0.16]

Step 5 — sigmoid and threshold. Apply sigmoid and threshold at 0.5 for this binary crop/non-crop map:

P(crop) =
[0.73  0.69  0.81  0.54]
[0.59  0.68  0.74  0.46]
[0.73  0.75  0.86  0.55]
[0.56  0.61  0.70  0.46]

predicted mask =
[1  1  1  1]
[1  1  1  0]
[1  1  1  1]
[1  1  1  0]

Compared with the ground truth (columns 0–1 non-crop, columns 2–3 crop), the network gets the boundary roughly right but makes two kinds of errors: it over-predicts crop at (row 0, col 0–1) and (row 2, col 0–1) — false positives creeping left of the true boundary — and it under-predicts at (row 1, col 3) and (row 3, col 3) — false negatives inside the crop region, in exactly the interior columns where A was zero. That second error traces directly back to Step 1: the skip path contributes nothing useful at column 3, so wherever the bottleneck path's contribution to L at that pixel is weak, the combined logit dips below the decision threshold. This is a toy, hand-picked, untrained example — a real U-Net trains all of these kernels jointly by gradient descent until they cooperate — but the error pattern is real and reproducible from the numbers above, and it demonstrates precisely why a single filter, or a network that leans too hard on one of its two paths, is not enough.

Loss functions for a pixel-dense output

With Y the ground truth and P(crop) the predicted probability, per-pixel binary cross-entropy is BCE(i,j) = −[Y·log P + (1−Y)·log(1−P)], averaged over all H×W pixels. For the grid above, the mean BCE works out to 0.799 — worse than a coin flip's expected loss of ln 2 ≈ 0.693, which is consistent with a network that has not yet been trained on this exact boundary.

Cross-entropy treats every pixel as an independent classification problem, which creates a real issue in segmentation that classification never faces: severe class imbalance within a single image. A satellite tile where 95% of pixels are background and 5% are a canal is dominated, in a pixel-averaged loss, by how well the network predicts "background" — a network that predicts background everywhere scores a deceptively low average loss while being useless for the one class anyone cares about. Two overlap-based metrics correct for this by only counting the region that matters:

Intersection over Union (IoU)  = |predicted ∩ truth| / |predicted ∪ truth|
Dice coefficient                = 2·|predicted ∩ truth| / (|predicted| + |truth|)

For the predicted mask above: 6 pixels correctly predicted crop (the intersection), the predicted mask covers 14 crop pixels total, the ground truth covers 8, and the union of the two is 16 pixels. That gives IoU = 6/16 = 0.375 and Dice = 2×6/(14+8) = 12/22 ≈ 0.545. Dice is always ≥ IoU for the same prediction (it weighs the intersection twice), and training a segmentation network directly against 1 − Dice ("soft Dice loss"), often summed with pixel-wise cross-entropy, is standard practice precisely because it keeps rare-class boundaries from being drowned out the way plain cross-entropy would drown them.

The mechanism, end to end

Semantic vs. instance segmentation — the misconception to fix

Students who have just seen object detection tend to assume segmentation is a strict upgrade: "detection draws a rough box, segmentation must draw the exact outline of that same object." That is wrong, and it is wrong in a way that matters for real deployments. Plain semantic segmentation assigns one class label per pixel and stops there — it has no concept of "object instance." If a satellite tile contains three adjacent rice paddies belonging to three different farmers, a semantic segmentation network paints all of their pixels the same colour, "rice," and the three parcels are visually indistinguishable in the output; there is no field-1/field-2/field-3 boundary in the label map at all, only a single connected (or disconnected) blob of "rice" pixels. The diagram above makes this concrete: three overlapping green blobs on the left all carry the identical class label, versus three outlined, individually numbered blobs on the right.

Separating individual instances of the same class — three rice paddies, three pedestrians standing next to each other, three overlapping cars in a parking lot — is a different task, instance segmentation, and it is not solved by semantic segmentation alone. It requires combining detection (find each instance and its rough extent) with a per-instance mask prediction, which is why architectures for it (Mask R-CNN being the best known) build directly on top of a detection pipeline rather than on top of a plain U-Net. If your downstream task is "how many separate paddies are damaged and by how much each," a semantic segmentation network's single "rice" blob genuinely cannot answer the question — you would need to additionally run connected-component analysis, which only works if the instances don't touch, or move to an instance segmentation model. If your downstream task is "how many total hectares of rice are damaged, regardless of ownership boundary," semantic segmentation's per-pixel area count is exactly the right and simpler tool. Choosing between the two is a real design decision, not a matter of one being "the better version" of the other.

A minimal, runnable skip connection in code

The trace above worked with a single hand-picked channel; a real network learns many channels per layer and chains several encoder/decoder stages. The core mechanism — cache the pre-pool feature map, concatenate it back in during upsampling — is exactly this, expressed as two PyTorch modules:

import torch
import torch.nn as nn

class EncoderBlock(nn.Module):
    def __init__(self, in_ch, out_ch):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
        )
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)

    def forward(self, x):
        skip = self.conv(x)      # full-resolution feature, cached for the decoder
        down = self.pool(skip)   # halved resolution, passed deeper into the encoder
        return down, skip


class DecoderBlock(nn.Module):
    def __init__(self, in_ch, out_ch):
        super().__init__()
        self.up = nn.ConvTranspose2d(in_ch, out_ch, kernel_size=2, stride=2)
        self.conv = nn.Sequential(
            nn.Conv2d(out_ch * 2, out_ch, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
        )

    def forward(self, x, skip):
        x = self.up(x)                     # upsample: resolution doubles
        x = torch.cat([x, skip], dim=1)     # skip connection: concat on channel axis
        return self.conv(x)


enc = EncoderBlock(in_ch=3, out_ch=16)
dec = DecoderBlock(in_ch=16, out_ch=16)

x = torch.randn(1, 3, 64, 64)
down, skip = enc(x)          # down: (1,16,32,32), skip: (1,16,64,64)
out = dec(down, skip)        # out: (1,16,64,64) -- resolution restored via the skip

Trace the shapes: x enters at (1,3,64,64). The 3×3, padding-1 convolution preserves spatial size, giving skip at (1,16,64,64); the 2×2, stride-2 max-pool halves it to down at (1,16,32,32). In the decoder, the 2×2, stride-2 transposed convolution doubles 32 back to 64, giving (1,16,64,64); concatenating with skip along the channel axis produces (1,32,64,64); the final 3×3 convolution (which expects out_ch * 2 = 32 input channels — matching the concatenation exactly) brings it back to (1,16,64,64). A full U-Net is four or five of these encoder/decoder pairs stacked, with a plain convolutional bottleneck at the deepest point and a final 1×1 convolution mapping the last decoder's channels down to K class logits per pixel.

Active recall

Attempt every question before reading its answer.

1. Why does a standard image classifier (ResNet-style backbone ending in global average pooling and a fully-connected layer) fail as a segmentation network without architectural changes, even if you retrain it on pixel-labelled data?
2. In the worked trace, why was A — the encoder's pre-pool feature map — exactly zero in column 3, and why does that matter for what the skip connection can and cannot contribute?
3. Two overlapping farm plots of the same crop type sit in one satellite tile. A plain semantic segmentation model outputs one connected "crop" region covering both. What went wrong, and what class of model would fix it?
4. Suppose the final 1×1 convolution's weights are changed from w1=0.6 (on U) and w2=0.4 (on A) to w1=0.4, w2=0.6 — shifting emphasis toward the raw skip feature and away from the upsampled bottleneck path, bias unchanged at b=−0.5. Recompute the predicted mask for column 3 (all four rows) and state what happens to Dice.
5. Now suppose the skip connection is removed entirely, so the decoder predicts directly from U alone: L = U − 0.9 (a plain rescaling to keep the same rough operating point). The resulting predicted mask has higher Dice (0.6 vs. 0.545) than the with-skip version above. Does this mean skip connections are unnecessary? Look at the actual mask pattern before answering.
6. Why is Dice loss (or a combined cross-entropy + Dice loss) preferred over plain pixel-wise cross-entropy when a class covers a small fraction of the image, such as a canal in a mostly-background satellite tile?

Answers

1. Global average pooling collapses the entire spatial grid into a single vector per channel before the final classification layer, which is exactly what makes classification robust to where things are — but it permanently discards the pixel-to-pixel spatial layout needed to say which pixel is which class. Even retrained on pixel-labelled data, an architecture with a GAP layer has no path left to produce an H×W output at all; it must be restructured (remove the GAP and FC layers, add an upsampling/decoder path, typically with skip connections) before it can be a segmentation network.

2. The 3×3 kernel used, K = [[-1,0,1],[-1,0,1],[-1,0,1]], is a horizontal-gradient detector: it responds only where pixel intensity changes from left to right within its receptive field. Column 3 sits in the interior of the uniformly bright crop region (values 0.8–0.9 with no strong local contrast, and it is also the padded edge column, where the kernel's rightmost tap sees the zero-padding and produces a negative, ReLU-clipped response), so the filter reports nothing there even though those pixels are unambiguously crop. This means the skip connection cannot single-handedly rescue a prediction at column 3 — whatever the decoder gets right there has to come from the bottleneck/decoder path, not from the skip.

3. Nothing went wrong — this is expected behaviour of semantic segmentation, not a bug. It assigns one label per pixel and has no notion of separate object instances, so two adjacent same-class regions are indistinguishable in its output. Recovering the plot boundary between them requires instance segmentation (detection combined with per-instance mask prediction, e.g. Mask R-CNN), or, if the two plots don't touch, a simpler connected-component pass on top of the semantic mask.

4. With w1=0.4, w2=0.6, recomputing L = 0.4U + 0.6A − 0.5 for column 3 (where A = 0 in every row) gives L = 0.4U − 0.5: row 0 → 0.4(1.1)−0.5=−0.06, row 1 → 0.4(0.55)−0.5=−0.28, row 2 → 0.4(1.15)−0.5=−0.04, row 3 → 0.4(0.575)−0.5=−0.27. All four logits are now negative, so sigmoid(L) < 0.5 everywhere in column 3 and the predicted mask becomes 0 for all four rows there — up from 2 correct predictions and 2 false negatives (the original weighting) to 0 correct predictions and 4 false negatives. Intersection drops from 6 to 4, predicted-crop count drops from 14 to 12, and Dice falls from 2×6/22≈0.545 to 2×4/(12+8)=0.4. Leaning more heavily on the skip feature made things worse here specifically because column 3 is exactly where the skip feature is uninformative (see Q2) — a ripple effect that starts at the weight change and lands entirely on the one column where A carries no signal.

5. No — look at the actual predicted mask, not just the score. Without the skip, the mask is [[1,1,1,1],[1,0,1,0],[1,1,1,1],[1,0,1,0]]: rows 1 and 3 alternate 1,0,1,0 across columns, a checkerboard pattern that does not track the true vertical boundary at all — it is an artifact of the non-uniform transposed-convolution kernel from Step 3, scored well by lucky overlap counting on a 4×4 grid, not by actually finding the edge. The with-skip mask, by contrast, is spatially coherent — every row is either all-crop or crop-then-non-crop, never alternating — because the skip-fused feature suppresses the raw upsampling ripple. This is exactly why skip connections are standard in production segmentation networks despite this cherry-picked toy case scoring in the no-skip version's favour: a single scalar metric on a 16-pixel toy example can reward incoherent noise, but a real network trained end-to-end on many boundaries, not one hand-picked patch, does not get to rely on coincidence, and the skip path is what keeps its predictions anchored to actual image structure rather than to upsampling artifacts.

6. Cross-entropy is averaged per pixel, so a class that covers 5% of the image contributes only about 5% of the total loss regardless of whether the network gets that class right — a network that ignores the small class almost entirely can still post a low average loss. Dice (and IoU) measure overlap only within the region either the prediction or the ground truth claims, so getting the small class wrong directly and visibly tanks the score no matter how much correct background surrounds it. Combining both losses in training keeps the stable, well-behaved gradients cross-entropy provides while forcing the network to actually be accountable for minority-class pixels.

Think About It

Think about this: How would you explain semantic segmentation in deep learning to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind semantic segmentation in deep learning, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.