In August 2018, when the Periyar and Pamba rivers overtopped their banks across Kerala, ISRO's National Remote Sensing Centre had to answer a very specific question from disaster-response teams within hours: not "is this satellite tile flooded?" but "which exact pixels, out of every square metre in this scene, are underwater right now?" The difference between those two questions is the difference between two entirely different computer vision tasks, and understanding why one CNN architecture cannot answer both is the starting point for everything in this chapter. A classifier can tell you a 224×224 tile is "flooded" with 97% confidence. It cannot tell you that 6.4 square kilometres of it are underwater and 3.6 are not, because a classifier collapses an entire image into one label. To draw a flood-extent map you need a prediction for every single pixel — water or land — which is a structurally different problem called semantic segmentation. Compare this to a second, equally real problem: a Railway Protection Force camera on a crowded platform that must count how many people are standing near the edge, so it can trigger a safety alert past a threshold. Here a pixel-wise "person / not-person" mask is nearly useless, because if forty people are standing shoulder to shoulder, their pixels merge into one undifferentiated blob with no boundary between individuals. What you need instead is a bounding box and a confidence score around each discrete person — object detection. These two "advanced" tasks, detection and segmentation, both build on the convolutional feature extractors you already know, but they attach fundamentally different output heads and loss functions to solve fundamentally different problems. This chapter builds both from first principles, with every number checked by hand.
Receptive fields: why deep and narrow beats shallow and wide
Before detection or segmentation can work, the convolutional backbone underneath them has to see a large enough region of the input to recognise objects that span many pixels — not just the 3×3 or 5×5 patch a single filter looks at. The region of the input image that influences one output unit is called its receptive field, and it grows as you stack layers. For a stack of n convolutional layers, each with kernel size k and stride 1, the receptive field at the input is RF = 1 + n(k − 1). Stack two 3×3 convolutions and you get RF = 1 + 2(2) = 5 — the same receptive field as a single 5×5 convolution. This is not a coincidence graders should wave past; it is the exact design argument behind VGGNet, and it matters because the two options cost very different numbers of parameters.
Take 64 input channels and 64 output channels throughout, ignoring bias terms. A single 5×5 convolutional layer has 5 × 5 × 64 × 64 = 102,400 learnable weights. Two stacked 3×3 layers have 2 × (3 × 3 × 64 × 64) = 2 × 36,864 = 73,728 weights — 28,672 fewer, a 28% reduction — while covering the identical 5×5 receptive field and adding an extra ReLU nonlinearity in between, which strictly increases the function class the network can represent. This is why every modern CNN backbone (VGG, ResNet, EfficientNet) is built from stacks of small 3×3 filters rather than a few large ones: you get equal or greater representational reach for fewer parameters and less compute, at the cost of one more layer of depth.
The other quantity worth pinning down exactly is how spatial size shrinks as feature maps pass through convolution and pooling. For input width W, kernel K, padding P, and stride S, the output width is O = ⌊(W − K + 2P) / S⌋ + 1. Trace a 224×224 satellite tile through a "same"-padded 3×3 convolution (stride 1, padding 1) and then a 2×2 max-pool (stride 2, no padding):
def conv_output_size(W, K, P, S):
return (W - K + 2 * P) // S + 1
print(conv_output_size(224, 3, 1, 1)) # convolution, same padding
print(conv_output_size(224, 2, 0, 2)) # max-pool, stride 2
print(conv_output_size(112, 2, 0, 2)) # second pool
print(conv_output_size(56, 2, 0, 2)) # third pool
Tracing by hand: the 3×3 same-padded convolution gives (224 − 3 + 2)/1 + 1 = 224, unchanged, as designed. The first pool gives (224 − 2)/2 + 1 = 112. The second gives (112 − 2)/2 + 1 = 56. The third gives (56 − 2)/2 + 1 = 28. So the program prints 224, 112, 56, 28 — the four resolution stages every encoder in this chapter's diagram passes through, and channel depth typically doubles at each halving (64 → 128 → 256 → 512) to compensate for the lost spatial information with richer per-location features.
Object detection: why exhaustive search doesn't work, and what replaces it
The naive way to detect objects is to slide a classifier window across the image at every position and every scale and ask "is there an object centred here?" This does not scale. Slide a 64×64 window with stride 8 across a 224×224 image and the number of positions is ((224 − 64)/8 + 1)² = (160/8 + 1)² = 21² = 441 — for one window size only. Real objects appear at many scales and aspect ratios, so a practical exhaustive search needs on the order of ten to twenty scale/ratio combinations, pushing the count into the thousands of forward passes per image. That cost is why every modern detector instead computes a single shared CNN feature map over the whole image once, then evaluates a small fixed set of candidate boxes — called anchor boxes — at each cell of that feature map in one pass. A 7×7 feature map with 9 anchors per cell (3 scales × 3 aspect ratios) yields only 7 × 7 × 9 = 441 candidate boxes evaluated from a single backbone pass, not thousands of separate backbone passes.
Each anchor produces a predicted box and a confidence score, and the network is trained against whichever ground-truth box it overlaps best with, measured by Intersection over Union (IoU): the area where predicted and true boxes overlap, divided by the area their union covers. For predicted box A = (x1=100, y1=100, x2=300, y2=300) and ground truth B = (x1=110, y1=105, x2=305, y2=295) — coordinates in pixels, corners top-left to bottom-right — the overlap rectangle has x from max(100,110)=110 to min(300,305)=300, and y from max(100,105)=105 to min(300,295)=295. That's a 190×190 intersection, area 36,100. Box A's own area is 200×200 = 40,000; box B's is 195×190 = 37,050. Union is 40,000 + 37,050 − 36,100 = 40,950. IoU is 36,100 / 40,950 ≈ 0.8816.
def iou(boxA, boxB):
xA = max(boxA[0], boxB[0])
yA = max(boxA[1], boxB[1])
xB = min(boxA[2], boxB[2])
yB = min(boxA[3], boxB[3])
inter = max(0, xB - xA) * max(0, yB - yA)
areaA = (boxA[2] - boxA[0]) * (boxA[3] - boxA[1])
areaB = (boxB[2] - boxB[0]) * (boxB[3] - boxB[1])
return inter / float(areaA + areaB - inter)
print(round(iou((100, 100, 300, 300), (110, 105, 305, 295)), 4))
That prints 0.8816, matching the hand trace exactly. A detector will produce many overlapping boxes around the same real object because nearby anchor cells all fire on it. Non-Maximum Suppression (NMS) cleans this up: sort all candidate boxes by confidence, keep the highest-confidence box, discard every remaining box whose IoU with it exceeds a threshold (they are treated as duplicates of the same object), then repeat with whatever survives. Trace it on three detections of a car — Box1 confidence 0.95 at (100,100,300,300), Box2 confidence 0.88 at (110,105,305,295), Box3 confidence 0.75 at (400,400,500,500) — with threshold 0.5. Sorted by confidence: Box1, Box2, Box3. Keep Box1. Compare it against Box2: IoU is the 0.8816 computed above, which exceeds 0.5, so Box2 is suppressed as a duplicate. Compare Box1 against Box3: the boxes don't overlap at all (Box3 starts at x=400, well past Box1's x2=300), so IoU = 0. That's below the threshold, so Box3 survives as a genuinely separate detection. Final output: Box1 (0.95) and Box3 (0.75).
Semantic segmentation and the U-Net architecture
Detection answers "where are the discrete objects and how many?" Segmentation answers "what is at every pixel?" — exactly what the Kerala flood-mapping problem needed. The architecture that made pixel-dense prediction practical is U-Net: an encoder that repeatedly downsamples the image (exactly the 224→112→56→28 pipeline traced above, extracting increasingly abstract features while losing precise spatial location), followed by a decoder that upsamples back to full resolution to produce a per-pixel mask.
Upsampling is done with a transposed convolution, a learnable operation whose output size follows O = (I − 1)S − 2P + K. Reversing the three pooling steps with stride-2, kernel-2, padding-0 transposed convolutions: from the 28×28 bottleneck, O = (28−1)×2 + 2 = 56; from there, O = (56−1)×2 + 2 = 112; from there, O = (112−1)×2 + 2 = 224 — back to full resolution, exactly mirroring the encoder's downsampling in reverse.
But upsampling alone produces blurry, poorly localised masks, because the bottleneck at 28×28 has genuinely thrown away fine spatial detail — you cannot recover where an edge sat if that information was never carried through. U-Net's fix is the skip connection: at each decoder stage, the encoder's feature map from the matching resolution (before it was pooled away) is concatenated onto the upsampled decoder feature map, and a convolution then fuses the two. The encoder's 56×56 features (extracted before the third pooling) get concatenated with the decoder's upsampled-to-56×56 features; same at 112×112 and 224×224. This gives the decoder access to both "what" (the deep, abstract, low-resolution features that know this pixel belongs to a "flooded region") and "where" (the shallow, high-resolution features that know exactly which pixel that is) — which is precisely why U-Net-style networks, not plain classifiers, are what NRSC-style pipelines use to turn a satellite tile into a flood-extent map accurate enough to guide boat rescues to the correct street.
A misconception worth correcting: what "raising the threshold" does in NMS
Students consistently assume that raising the IoU threshold in Non-Maximum Suppression makes it "stricter," and that stricter must mean more aggressive removal of duplicate boxes. It is the opposite. The threshold is the overlap level above which a box gets discarded as a duplicate — raise it, and you require boxes to overlap almost completely before either is removed, so fewer boxes get suppressed and more near-duplicate detections survive. Lower it, and even loosely overlapping boxes get treated as duplicates, so more get suppressed. Revisit the car example: Box1 and Box2 had IoU 0.8816. At threshold 0.5, that exceeds the threshold, so Box2 is suppressed. Raise the threshold to 0.9: now 0.8816 is below 0.9, so Box2 survives — the detector now reports two overlapping boxes for the same car instead of one. A threshold of 0.9 is the loose, permissive setting; a threshold of 0.3 is the strict, aggressive one. "Higher threshold" and "more suppression" point in opposite directions, and getting this backwards is exactly the kind of error that produces silently wrong evaluation code — a detector that looks like it is under-suppressing duplicates when in fact the engineer just set the threshold the wrong way round.
Active recall
Attempt these before reading the answers below.
- Compute the IoU of box A = (0, 0, 100, 100) and box B = (50, 0, 150, 100).
- A 128×128 image passes through a 5×5 convolution with stride 1, padding 0, then two successive 2×2 max-pools (stride 2, padding 0). What is the final spatial size?
- Compare parameter counts (32 input and output channels throughout, no bias) for a single 7×7 convolution versus three stacked 3×3 convolutions with the same receptive field. Which uses fewer parameters, and by how many?
- Three candidate boxes for one object: A (confidence 0.9), B (confidence 0.6, IoU with A = 0.4), C (confidence 0.85, IoU with A = 0.55). Run NMS with threshold 0.5. Which boxes survive?
- A railway platform safety camera needs to report the exact count of people standing within two metres of the platform edge. Should the system use semantic segmentation or object detection, and why?
- If an engineer raises an NMS IoU threshold from 0.5 to 0.9 and the number of duplicate boxes reported per object goes up, is that expected behaviour or a bug?
Answer 1. Intersection x-range: max(0,50)=50 to min(100,150)=100, width 50. y-range: max(0,0)=0 to min(100,100)=100, height 100. Intersection area = 50 × 100 = 5,000. Each box has area 10,000, so union = 10,000 + 10,000 − 5,000 = 15,000. IoU = 5,000 / 15,000 = 0.333.
Answer 2. Conv: (128 − 5)/1 + 1 = 124. First pool: (124 − 2)/2 + 1 = 62. Second pool: (62 − 2)/2 + 1 = 31. Final size: 31×31.
Answer 3. Three stacked 3×3 convolutions give receptive field 1 + 3(3−1) = 7, matching a single 7×7 convolution. Single 7×7: 7 × 7 × 32 × 32 = 50,176 parameters. Three 3×3 layers: 3 × (3 × 3 × 32 × 32) = 3 × 9,216 = 27,648. The stacked version uses 22,528 fewer parameters — about 45% fewer — for the same receptive field, plus two extra nonlinearities.
Answer 4. Sort by confidence: A (0.9), C (0.85), B (0.6). Keep A. Compare A to C: IoU 0.55 > 0.5, so C is suppressed. Compare A to B: IoU 0.4 < 0.5, so B is not suppressed by A, and no higher-confidence box remains to test it against, so B survives. Surviving boxes: A (0.9) and B (0.6).
Answer 5. Object detection. Semantic segmentation only labels each pixel's class ("person" or "not person") without separating one person's pixels from an adjacent person's — when people stand shoulder to shoulder the mask merges into one undifferentiated region with no count information. Detection produces a separate bounding box and confidence per individual, so the boxes can simply be counted.
Answer 6. Expected behaviour, not a bug. Raising the threshold makes suppression more permissive (only near-identical boxes, with very high overlap, get removed), so more near-duplicate boxes survive per object — the opposite of what "stricter-sounding" language suggests.
Think About It
Think about this: How would you explain advanced computer vision to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where advanced computer vision is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting advanced computer vision to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind advanced computer vision, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.