Object Detection Architectures and Methods

Drive up to an NHAI toll gantry on a national highway and a camera above the lane has a fraction of a second to answer a question that is subtly different from "what is in this image." It has to answer "where is each vehicle, and what class is it" — because a truck, a car, and a two-wheeler pay different toll rates, and the gantry frequently has three or four vehicles in frame at once, at different distances, some partially hidden behind others. A single label for the whole frame ("truck") is useless. So is a system that only classifies pixels as road-versus-vehicle without telling you which pixels belong to which individual vehicle. What the gantry needs is a list: for every vehicle in the frame, a tight rectangle around it, a class, and a confidence score. That list — a variable-length set of (box, class, confidence) tuples per image — is the actual output contract of object detection, and getting a network to produce it reliably, in real time, is a genuinely different engineering problem from either whole-image classification or per-pixel labeling. This chapter builds the two dominant architectural families that solve it, the geometric machinery (anchor boxes, non-max suppression) both families depend on, and traces every number by hand so the mechanism is not just described but verifiable.

What detection actually has to produce

A classifier's output is fixed in shape: one probability vector over C classes, regardless of what the image contains. Detection's output is not fixed in shape — an image might contain zero vehicles or twelve, and the network has no way to know in advance which. This is the core architectural puzzle every detector solves differently: how do you get a fixed-size neural network (fixed number of layers, fixed weight tensors) to emit a variable number of structured predictions? The two families in this chapter answer that question in almost opposite ways. Two-stage detectors first generate a large, fixed number of candidate regions, then classify and refine each one, throwing most of them away. One-stage detectors densely predict a box (or several) at every location on a grid, all in a single forward pass, and rely on post-processing to prune the flood of overlapping guesses down to one detection per real object. Every detector, of either family, ultimately needs a way to measure "how well does this predicted box match this true box," and that measurement — Intersection over Union — is worth fixing before going further, because both the training-time anchor assignment and the inference-time pruning step are built directly on top of it.

IoU of two boxes is the area of their overlap divided by the area of their union. If a predicted box and the ground-truth box are identical, IoU is 1; if they don't touch, IoU is 0. Detectors use thresholds on this single number to decide, during training, which candidate boxes count as "found the object," and during inference, which of several overlapping detections are duplicates of the same object.

Two-stage detectors: propose, then commit

The R-CNN family solves the variable-count problem by first asking "where might there be an object at all," cheaply and approximately, and only then spending the expensive classification computation on those candidates. R-CNN (Girshick et al., CVPR 2014) used an external, non-learned algorithm called selective search to generate roughly 2,000 candidate regions per image, cropped and warped each one, and ran a full CNN forward pass on every single crop — accurate, but so slow it processed under one image per minute. Fast R-CNN (Girshick, ICCV 2015) fixed the redundant computation: run the CNN once on the whole image to get a shared feature map, then use ROI pooling to crop the relevant patch of that one feature map for each of the ~2,000 proposals instead of re-running the network 2,000 times. That is a large speedup, but selective search itself — a fixed, non-learned proposal generator — was still the bottleneck and still not tuned to the actual objects the network needed to find.

Faster R-CNN (Ren, He, Girshick, and Sun, NeurIPS 2015) replaced selective search with a learned Region Proposal Network (RPN) that shares the same backbone feature map as the classification head, making proposal generation itself trainable and fast. The RPN doesn't propose boxes from nothing — it slides over the feature map and, at every location, asks whether each of several pre-defined reference boxes, called anchors, looks like it contains an object worth refining. This is the origin of the anchor box: not a prediction, but a fixed geometric template the network learns to adjust and score.

Anchor boxes: the geometry a detector reasons over

At every spatial location on the feature map, the RPN places a small set of anchors of different scales and aspect ratios — typically three scales × three ratios, giving nine anchors per location. Each anchor is compared against every ground-truth box in the image using IoU, and Faster R-CNN's assignment rule is exactly two thresholds: an anchor is a positive training example if its IoU with some ground-truth box exceeds 0.7 (or it is the single highest-IoU anchor for that ground truth, as a safety net), a negative example if its IoU with every ground-truth box is below 0.3, and anything in between — an anchor that overlaps a real object moderately but not decisively — is simply ignored during that training step. The reason for having several aspect ratios at all, rather than one square anchor everywhere, is precisely that real objects aren't square: a bus seen from the side is wide and short, a pedestrian is narrow and tall, and a single anchor shape would frequently fail to reach the positive threshold for either.

Here is a fully worked instance. Suppose an RPN cell sits at image position (100, 100) and the true bounding box for a car there is the rectangle from (70, 60) to (150, 140) — an 80×80 pixel box, area 6,400. Three candidate anchors are centered at the same point: anchor A, an 80×80 square, spanning (60, 60) to (140, 140); anchor B, a 120×60 wide rectangle, spanning (40, 70) to (160, 130); anchor C, a 60×120 tall rectangle, spanning (70, 40) to (130, 160). Computing IoU by hand for anchor A: the overlap rectangle runs from (70, 60) to (140, 140), width 70 and height 80, area 5,600; the union is 6,400 + 6,400 − 5,600 = 7,200; IoU = 5,600 / 7,200 ≈ 0.778. For anchor B: the overlap runs from (70, 70) to (150, 130), width 80 and height 60, area 4,800; union is 6,400 + 7,200 − 4,800 = 8,800; IoU = 4,800 / 8,800 ≈ 0.545. Anchor C works out identically by symmetry, also ≈ 0.545. Under the 0.7 / 0.3 rule, only anchor A crosses into positive territory; B and C fall in the ignored band and contribute nothing to this training step, even though both overlap the car by more than half. The code below reproduces these three numbers exactly, so the arithmetic above can be checked rather than trusted.

def compute_iou(box_a, box_b):
    x1 = max(box_a[0], box_b[0])
    y1 = max(box_a[1], box_b[1])
    x2 = min(box_a[2], box_b[2])
    y2 = min(box_a[3], box_b[3])
    inter_w = max(0, x2 - x1)
    inter_h = max(0, y2 - y1)
    intersection = inter_w * inter_h
    area_a = (box_a[2] - box_a[0]) * (box_a[3] - box_a[1])
    area_b = (box_b[2] - box_b[0]) * (box_b[3] - box_b[1])
    return intersection / (area_a + area_b - intersection)

ground_truth = (70, 60, 150, 140)      # the car, 80x80 px
anchor_a     = (60, 60, 140, 140)      # square 80x80
anchor_b     = (40, 70, 160, 130)      # wide 120x60
anchor_c     = (70, 40, 130, 160)      # tall 60x120

for name, box in [("A", anchor_a), ("B", anchor_b), ("C", anchor_c)]:
    print(name, round(compute_iou(box, ground_truth), 3))

# output:
# A 0.778
# B 0.545
# C 0.545

One-stage detectors: predict everything in one pass

YOLO (Redmon, Divvala, Girshick, and Farhadi, CVPR 2016) abandons the propose-then-classify split entirely. It divides the image into an S×S grid and makes every grid cell directly responsible for predicting B candidate boxes plus class probabilities, in a single forward pass — no separate proposal network, no ROI pooling on a second pass. In the original YOLO design, S = 7, B = 2, and the model was trained on PASCAL VOC's C = 20 classes. Each predicted box carries 5 numbers — center x, center y, width, height, and an objectness confidence — so each grid cell's output depth is B × 5 + C = 2 × 5 + 20 = 30, and the full output tensor is 7 × 7 × 30 = 1,470 values per image, produced in one pass through the network rather than thousands of per-region passes. This is the entire source of one-stage detectors' speed advantage: the network commits to its answer everywhere at once instead of first deciding where to look.

SSD (Liu et al., ECCV 2016) keeps the single-pass philosophy but fixes YOLO's weak spot for small objects by predicting from several feature maps at different resolutions simultaneously — coarse, deep layers for large objects, finer, shallower layers for small ones — rather than committing every prediction to one 7×7 grid derived from the deepest layer alone. Both YOLO and SSD, like the RPN, place anchor boxes (YOLO's later versions and SSD both use them explicitly) at each grid cell and regress offsets from those anchors rather than predicting raw coordinates from scratch, because directly regressing pixel coordinates from nothing is a much harder optimization problem than nudging a reasonable starting guess.

Why one-stage detectors used to lag — and why that isn't really about having "one stage"

A common misconception is that one-stage detectors are inherently a speed-for-accuracy trade: fewer computation stages necessarily means a cruder answer, full stop. That's not what actually happened historically, and the distinction matters for reasoning about newer architectures correctly. The real cause, identified by Lin, Goyal, Girshick, He, and Dollár in the RetinaNet paper (Focal Loss for Dense Object Detection, ICCV 2017), is extreme class imbalance. A dense one-stage detector evaluates something on the order of 100,000 candidate anchor locations across an image, and in a typical photo only a handful of those locations actually contain an object — everything else is background. A standard cross-entropy loss, summed over all 100,000 locations, gets overwhelmed by the enormous number of easy, correctly-classified background anchors; their small individual losses add up to dominate the total gradient, drowning out the rare, informative signal from the few genuine object locations. A two-stage detector never faces this problem in the same way, because its RPN has already thrown away the vast majority of background locations before the expensive classification head ever sees them — the second stage trains on a small, curated, roughly balanced set of proposals.

RetinaNet's fix was not to add a second stage; it was to change the loss function. Focal loss multiplies the standard cross-entropy term by a factor that shrinks toward zero for examples the model already classifies confidently and correctly, so easy background anchors contribute almost nothing to the gradient while hard, informative examples keep their full weight. With that one change, a single-pass, dense detector matched and in some settings exceeded two-stage accuracy while keeping the one-pass speed advantage. So the accuracy gap was never an inherent property of "predicting everything in one shot" — it was a fixable imbalance problem in the objective function, and treating one-stage detectors as permanently the "fast but worse" option is exactly the kind of claim that stopped being true once the actual cause was diagnosed and addressed.

Non-max suppression: from a flood of boxes to one detection per object

Whether a detector is two-stage or one-stage, it does not produce one box per object directly — it produces many overlapping boxes around the same object, each with its own confidence, because nearby anchors, nearby grid cells, or nearby proposals all tend to fire on the same visible thing. Non-max suppression (NMS) is the deterministic algorithm that turns that redundant flood into one box per object: sort all candidate boxes for a class by confidence, keep the highest-confidence box, discard every remaining box whose IoU with it exceeds a threshold (they're judged to be duplicates of the same detection), then repeat on whatever survives. NMS is applied separately per class, so a car box and a person box that happen to overlap are never compared against each other — only boxes competing to describe the same object of the same class get suppressed.

Trace it by hand on four candidate "car" boxes: Box1 = (70, 60, 150, 140), confidence 0.95; Box2 = (75, 65, 155, 145), confidence 0.88; Box3 = (65, 55, 145, 135), confidence 0.75; Box4 = (300, 300, 380, 380), confidence 0.60, a genuinely separate vehicle far away in the frame. Sorted by confidence, the order is already Box1, Box2, Box3, Box4. Keep Box1. Compute IoU(Box1, Box2): overlap runs from (75, 65) to (150, 140), 75×75 = 5,625; union is 6,400 + 6,400 − 5,625 = 7,175; IoU ≈ 0.784 — well above a 0.5 threshold, so Box2 is suppressed as a duplicate. IoU(Box1, Box3) works out to the same 0.784 by symmetric geometry, so Box3 is suppressed too. IoU(Box1, Box4) is 0, since the boxes don't overlap at all, so Box4 survives to the next round. With Box2 and Box3 gone, only Box4 remains; it's kept automatically. Final output: Box1 and Box4 — exactly the two real vehicles in the scene, with the near-duplicate detections removed.

def nms(boxes, scores, iou_threshold=0.5):
    order = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
    keep = []
    while order:
        current = order.pop(0)
        keep.append(current)
        order = [i for i in order
                 if compute_iou(boxes[current], boxes[i]) <= iou_threshold]
    return keep
    # compute_iou is the function defined earlier in this chapter

boxes  = [(70, 60, 150, 140), (75, 65, 155, 145),
          (65, 55, 145, 135), (300, 300, 380, 380)]
scores = [0.95, 0.88, 0.75, 0.60]
print(nms(boxes, scores))

# output:
# [0, 3]

Past anchors and NMS: set-based detection

Anchor boxes and NMS are both hand-designed compromises: someone has to choose how many anchor shapes, at what scales, and what IoU threshold suppresses a "duplicate," and getting those hyperparameters wrong for a new dataset silently costs accuracy. DETR (Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko, ECCV 2020) removes both. It treats detection as direct set prediction: a transformer decoder outputs a fixed number of learned "object query" slots in parallel, each one either predicting an object or predicting "nothing here," and training uses a bipartite (Hungarian) matching between predicted and ground-truth boxes so each real object is claimed by exactly one query — which makes NMS unnecessary, because the model was trained never to produce duplicates in the first place. It's a genuinely different way to solve the same variable-count problem this chapter opened with, worth knowing exists even though anchor-based, NMS-based detectors of the kind built above remain the dominant, most deployable choice for real-time systems like a toll gantry.

Active recall

Work through these before reading the answers.

1. A model outputs a single softmax vector over 20 classes for an entire image. Is this object detection? Why or why not, and what would need to be added?
2. Compute the IoU of predicted box P = (20, 20, 100, 100) against ground truth G = (50, 40, 130, 120), where both are (x1, y1, x2, y2) in pixels.
3. In the anchor worked example, suppose the assignment rule changes from "positive if IoU is above 0.7" to "positive if IoU is above 0.5." Which anchors become positive now, and what training problem can this introduce?
4. In the NMS worked example, lower the suppression threshold from 0.5 to 0.3. Does the kept output change? Show why or why not.
5. Now instead raise Box2's confidence from 0.88 to 0.97, keeping the threshold at 0.5. Trace the full NMS output.
6. A classmate says: "YOLO is just a worse, faster version of Faster R-CNN — you always trade accuracy for speed with a single-pass detector." What's wrong with this claim, and what actually explained the historical accuracy gap?

Worked answers

1. No — a single softmax vector over the whole image is whole-image classification. Detection additionally requires localization (a bounding box per instance) and a variable output count, since an image can contain zero, one, or many objects. What's missing is a mechanism producing a list of (box, class, confidence) tuples rather than one fixed-size vector — exactly the problem the RPN-plus-anchors and dense-grid-plus-anchors designs in this chapter solve in two different ways.
2. Intersection: x1 = max(20,50) = 50, y1 = max(20,40) = 40, x2 = min(100,130) = 100, y2 = min(100,120) = 100, giving a 50×60 overlap, area 3,000. Areas of P and G are each 80×80 = 6,400. Union = 6,400 + 6,400 − 3,000 = 9,800. IoU = 3,000 / 9,800 ≈ 0.306.
3. Both B (0.545) and C (0.545) now cross the 0.5 threshold and become positive alongside A (0.778). Three anchors of very different shape are all told "this is the object," even though B and C overlap the car by barely more than half. Training the box regressor to pull these looser, less precise matches toward the ground truth adds noisy, weaker-signal gradient updates that dilute the strong signal from A — precisely why Faster R-CNN keeps a wide ignored band (0.3–0.7) rather than a single cutoff.
4. No change. IoU(Box1, Box2) and IoU(Box1, Box3) are both ≈0.784, already above 0.5, so they were already above 0.3 as well — still suppressed. IoU(Box1, Box4) is 0 (the boxes don't overlap), which stays below 0.3 regardless — Box4 still survives. Output is still {Box1, Box4}. Lowering an NMS threshold only changes the outcome for a pair whose IoU falls between the old and new value; no pair here does.
5. Sorted by confidence, the new order is Box2 (0.97), Box1 (0.95), Box3 (0.75), Box4 (0.60). Keep Box2. IoU(Box2, Box1) is the same 0.784 computed earlier (by symmetry), which exceeds 0.5, so Box1 is suppressed. IoU(Box2, Box3): overlap runs from (75,65) to (145,135), a 70×70 region, area 4,900; union is 6,400 + 6,400 − 4,900 = 7,900; IoU = 4,900 / 7,900 ≈ 0.620 — also above 0.5, so Box3 is suppressed too, but by a newly computed value, not the old 0.784. IoU(Box2, Box4) is still 0, so Box4 survives and is kept next. Final kept set: {Box2 (0.97), Box4 (0.60)} — the same two real objects survive, but the surviving "car" box and the exact suppression numbers both changed.
6. The claim treats the accuracy gap as inherent to having one pass instead of two, but the actual cause (Lin et al., 2017) was extreme foreground/background class imbalance: a dense one-stage detector scores on the order of 100,000 anchor locations per image, and the flood of easy, correctly-classified background anchors overwhelms a standard loss function's gradient. Two-stage detectors dodge this because their RPN already discards most background before the classifier ever sees it. RetinaNet's focal loss fixed the imbalance directly — down-weighting easy negatives in the loss — letting a single-pass detector reach two-stage accuracy without adding a second stage. The gap was a fixable optimization problem, not a fixed property of single-pass architectures.

Think About It

Think about this: How would you explain object detection architectures and methods to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where object detection architectures and methods is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting object detection architectures and methods to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind object detection architectures and methods, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.