Computer Vision in Production: YOLO and Faster R-CNN

It is 8:40 in the morning at a busy signal on a six-lane road in an Indian city. Two-wheelers are backed up four deep, autos are wedging into gaps that barely exist, and somewhere in that churn a rider is going without a helmet, another bike is carrying three people, and a delivery rider has just crept past the stop line on amber. A camera mounted on the signal pole is watching all of it, and within seconds an automated system has drawn a box around the helmetless rider's head, another box around that bike's number plate, read the plate characters, matched them against the vehicle registration database, and queued an e-challan, with no traffic constable needed to physically flag the rider down. Versions of this pipeline already run at busy junctions in several Indian cities, processing live video around the clock.

Strip away the plate-reading and the database lookup, and the core problem the camera has to solve, on every single frame, is this: given one image that might contain fifteen vehicles and thirty road users, find every object worth caring about, draw a tight box around each one, and say what it is, all within a fraction of a second, continuously, forever. That problem is called object detection, and it is a meaningfully harder problem than the image classification you have already studied, where a model looks at one photo and outputs a single label such as "cat" or "no-helmet." A junction camera cannot get away with one label per frame; it has to localize and classify every object at once, fast enough to keep up with live video. This chapter looks at the two architectures that, between them, define how production object detection actually gets built: Faster R-CNN, which favours accuracy by examining an image in two careful passes, and YOLO ("You Only Look Once"), which favours speed by looking exactly once.

From "What" to "What, and Where"

An image classifier answers one question: what is the dominant thing in this picture? Object detection has to answer two questions for every object present in a scene: what is it, and where exactly is it? "Where" is expressed as a bounding box, usually written as the pixel coordinates of its top-left and bottom-right corners, (x1, y1, x2, y2). A detector's output for one frame is a list of such boxes, each tagged with a class label ("helmet", "no-helmet", "number-plate") and a confidence score between 0 and 1.

To know whether a predicted box is actually a good match for a real object, detectors need a way to measure how well two rectangles overlap. That measure is Intersection over Union (IoU): the area where the predicted box and the ground-truth box overlap, divided by the total area the two boxes cover between them.

IoU = area(prediction ∩ ground truth) / area(prediction ∪ ground truth)

IoU always falls between 0 (no overlap) and 1 (a perfect pixel-for-pixel match). An IoU of 0.5 or higher against the ground truth is the usual bar for calling a detection correct during evaluation, and the same calculation is the engine behind cleaning up duplicate detections in production, which is exactly what the worked example later in this chapter traces by hand.

Two Philosophies, One Problem

Every modern production detector solves the localize-and-classify problem with one of two broad strategies.

A two-stage detector first asks "where might there be an object at all?", generating a shortlist of candidate regions, and only then asks "what exactly is in each candidate region?" Splitting the work this way lets the second stage focus its full attention on a small number of promising regions instead of the whole image, which tends to produce more accurate boxes and labels. Faster R-CNN is the defining architecture in this family.

A one-stage detector skips the shortlist step and predicts boxes and class labels directly from the raw image in a single forward pass through one network. There is no separate propose-then-classify pipeline; everything happens at once, which is dramatically faster because the network runs only once per image instead of once per candidate region. YOLO is the architecture that established this family.

Neither approach is universally better. They sit at different points on a speed-versus-accuracy curve, and which one belongs in a given production system depends on the constraint that actually matters: a live traffic camera cares about frames per second, while a system auditing scanned documents overnight can afford to be slow in exchange for squeezing out every last point of accuracy. Knowing both architectures well enough to tell which job calls for which is itself the production skill.

Inside Faster R-CNN: Look Twice, Carefully

Faster R-CNN, introduced by Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun in 2015, is the third and fastest member of a lineage that started with a much slower idea. The original R-CNN (2014) generated close to two thousand candidate regions per image using a classical, non-learned algorithm called Selective Search, then ran a full convolutional network separately on every one of those regions to classify it. It worked, but at roughly 47 seconds per image on a GPU, it was far too slow for anything resembling production. Fast R-CNN (2015) sped up the classification step by running the CNN once on the whole image and pooling features for each region from a single shared feature map, instead of re-running the network per region. It still depended on Selective Search to generate those regions in the first place, though, and Selective Search ran on the CPU, taking about two seconds per image and becoming the new bottleneck.

Faster R-CNN's contribution was to remove Selective Search entirely and replace it with a small neural network of its own: the Region Proposal Network (RPN). The RPN slides over the same convolutional feature map that the classification stage uses, so the two stages share the expensive backbone computation (typically a network such as VGG or ResNet, pretrained on ImageNet), and generating proposals costs almost no extra work. At every position on the feature map, the RPN considers a fixed set of reference boxes called anchor boxes: nine per position in the original paper, formed from three scales crossed with three aspect ratios. For each anchor, the RPN predicts an objectness score (does this look like it contains some object, of any class?) and four regression offsets that nudge the anchor's coordinates toward a tighter fit. Anchors with a high objectness score, after their offsets are applied, become the region proposals passed forward; typically a few hundred survive once the weakest candidates are discarded.

Those proposals then reach the second stage. RoI Pooling takes each proposed region, which can be any size since real objects vary in size, and crops and resizes its features into a fixed-size grid, so that every proposal produces a feature map of identical shape regardless of its original dimensions. That fixed-size feature map flows through fully connected layers that output a final class label, one of the real classes or "background" if the proposal turned out to contain nothing useful, and a second, more precise set of box offsets. This is what makes Faster R-CNN a two-stage detector in the fullest sense: stage one, the RPN, roughly says "something is probably here"; stage two looks again, carefully, and says exactly what it is and exactly where its edges are. That second look is what gives Faster R-CNN its accuracy advantage, and its cost. Even with proposal generation folded into the shared network, the original paper reported about 5 frames per second on a contemporary GPU using a VGG-16 backbone, respectable but well short of what a live 25-30 fps camera feed needs from a detector that has to keep up in real time.

In practice, nobody builds Faster R-CNN from scratch for a project like this; production teams load a backbone already pretrained on a large dataset and fine-tune the detection heads on their own labelled violation data. A minimal inference call, once a model is trained, looks like this:

import torch
from torchvision.models.detection import (
    fasterrcnn_resnet50_fpn, FasterRCNN_ResNet50_FPN_Weights
)
from torchvision.transforms.functional import to_tensor
from PIL import Image

weights = FasterRCNN_ResNet50_FPN_Weights.DEFAULT
model = fasterrcnn_resnet50_fpn(weights=weights)
model.eval()

frame = Image.open("junction_frame.jpg").convert("RGB")
image_tensor = to_tensor(frame)

with torch.no_grad():
    predictions = model([image_tensor])

boxes = predictions[0]["boxes"]
labels = predictions[0]["labels"]
scores = predictions[0]["scores"]

for box, label, score in zip(boxes, labels, scores):
    if score > 0.7:
        print(f"class {label.item()}, confidence {score.item():.2f}, box {box.tolist()}")

Notice the shape of the output: a list of boxes, labels, and scores for the image, exactly the raw material Non-Maximum Suppression consumes, and precisely the result the RPN-then-RoI-pooling pipeline described above was built to produce.

Inside YOLO: Look Once, Everywhere at Once

YOLO, introduced by Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi in 2015, discards the idea of proposals altogether. YOLO divides the input image into an S × S grid — the original paper used a 7×7 grid on a 448×448 input — and makes every grid cell directly responsible for detecting any object whose center falls inside it. In one forward pass, each cell predicts the coordinates of a fixed number of candidate boxes relative to itself, a confidence score for each box, and a set of class probabilities for whatever it believes it is looking at. Run the image through the network exactly once, decode this grid of predictions, and every detection in the image falls out; no second pass, no separate proposal network.

The confidence score each cell predicts is defined precisely as Pr(object) × IoU(predicted box, ground truth box), using exactly the IoU measure defined earlier in this chapter: the network is trained to output a high number only when it believes both that something is really there and that its proposed box is a tight fit. At inference time this confidence is multiplied by the predicted class probability to get a class-specific score for each box, low-scoring boxes are dropped, and, because a single object near a grid boundary is often "seen" by more than one neighbouring cell, the survivors still need to be cleaned up with the same duplicate-removal procedure worked through step by step later in this chapter, called Non-Maximum Suppression.

The original 2015 YOLO traded away some accuracy for its speed, particularly on small or tightly clustered objects — exactly what a busy Indian traffic junction produces, with several two-wheelers packed close together. YOLOv2, also called YOLO9000 (2016), closed much of that gap by predicting boxes relative to a set of anchor boxes whose shapes were chosen by running k-means clustering on the box shapes actually found in the training data, rather than predicting raw coordinates from scratch, borrowing the anchor idea from the two-stage world while keeping single-pass speed. YOLOv3 (2018) went further, predicting detections at three different grid resolutions at once, so both large objects like cars and small ones like a distant helmet could be caught by a grid cell of an appropriate size. After Redmon stepped back from computer vision research, the YOLO lineage was carried forward by other research groups and by Ultralytics, whose PyTorch-based releases starting with YOLOv5 became a de facto industry standard for exactly the kind of real-time deployment a traffic-camera pipeline needs. The lineage keeps evolving, but the core idea from 2015 has not changed: one grid, one forward pass, one shot.

Worked Example: Cleaning Up Duplicate Detections with Non-Maximum Suppression

Both architectures above eventually need the same cleanup step, because both tend to produce more than one box for the same real object: Faster R-CNN because several overlapping anchors at slightly different scales all fire on the same region, YOLO because an object parked near a grid-cell boundary can trigger predictions from more than one neighbouring cell. Non-Maximum Suppression (NMS) is the standard algorithm used to collapse a pile of overlapping raw detections down to one box per real object, and it leans entirely on the IoU calculation introduced earlier.

Suppose a detector processing one frame from the junction camera outputs four raw "no-helmet-rider" detections, all above the confidence threshold, each written as (x1, y1, x2, y2, confidence):

• Box A: (100, 100, 220, 260), confidence 0.91, around the head and shoulders of rider 1
• Box B: (112, 108, 228, 264), confidence 0.85, a second, slightly shifted box, also around rider 1
• Box C: (108, 96, 224, 258), confidence 0.62, a third box, also around rider 1
• Box D: (400, 150, 500, 300), confidence 0.77, around rider 2, a genuinely different motorcycle elsewhere in the frame

Three of these boxes describe the same rider, and NMS has to work that out using only geometry and confidence; it has no notion of "rider 1" versus "rider 2." Start with box A against box B. The intersection rectangle has corners x1 = max(100, 112) = 112, y1 = max(100, 108) = 108, x2 = min(220, 228) = 220, and y2 = min(260, 264) = 260, giving an intersection width of 220 - 112 = 108 and height of 260 - 108 = 152, so intersection area = 108 × 152 = 16,416. Box A's own area is 120 × 160 = 19,200 and box B's is 116 × 156 = 18,096, so the union area is 19,200 + 18,096 - 16,416 = 20,880. That gives:

IoU(A, B) = 16,416 / 20,880 ≈ 0.786

The same arithmetic for A against C gives an intersection area of 112 × 158 = 17,696 against a union of 20,296, for IoU(A, C) ≈ 0.872. Both comfortably clear the standard IoU threshold of 0.5, confirming what the raw coordinates already suggest: B and C are near-duplicates of A. Box D, on the other hand, spans x from 400 to 500 while A spans only 100 to 220; the two rectangles never overlap on the x-axis at all, so IoU(A, D) = 0.

NMS now runs as follows: sort all boxes by confidence in descending order, repeatedly take the highest-confidence box still standing, keep it as a final detection, and discard every remaining box whose IoU against it exceeds the threshold (0.5 here), since those are almost certainly duplicates of the object just kept.

Sorted by confidence: A (0.91), B (0.85), D (0.77), C (0.62)

Step 1: keep A (highest confidence)
        IoU(A, B) = 0.786 > 0.5  -> discard B
        IoU(A, D) = 0.000 <= 0.5 -> D survives this round
        IoU(A, C) = 0.872 > 0.5  -> discard C
        remaining: [D]

Step 2: keep D (only one left)
        remaining: []

Final kept boxes: A (0.91), D (0.77)

Two real motorcycles were in the frame, and NMS correctly reduces four raw boxes down to exactly two, one per rider, purely from geometry and confidence. The same logic, written as code, produces exactly this result:

def iou(box_a, box_b):
    xa1, ya1, xa2, ya2 = box_a
    xb1, yb1, xb2, yb2 = box_b

    inter_x1 = max(xa1, xb1)
    inter_y1 = max(ya1, yb1)
    inter_x2 = min(xa2, xb2)
    inter_y2 = min(ya2, yb2)

    inter_w = max(0, inter_x2 - inter_x1)
    inter_h = max(0, inter_y2 - inter_y1)
    inter_area = inter_w * inter_h

    area_a = (xa2 - xa1) * (ya2 - ya1)
    area_b = (xb2 - xb1) * (yb2 - yb1)

    return inter_area / (area_a + area_b - inter_area)


def non_max_suppression(detections, iou_threshold=0.5):
    detections = sorted(detections, key=lambda d: d[1], reverse=True)
    keep = []

    while detections:
        best_box, best_score = detections.pop(0)
        keep.append((best_box, best_score))
        detections = [
            (box, score) for box, score in detections
            if iou(best_box, box) <= iou_threshold
        ]

    return keep


detections = [
    ((100, 100, 220, 260), 0.91),  # rider 1, box A
    ((112, 108, 228, 264), 0.85),  # rider 1, box B (duplicate)
    ((108, 96, 224, 258), 0.62),   # rider 1, box C (duplicate)
    ((400, 150, 500, 300), 0.77),  # rider 2, box D
]

for box, score in non_max_suppression(detections):
    print(box, score)

# (100, 100, 220, 260) 0.91
# (400, 150, 500, 300) 0.77

This is not a minor implementation detail. NMS runs on every frame, on every detector, in every production object detection system, whether it is Faster R-CNN's final classification scores or YOLO's grid predictions being cleaned up. A detector that skipped this step would report three or four e-challans for a single real violation.

Choosing (and Deploying) the Right Detector

In production, the choice between architectures like these is rarely about which one is "more accurate" in the abstract. It is about matching the architecture to the constraint that actually bites. Two numbers dominate the conversation: speed, usually measured in frames per second, and accuracy, usually measured as mean Average Precision (mAP). mAP averages, across all object classes, how well the ranked list of detections for each class matches the ground truth at a chosen IoU threshold, commonly 0.5. A live camera feed from a traffic junction typically arrives at 25 or 30 frames per second; a detector that cannot process frames at least that fast falls behind and starts dropping frames, silently missing violations. That constraint alone rules out running a heavy two-stage detector on every frame at the edge, which is exactly the gap YOLO-family models were built to fill.

This is also why real deployments rarely rely on a single model end to end. A common pattern is a two-tier pipeline: a fast, lightweight YOLO-family model runs directly on hardware installed near the camera — compact GPU-equipped boards such as NVIDIA's Jetson line are a common choice for exactly this kind of on-site video analytics — scanning every frame in real time and flagging candidate violations. Only those flagged frames, not the full video stream, are sent onward, which matters enormously at city scale, since continuously streaming raw HD video from every junction camera to a central server would demand far more bandwidth than sending an occasional cropped image and a text label. A slower, more accurate model, potentially one built on Faster R-CNN's two-stage design, does not need to run in real time because it only has to process the much smaller stream of already-flagged frames; it can verify the violation and read the number plate carefully before a legally binding challan is issued. The edge model optimizes for speed and for catching everything it can; the verification model optimizes for precision, because issuing a wrong challan carries real consequences for a real person.

That two-tier pattern is the practical answer to the "which one should I use" question that opens every comparison of these architectures: not "YOLO or Faster R-CNN" as a permanent choice, but a question of which stage of the pipeline is being built. Anything that has to run continuously, in real time, on modest hardware — traffic monitoring, live sports tracking, a robot's obstacle avoidance — reaches for a one-stage, YOLO-family detector. Anything that runs once on a smaller, already-narrowed set of images, where a wrong answer is costly, can afford the two-stage design's extra careful look: the verification step before a legal challan, a radiologist-facing second opinion on a scan, or cataloguing a fixed archive of satellite images.

Back at the junction, then, the camera pole is very likely running exactly this kind of split system: a compact, YOLO-family model watching every frame of live traffic and pausing only on the handful that look like violations, and a slower, more careful model, quite possibly built on the two-stage principles Faster R-CNN established, double-checking the number plate before the challan is finalized. Neither architecture alone would make a good production traffic system; one is too slow to watch every frame, and the other alone would issue challans on nothing more than a fast first guess. The two together, connected by the same IoU and Non-Maximum Suppression machinery worked through by hand in this chapter, are what actually turn a security camera into a functioning enforcement system. That is the real lesson underneath both architectures: production computer vision is rarely one clever model, but a pipeline of models, each doing the part of the job it is fastest or most accurate at.

Think About It

Think about this: How would you explain computer vision in production: yolo and faster r-cnn to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind computer vision in production: yolo and faster r-cnn, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.