Advanced Computer Vision

Advanced Computer Vision

From medical imaging to autonomous vehicles, computer vision powers critical AI applications. While basic CNNs excel at image classification ("is this a cat?"), advanced vision tackles harder problems: "where is every object in the image?" (detection), "what is every pixel?" (segmentation), and "what 3D scene created this 2D image?" (3D understanding). These are the frontiers of modern computer vision.

From Classification to Detection: YOLO and R-CNN

Image classification outputs one label per image. Object detection outputs bounding boxes with labels for multiple objects. This is vastly harder: CNN must simultaneously localize and classify multiple objects in variable locations.

Two Approaches: 1. Region-based (R-CNN): - Generate region proposals (potential object locations) - Classify each region - Refine bounding boxes - Slow but accurate

2. Single-stage detectors (YOLO): - Divide image into grid - Each cell predicts objects at that location - One forward pass → all detections - Fast and suitable for real-time applications

YOLO (You Only Look Once): Divide 416×416 image into 13×13 grid. Each cell predicts: - Bounding box coordinates (x, y, w, h) - Confidence (how likely object is here) - Class probabilities (person, car, dog, etc.) Total output: 13 × 13 × (5 + num_classes)

Loss combines three components: L = L_coord (bounding box accuracy) + L_confidence (whether object present) + L_class (object class) L = λ_coord Σ 1_ij^obj [(x_i - x̂_i)² + (y_i - ŷ_i)²] + λ_coord Σ 1_ij^obj [(√w_i - √ŵ_i)² + (√h_i - √ĥ_i)²] + Σ 1_ij^obj (C_i - Ĉ_i)² + λ_noobj Σ 1_ij^noobj (C_i - Ĉ_i)² + Σ 1_i^obj Σ_c (p_i(c) - p̂_i(c))²

import torch
import torch.nn as nn

class YOLOv3Detection(nn.Module):
    def __init__(self, num_classes=80):
        super().__init__()
        # Backbone: pretrained ResNet or Darknet
        self.backbone = darknet53_backbone()

        # Neck: FPN (Feature Pyramid Network)
        self.fpn = FeaturePyramidNetwork()

        # Head: detection at 3 scales
        self.scale1_detection = DetectionHead(num_classes)
        self.scale2_detection = DetectionHead(num_classes)
        self.scale3_detection = DetectionHead(num_classes)

    def forward(self, x):
        features = self.backbone(x)
        fpn_features = self.fpn(features)

        # Predictions at 3 scales
        out1 = self.scale1_detection(fpn_features[0])  # 52×52
        out2 = self.scale2_detection(fpn_features[1])  # 26×26
        out3 = self.scale3_detection(fpn_features[2])  # 13×13

        return out1, out2, out3

# Non-maximum suppression to remove duplicate detections
def nms(detections, threshold=0.5):
    """Remove overlapping detections"""
    if len(detections) == 0:
        return []

    # Sort by confidence
    detections = sorted(detections, key=lambda x: x['conf'], reverse=True)
    keep = [detections[0]]

    for det in detections[1:]:
        remove = False
        for kept in keep:
            iou = compute_iou(det['box'], kept['box'])
            if iou > threshold:
                remove = True
                break

        if not remove:
            keep.append(det)

    return keep

YOLO Evolution: - YOLOv1 (2016): First real-time detector - YOLOv2: Better accuracy, multi-scale training - YOLOv3: Predictions at multiple scales, better for small objects - YOLOv4/v5: Incremental improvements, widely used in industry - YOLOv8: Latest (2023), Ultralytics version

R-CNN Family: Accuracy Over Speed

R-CNN (Girshick et al., 2014): 1. Generate 2000 region proposals using selective search 2. Extract CNN features for each region 3. Classify each region with SVM 4. Refine bounding boxes Slow (~50 seconds per image) but groundbreaking accuracy improvement.

Fast R-CNN (2015): - Extract CNN features once on full image - Use Region of Interest (RoI) pooling to extract features for each proposal - Single network for classification and bounding box refinement 10x faster than R-CNN.

Faster R-CNN (2016): - Replace selective search with Region Proposal Network (RPN) - RPN shares features with detection network - End-to-end trainable Near real-time speed with high accuracy.

Mask R-CNN (2017): - Extend Faster R-CNN to segment object boundaries - Adds mask prediction branch parallel to classification - Enables instance segmentation: different pixel label for each object instance

Semantic Segmentation: Per-Pixel Classification

Every pixel gets a class label. Unlike object detection (bounding boxes), segmentation provides precise boundaries.

FCN (Fully Convolutional Networks): Replace fully connected layers with convolutional layers. Network outputs same resolution as input (after upsampling). Trained end-to-end with per-pixel loss.

U-Net (Ronneberger et al., 2015): Encoder-decoder architecture with skip connections: - Encoder: downsamples to compress features - Decoder: upsamples back to original resolution - Skip connections: concatenate encoder features to decoder Excellent for medical image segmentation. Very efficient, works well with limited training data.

DeepLab Series: - Atrous (dilated) convolutions: increase receptive field without downsampling - Atrous Spatial Pyramid Pooling (ASPP): multiple parallel dilated convolutions - CRF post-processing: enforce spatial consistency DeepLabv3 is SOTA for semantic segmentation.

import torch
import torch.nn as nn
import torch.nn.functional as F

class UNet(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        # Encoder
        self.enc1 = double_conv(in_channels, 64)
        self.pool1 = nn.MaxPool2d(2, 2)
        self.enc2 = double_conv(64, 128)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.enc3 = double_conv(128, 256)

        # Bottleneck
        self.pool3 = nn.MaxPool2d(2, 2)
        self.bottleneck = double_conv(256, 512)

        # Decoder
        self.up3 = nn.ConvTranspose2d(512, 256, 2, stride=2)
        self.dec3 = double_conv(512, 256)

        self.up2 = nn.ConvTranspose2d(256, 128, 2, stride=2)
        self.dec2 = double_conv(256, 128)

        self.up1 = nn.ConvTranspose2d(128, 64, 2, stride=2)
        self.dec1 = double_conv(128, 64)

        self.final = nn.Conv2d(64, out_channels, 1)

    def forward(self, x):
        # Encoder with skip connections
        e1 = self.enc1(x)
        x = self.pool1(e1)

        e2 = self.enc2(x)
        x = self.pool2(e2)

        e3 = self.enc3(x)
        x = self.pool3(e3)

        # Bottleneck
        x = self.bottleneck(x)

        # Decoder with skip connections
        x = self.up3(x)
        x = torch.cat([x, e3], dim=1)
        x = self.dec3(x)

        x = self.up2(x)
        x = torch.cat([x, e2], dim=1)
        x = self.dec2(x)

        x = self.up1(x)
        x = torch.cat([x, e1], dim=1)
        x = self.dec1(x)

        x = self.final(x)
        return x

def double_conv(in_ch, out_ch):
    return nn.Sequential(
        nn.Conv2d(in_ch, out_ch, 3, padding=1),
        nn.BatchNorm2d(out_ch),
        nn.ReLU(inplace=True),
        nn.Conv2d(out_ch, out_ch, 3, padding=1),
        nn.BatchNorm2d(out_ch),
        nn.ReLU(inplace=True)
    )

Instance Segmentation: Mask R-CNN Deep Dive

Instance segmentation combines detection (where) and segmentation (what). Mask R-CNN extends Faster R-CNN: 1. Generate region proposals (RPN) 2. Classify each region 3. Refine bounding boxes 4. Predict pixel-level mask for each region The mask branch is a small FCN that predicts binary mask (object / background) for each proposed region. This enables distinguishing overlapping objects.

3D Vision: From 2D Images to 3D Understanding

Humans see 3D. Cameras capture 2D projections. Recovering 3D from 2D is fundamentally ambiguous—many 3D scenes could produce same 2D image. Yet with learning, networks can estimate 3D structure.

Depth Estimation: Predict per-pixel depth (distance from camera). Monocular depth estimation uses single image; stereo uses two images. 3D Object Detection: Predict 3D bounding box (8 corners in 3D space). Used in autonomous driving to understand 3D scene. Neural Radiance Fields (NeRF): Represent scene as continuous function. Given position and viewing direction, output color. Enables novel view synthesis—render scene from any viewpoint.

Real-World Applications in India

Autonomous Vehicles: Companies like Yulu use YOLO variants for obstacle detection. Medical Imaging: Diagnostic companies use Mask R-CNN for tumor segmentation, pathology analysis. Agriculture: Startups use crop disease detection via semantic segmentation. Retail: Amazon Go-style systems use instance segmentation for counting and identification. Manufacturing: Quality control: detect defects via object detection.

Key Takeaways

• Object detection: localize multiple objects with bounding boxes
• YOLO is single-stage detector: fast, real-time suitable
• R-CNN family is two-stage: slower but more accurate
• Mask R-CNN adds instance segmentation to Faster R-CNN
• Semantic segmentation: per-pixel classification
• U-Net: efficient encoder-decoder for medical imaging
• DeepLab: SOTA semantic segmentation with atrous convolutions
• 3D vision: depth estimation, 3D detection, NeRF for novel view synthesis
• Real applications: autonomous driving, medical imaging, agriculture, retail
• Key metrics: mAP, IoU (Intersection over Union), per-pixel accuracy

Engineering Perspective: Advanced Computer Vision

When you sit for a technical interview at any top company — whether it is Google, Microsoft, Amazon, or an Indian unicorn like Zerodha, Razorpay, or Meesho — they are not just testing whether you know the definition of advanced computer vision. They are testing whether you can APPLY these concepts to solve novel problems, whether you understand the TRADEOFFS involved, and whether you can reason about system behaviour at scale.

This chapter approaches advanced computer vision with that depth. We will examine not just what it is, but why it works the way it does, what alternatives exist and when to choose each one, and how real systems use these ideas in production. ISRO's mission control systems, India's UPI payment network handling 10 billion transactions per month, Aadhaar's biometric authentication serving 1.4 billion identities — all rely on the principles we discuss here.

Transformer Architecture: The Engine Behind GPT and Modern AI

The Transformer architecture, introduced in the landmark 2017 paper "Attention Is All You Need," revolutionised NLP and eventually all of deep learning. Here is the core mechanism:

# Self-Attention Mechanism (simplified)
import numpy as np

def self_attention(Q, K, V, d_k):
    """
    Q (Query): What am I looking for?
    K (Key):   What do I contain?
    V (Value): What do I actually provide?
    d_k:       Dimension of keys (for scaling)
    """
    # Step 1: Compute attention scores
    scores = np.matmul(Q, K.T) / np.sqrt(d_k)

    # Step 2: Softmax to get probabilities
    attention_weights = softmax(scores)

    # Step 3: Weighted sum of values
    output = np.matmul(attention_weights, V)
    return output

# Multi-Head Attention: Run multiple attention heads in parallel
# Each head learns different relationships:
# Head 1: syntactic relationships (subject-verb agreement)
# Head 2: semantic relationships (word meanings)
# Head 3: positional relationships (word order)
# Head 4: coreference (pronoun → noun it refers to)

The key insight of self-attention is that every token can attend to every other token simultaneously (unlike RNNs which process sequentially). This parallelism enables efficient GPU training. The computational complexity is O(n²·d) where n is sequence length and d is dimension, which is why context windows are a major engineering challenge.

State-of-the-art developments include: sparse attention (reducing O(n²) to O(n·√n)), mixture of experts (MoE — activating only a subset of parameters per input), retrieval-augmented generation (RAG — grounding responses in external documents), and constitutional AI (alignment through principles rather than RLHF alone). Indian researchers at institutions like IIT Bombay, IISc Bangalore, and Microsoft Research India are actively contributing to these frontiers.

India's Scale Challenges: Engineering for 1.4 Billion

Building technology for India presents unique engineering challenges that make it one of the most interesting markets in the world. UPI handles 10 billion transactions per month — more than all credit card transactions in the US combined. Aadhaar authenticates 100 million identities daily. Jio's network serves 400 million subscribers across 22 telecom circles. Hotstar streamed IPL to 50 million concurrent viewers — a world record. Each of these systems must handle India's diversity: 22 official languages, 28 states with different regulations, massive urban-rural connectivity gaps, and price-sensitive users expecting everything to work on ₹7,000 smartphones over patchy 4G connections. This is why Indian engineers are globally respected — if you can build systems that work in India, they will work anywhere.

Advanced Algorithms: Dynamic Programming and Graph Theory

Dynamic Programming (DP) solves complex problems by breaking them into overlapping subproblems. This is a favourite in competitive programming and interviews:

# Longest Common Subsequence — classic DP problem
# Used in: diff tools, DNA sequence alignment, version control

def lcs(s1, s2):
    m, n = len(s1), len(s2)
    dp = [[0] * (n + 1) for _ in range(m + 1)]

    for i in range(1, m + 1):
        for j in range(1, n + 1):
            if s1[i-1] == s2[j-1]:
                dp[i][j] = dp[i-1][j-1] + 1
            else:
                dp[i][j] = max(dp[i-1][j], dp[i][j-1])

    return dp[m][n]

# Dijkstra's Shortest Path — used by Google Maps!
import heapq

def dijkstra(graph, start):
    dist = {node: float('inf') for node in graph}
    dist[start] = 0
    pq = [(0, start)]  # (distance, node)

    while pq:
        d, u = heapq.heappop(pq)
        if d > dist[u]:
            continue
        for v, weight in graph[u]:
            if dist[u] + weight < dist[v]:
                dist[v] = dist[u] + weight
                heapq.heappush(pq, (dist[v], v))

    return dist

# Real use: Google Maps finding shortest route from
# Connaught Place to India Gate, considering traffic weights

Dijkstra's algorithm is how mapping applications find optimal routes. When you ask Google Maps to navigate from Mumbai to Pune, it models the road network as a weighted graph (intersections are nodes, roads are edges, travel time is weight) and runs a variant of Dijkstra's algorithm. Indian highways, city roads, and even railway networks can all be modelled this way. IRCTC's route optimisation for trains across 13,000+ stations uses graph algorithms at its core.

Research Frontiers and Open Problems in Advanced Computer Vision

Beyond production engineering, advanced computer vision connects to active research frontiers where fundamental questions remain open. These are problems where your generation of computer scientists will make breakthroughs.

Quantum computing threatens to upend many of our assumptions. Shor's algorithm can factor large numbers efficiently on a quantum computer, which would break RSA encryption — the foundation of internet security. Post-quantum cryptography is an active research area, with NIST standardising new algorithms (CRYSTALS-Kyber, CRYSTALS-Dilithium) that resist quantum attacks. Indian researchers at IISER, IISc, and TIFR are contributing to both quantum computing hardware and post-quantum cryptographic algorithms.

AI safety and alignment is another frontier with direct connections to advanced computer vision. As AI systems become more capable, ensuring they behave as intended becomes critical. This involves formal verification (mathematically proving system properties), interpretability (understanding WHY a model makes certain decisions), and robustness (ensuring models do not fail catastrophically on edge cases). The Alignment Research Center and organisations like Anthropic are working on these problems, and Indian researchers are increasingly contributing.

Edge computing and the Internet of Things present new challenges: billions of devices with limited compute and connectivity. India's smart city initiatives and agricultural IoT deployments (soil sensors, weather stations, drone imaging) require algorithms that work with intermittent connectivity, limited battery, and constrained memory. This is fundamentally different from cloud computing and requires rethinking many assumptions.

Finally, the ethical dimensions: facial recognition in public spaces (deployed in several Indian cities), algorithmic bias in loan approvals and hiring, deepfakes in political campaigns, and data sovereignty questions about where Indian citizens' data should be stored. These are not just technical problems — they require CS expertise combined with ethics, law, and social science. The best engineers of the future will be those who understand both the technical implementation AND the societal implications. Your study of advanced computer vision is one step on that path.

Key Vocabulary

Here are important terms from this chapter that you should know:

🏗️ Architecture Challenge

Design the backend for India's election results system. Requirements: 10 lakh (1 million) polling booths reporting simultaneously, results must be accurate (no double-counting), real-time aggregation at constituency and state levels, public dashboard handling 100 million concurrent users, and complete audit trail. Consider: How do you ensure exactly-once delivery of results? (idempotency keys) How do you aggregate in real-time? (stream processing with Apache Flink) How do you serve 100M users? (CDN + read replicas + edge computing) How do you prevent tampering? (digital signatures + blockchain audit log) This is the kind of system design problem that separates senior engineers from staff engineers.

The Frontier

You now have a deep understanding of advanced computer vision — deep enough to apply it in production systems, discuss tradeoffs in system design interviews, and build upon it for research or entrepreneurship. But technology never stands still. The concepts in this chapter will evolve: quantum computing may change our assumptions about complexity, new architectures may replace current paradigms, and AI may automate parts of what engineers do today.

What will NOT change is the ability to think clearly about complex systems, to reason about tradeoffs, to learn quickly and adapt. These meta-skills are what truly matter. India's position in global technology is only growing stronger — from the India Stack to ISRO to the startup ecosystem to open-source contributions. You are part of this story. What you build next is up to you.

Crafted for Class 10–12 • Programming & Coding • Aligned with NEP 2020 & CBSE Curriculum