Image Segmentation: Pixel-Level Classification

The Blur Behind You

Open the camera app on almost any smartphone sold in India today, switch to "Portrait" mode, and photograph a friend standing in front of a busy background — the pillars outside a metro station, the crowd at a college fest. The photo that comes out has your friend in sharp focus while everything behind them dissolves into a soft, professional-looking blur, as if shot on an expensive DSLR lens.

Look closely at the edge where your friend's hair meets that blurred background. The blur doesn't stop at some neat rectangle — it hugs the exact, irregular outline of their head, shoulders, collar, even loose strands of hair. For the app to pull this off, it has to decide, for every one of the millions of pixels in the photo, whether that pixel belongs to "person" or to "everything else." Not a rough box around the person — the precise boundary, pixel by pixel.

That pixel-by-pixel decision has a name: image segmentation, the task of assigning a class label to every individual pixel in an image, rather than to the image as a whole or to a rough rectangle drawn around an object.

From a Label, to a Box, to a Boundary

Image classification answers one question about an entire picture: "What is the main thing in this image?" Feed it a photo and it returns a single label — "dog," "cricket bat," "auto-rickshaw" — with no information about where in the image that object actually sits.

Object detection goes further: it finds every object of interest and draws a rectangular bounding box around each one, with a label attached, something like "person: x=120, y=40, width=180, height=310." This tells you roughly where each object is, but a rectangle is a crude fit. A box drawn around a person standing with their arms at their sides wastes a lot of its area on background — the gaps beside the waist, the space above the head, the corners.

Image segmentation throws away the rectangle entirely and paints in the true shape instead. Every pixel gets its own label. Three ways of describing where a person stands in a photo make the difference clear:

• Classification: "There is a person somewhere in this photo."
• Detection: "There is a person inside this rectangle."
• Segmentation: "Here is the exact set of pixels that make up the person — and no others."

That precision is exactly what a blur effect needs. You cannot convincingly blur "everything outside a rectangle" without slicing off part of a shoulder or leaving a ring of blur wrapped around a sharp head. You need the true outline.

Semantic Segmentation vs. Instance Segmentation

Segmentation itself comes in two common flavours, and the difference matters a great deal depending on what you are building.

Semantic segmentation assigns every pixel a class label but does not distinguish between separate objects of the same class. Imagine a drone photograph of a busy stretch of MG Road with six cars in it. A semantic segmentation model paints every pixel belonging to any car the same colour, say blue, every pixel of road surface a different colour, and every pixel of sky a third. If two cars happen to be parked bumper to bumper, their blue regions simply merge into one connected blob. The model knows "these pixels are car" — it has no idea whether it's looking at six separate vehicles or one oddly long one.

Instance segmentation goes one step further and separates individual objects even within the same class. In that same street photo, an instance segmentation model would return six distinct masks, one per car — car #1, car #2, and so on — each with its own outline, even where two cars visually touch.

Which one you need depends on the job. A satellite system mapping how much of a district is forest versus farmland versus built-up area only needs semantic segmentation; it does not care how many individual trees exist, only which pixels are "forest." Now picture a system built to count how many vehicles are queued at a toll plaza — that needs instance segmentation, because "vehicle pixels" alone cannot tell you if a blob is three motorcycles packed close together or one truck.

One landmark instance-segmentation model is Mask R-CNN, introduced by researchers at Facebook AI Research (FAIR) in 2017. It extended an existing object-detection network so that, alongside every bounding box and class label, it also predicts a precise pixel mask for that specific object — bolting segmentation directly onto detection.

How a Computer Actually Stores a Segmentation

Recall that a digital image is stored as a grid of pixels, and every pixel holds one or more numbers — for a grayscale image, a single brightness value from 0 (pure black) to 255 (pure white); for a colour image, three such values, one each for red, green, and blue.

A segmentation output is stored the same way: as a grid of the exact same height and width as the input image, except that every cell holds a class index instead of a colour — a small integer standing for "background," "person," "road," "sky," and so on. This grid is called a segmentation mask, or sometimes a label map. If class 0 means background and class 1 means person, a mask for a tiny image with a person in the middle might look like this:

0 0 0 0 0
0 1 1 1 0
0 1 1 1 0
0 1 1 1 0
0 0 0 0 0

Every 0 says "this pixel is background"; every 1 says "this pixel is person." A real photo's mask works identically, just at a much larger scale, with more class numbers if there are more categories to label (0 = background, 1 = person, 2 = road, 3 = vehicle, and so on).

Producing this grid — deciding the correct number for every single cell — is the actual work of a segmentation system. Let's build one by hand.

Worked Example: Segmenting a Tiny Photo by Hand

Suppose you have a heavily simplified 5×5 grayscale photo: a bright person standing against a dark wall. Each number below is one pixel's brightness, from 0 (black) to 255 (white):

  60  65  60  55 190
  58 200 210 195  62
  65 205  90 200  68
  60 195 205 190  58
  55  60  65  70  60

You, the human labeller, know the true shape of the person in this photo — perhaps you were standing there when it was taken. The true 3×3 block of person-pixels sits in the middle (rows 2–4, columns 2–4), even though one of those pixels (row 3, column 3, value 90) is dark because of a shadow crossing the subject's face, and one background pixel (row 1, column 5, value 190) is bright because of sunlight glaring off the wall. This ground truth mask looks like this:

0 0 0 0 0
0 1 1 1 0
0 1 1 1 0
0 1 1 1 0
0 0 0 0 0

Now suppose your segmentation "model" is the simplest one imaginable: a brightness threshold. Classify any pixel brighter than 150 as person (1); anything else as background (0). Apply that single rule to all 25 pixels above, and you get this predicted mask:

0 0 0 0 1
0 1 1 1 0
0 1 0 1 0
0 1 1 1 0
0 0 0 0 0

Compare the two masks cell by cell. Two pixels disagree:

• Row 1, column 5: predicted person (190 > 150), but truly background — that glare on the wall. This is a false positive.
• Row 3, column 3: predicted background (90 is below the threshold), but truly person — that shadow on the face. This is a false negative.

Every other pixel — 23 of the 25 — matches: 8 pixels are correctly predicted as person (true positives), and 15 are correctly predicted as background (true negatives).

With those four counts — TP = 8, FP = 1, FN = 1, TN = 15 — three standard segmentation metrics can now be computed.

Pixel accuracy, the simplest of the three, is just the fraction of all pixels labelled correctly:

accuracy = (TP + TN) / total pixels = (8 + 15) / 25 = 92%

Intersection over Union (IoU) compares only the "person" region: the overlap between the predicted person-area and the true person-area, divided by the combined area covered by either one:

IoU = TP / (TP + FP + FN) = 8 / (8 + 1 + 1) = 80%

Dice coefficient, closely related to IoU and especially popular in medical-imaging research, doubles the overlap before dividing:

Dice = (2 × TP) / (2 × TP + FP + FN) = 16 / 18 ≈ 88.9%

Notice that accuracy, at 92%, looks the most impressive of the three — and is also the most misleading. Sixteen of this image's 25 pixels are background, so a lazy model that predicted "background" for every single pixel, without ever looking at the image, would still score (0 + 16) / 25 = 64% accuracy. On a more realistic photo, where a person might occupy only 5–10% of the frame, that same do-nothing model could still score around 90% accuracy or higher while being completely useless — it never identifies a single person pixel correctly. IoU does not have this blind spot, because true negatives never appear in its formula: a model gets no credit at all for correctly ignoring background it was never in danger of misreading. This is why IoU, not raw pixel accuracy, is the standard way segmentation systems are actually scored.

The Same Calculation in Code

Everything just computed by hand can be written as a short program, using the same numpy array conventions from earlier chapters:

import numpy as np

# The 5x5 grayscale image (pixel brightness, 0-255)
image = np.array([
    [60,  65,  60,  55, 190],
    [58, 200, 210, 195,  62],
    [65, 205,  90, 200,  68],
    [60, 195, 205, 190,  58],
    [55,  60,  65,  70,  60]
])

# Ground truth mask: 1 = person, 0 = background
ground_truth = np.array([
    [0, 0, 0, 0, 0],
    [0, 1, 1, 1, 0],
    [0, 1, 1, 1, 0],
    [0, 1, 1, 1, 0],
    [0, 0, 0, 0, 0]
])

# Predicted mask: threshold every pixel at brightness 150
predicted = (image > 150).astype(int)

# Compare predicted vs. ground truth, pixel by pixel
TP = np.sum((predicted == 1) & (ground_truth == 1))
FP = np.sum((predicted == 1) & (ground_truth == 0))
FN = np.sum((predicted == 0) & (ground_truth == 1))
TN = np.sum((predicted == 0) & (ground_truth == 0))

accuracy = (TP + TN) / predicted.size
iou = TP / (TP + FP + FN)
dice = (2 * TP) / (2 * TP + FP + FN)

print(f"TP={TP}  FP={FP}  FN={FN}  TN={TN}")
print(f"Pixel accuracy: {accuracy:.1%}")
print(f"IoU:            {iou:.1%}")
print(f"Dice:           {dice:.1%}")

Running this prints:

TP=8  FP=1  FN=1  TN=15
Pixel accuracy: 92.0%
IoU:            80.0%
Dice:           88.9%

The line predicted = (image > 150).astype(int) does the entire prediction step in one shot: numpy compares every element of the array against 150 simultaneously, producing a grid of True/False values, which .astype(int) converts to 1s and 0s — exactly the threshold rule applied by hand above, just carried out across all 25 pixels at once instead of one at a time.

Why a Fixed Threshold Isn't Enough

The two errors in the worked example — the glare misread as a person, the shadow misread as background — are not bad luck. They expose the fundamental weakness of any rule built around a single fixed number: real photographs have shadows, reflections, a wide range of skin tones, and backgrounds that are sometimes brighter than the foreground and sometimes darker. A threshold tuned to work well on one photo will misfire on the next one taken in different lighting.

This is why production segmentation systems do not use a fixed threshold at all. Instead, they use a Convolutional Neural Network (CNN) trained on thousands to millions of photographs that humans have already labelled, pixel by pixel, by hand. Rather than checking whether one pixel is bright enough, the network learns to recognise textures, edges, and shapes across small neighbourhoods of pixels — the soft edge where hair meets skin, the smooth gradient of a shadow, the sharp line where a road meets a footpath — and combines those local patterns with the broader shape of the object to decide every pixel's class.

One influential architecture built specifically for this task is U-Net, introduced in 2015 by Olaf Ronneberger, Philipp Fischer, and Thomas Brox at the University of Freiburg, originally to segment individual cells in microscope images. A U-Net has two halves, and its shape, drawn as a diagram, genuinely resembles the letter U. The first half, the encoder, repeatedly shrinks the image down while learning increasingly abstract features — building an understanding of what is in the image. The second half, the decoder, gradually expands that compressed understanding back out to the original resolution, deciding where each thing is, pixel by pixel. Special skip connections carry fine detail directly from early encoder layers across to their matching decoder layers, so that sharp boundaries — a strand of hair, the edge of a single cell — are not lost while the image is shrunk down and then rebuilt. U-Net and its many descendants remain a standard building block in medical-imaging and satellite-imagery segmentation systems today.

Where This Shows Up Around You

Once you know to look for it, pixel-level segmentation is doing quiet work across a wide range of applications in India:

• Healthcare: radiologists reviewing an MRI or CT scan use segmentation-assisted software to trace the boundary of a tumour or organ, turning a task that once took many manual minutes with a mouse into a starting outline that only needs to be checked and corrected.
• Agriculture: satellite platforms such as ISRO's Bhuvan classify land, pixel by pixel, into categories like crop cover, bare soil, water bodies, and built-up area — letting agricultural departments track land and crop patterns across areas far too large to survey on foot.
• Driver assistance systems: a self-driving or advanced driver-assistance system needs to know exactly which pixels are drivable road, which are a painted lane line, and which are a pedestrian standing at the edge of that lane — a bounding box around "pedestrian" is not precise enough to tell if a foot has stepped past the kerb.
• Your camera app: and, of course, the portrait-mode blur this chapter opened with.

Back to the Blur

Return to that photo of your friend against the crowd at the metro station. A modern phone builds the person-mask behind that blur using exactly the ideas from this chapter, just at a much larger scale: a neural network related in spirit to U-Net — sometimes helped by a second camera or a dedicated depth sensor — assigns every pixel a "subject" or "background" score in a fraction of a second. Every pixel scored as background gets blurred; every pixel scored as subject stays sharp. Apple's iPhone 7 Plus, launched in 2016, was among the first mass-market phones to popularise this effect, using its dual rear cameras to estimate depth; within about a year, Google's Pixel phones showed that a well-trained network could estimate the same subject-versus-background split from a single camera, with no depth sensor at all.

The quality of that blur — whether individual hair strands survive cleanly or get smeared away, whether a hand held up near the chest stays sharp or vanishes into the background — comes down to exactly the number this chapter taught you to compute: how high the IoU is between the phone's predicted mask and the true outline of your friend. A phone with a better segmentation model produces a mask closer to 100% IoU against reality, and a cleaner, more convincing photograph. The next time a blurred background looks a little too perfect, you will know there is a pixel-by-pixel classification decision — millions of them, made in a fraction of a second — sitting behind that one photograph.

Key Takeaways

• Image segmentation assigns a class label to every pixel, producing an exact object outline — sharper than a whole-image label (classification) and sharper than a rough rectangle (detection).
• Semantic segmentation labels pixels by class only; instance segmentation also separates individual objects that share the same class.
• A segmentation output is stored as a mask: a grid the same height and width as the image, where every cell holds a class index instead of a colour value.
• IoU and Dice are stricter, more honest evaluation metrics than raw pixel accuracy, because accuracy alone is inflated by large, easy-to-guess background regions.
• Real segmentation systems replace fixed rules like brightness thresholds with trained CNNs, such as U-Net, that learn to recognise object shapes under changing lighting and backgrounds — the same idea, running at a much larger scale, that blurs the background behind you in portrait mode.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where image segmentation: pixel-level classification is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting image segmentation: pixel-level classification to at least 3 other topics you have studied.