Capsule Networks: Beyond Convolutions

Every year CBSE and NTA digitize tens of millions of handwritten roll numbers, OMR bubbles, and answer-booklet digits. A convolutional network trained to read these digits has one job that sounds trivial and is not: tell a 6 from a 9. The two digits are built from identical parts — a closed loop and a curved tail — arranged in exactly the same way, differing only by a 180° rotation. A human reads the rotation instantly. A CNN, by the very design choice that makes it work at all, is partially blind to it. That is not a training-data problem you can fix by adding more rotated examples. It is a structural property of how convolution and pooling represent an image, and it is the exact gap capsule networks were built to close.

Why max pooling throws away the one thing you need

A convolutional layer slides a learned filter over the input and produces a feature map: high activation wherever the filter's pattern is present, regardless of where. Max pooling then takes, say, every 2×2 block of that feature map and keeps only the largest value, discarding the rest. This is deliberate: it gives the network translation invariance — shift the loop of a "6" two pixels to the left, and the pooled output barely changes, because the max is still coming from roughly the same neighborhood. That invariance is why CNNs generalize from a training set that never covers every possible pixel position.

But invariance is bought with a specific loss: once you keep only the maximum activation in a pooling window, you throw away exactly where within that window it fired, and in earlier layers, at what orientation and scale it fired. A capsule network's inventors describe this precisely with two different words. A function is invariant to a transformation T if f(T(x)) = f(x) — the output does not change when the input is transformed. A function is equivariant if f(T(x)) = T′(f(x)) — the output changes in a corresponding, predictable way. Max pooling gives you invariance for detecting that a loop and a tail are present somewhere in the digit. It gives you almost nothing about their relative pose — whether the tail hangs below-right of the loop (a 6) or above-left of it (a 9) — because that relational information is precisely what got maxed away. A deep enough stack of conv+pool layers can eventually relearn some of this by brute-force pattern memorization across huge datasets, which is why CNNs work as well as they do in practice. But the representation itself does not carry pose as a first-class quantity. Capsule networks, introduced by Sabour, Frosst, and Hinton in "Dynamic Routing Between Capsules" (2017), redesign the layer so that pose is carried explicitly, and equivariance — not invariance — is the design target for the internal representation.

A capsule: a vector of neurons, not a scalar

An ordinary neuron outputs one number: how strongly a feature is present. A capsule is a group of neurons whose combined output is a vector. Two properties are packed into that vector by construction:

• Its length (bounded into [0, 1) by a squashing function you will derive below) represents the probability that the entity the capsule detects is present in the input.
• Its orientation represents the entity's instantiation parameters — pose, in the broad sense: position, orientation, scale, deformation, even stroke thickness, depending on what the training data forces the capsule to encode.

Critically, the goal is for a capsule's vector to be equivariant to changes in the entity's pose: rotate the digit's loop-and-tail arrangement, and the capsule's output vector should rotate correspondingly, while its length — the "is a digit-loop present" confidence — stays roughly constant. A "6" capsule and a "9" capsule, seeing the same two parts (loop, tail) in opposite relative arrangement, are pushed toward disagreeing about which digit is present precisely because the pose relationship between the parts is preserved and checked, not discarded.

The squashing nonlinearity

A capsule's raw output before nonlinearity is a vector s_j that can have any length. Squashing needs to compress long vectors toward length 1 (near-certain presence) and short vectors toward length 0 (near-certain absence), while preserving direction exactly. The function used is:

v_j = ( ||s_j||^2 / (1 + ||s_j||^2) ) * ( s_j / ||s_j|| )

The first factor is a scalar in [0, 1) that grows monotonically with ||s_j||; the second factor is the unit vector in the direction of s_j. Trace it on a concrete vector: let s_j = [3, 4]. Then ||s_j|| = √(3²+4²) = √25 = 5, and ||s_j||² = 25. The scale factor is 25 / (1+25) = 25/26 ≈ 0.9615. The unit vector is [3,4]/5 = [0.6, 0.8]. So v_j = 0.9615 × [0.6, 0.8] = [0.5769, 0.7692], with ||v_j|| = 0.9615 exactly matching the scale factor (as it must, since a scale factor times a unit vector has that scale as its norm). A capsule with input norm 5 reports roughly 96% confidence, still in the exact direction of its raw evidence. A weak capsule with s_j = [0.3, 0.4] (norm 0.5) gets scale 0.25/1.25 = 0.20 — reported confidence drops to 20%, disproportionately punishing weak evidence, which is what you want from a presence detector.

Dynamic routing by agreement

The hard problem capsules solve is: given a layer of lower-level capsules (parts — "loop", "tail"), which higher-level capsule (whole — "6" or "9") should each part's vote go to? A fully-connected layer would blend every part into every whole with fixed learned weights, the same for every input. Routing instead computes, freshly for every single forward pass, how much weight each part sends to each whole — based on whether the parts agree about what whole they're voting for.

Each lower capsule i first produces a prediction vector for each higher capsule j by a learned transformation matrix: û_j|i = W_ij u_i. W_ij is a learned parameter (via ordinary backprop) that encodes the expected part-to-whole viewpoint transform — e.g., "if this is really a loop belonging to digit-6, its pose should map to the 6-capsule's pose like this." The routing algorithm then iterates r times (3 is typical):

initialize b_ij = 0 for every (i, j)
repeat r times:
    for every i:  c_i = softmax_j(b_i)        # coupling coefficients
    for every j:  s_j = sum_i  c_ij * u_hat_ji  # weighted sum of votes
    for every j:  v_j = squash(s_j)
    for every i, j:  b_ij += u_hat_ji . v_j     # agreement = dot product

c_ij is a softmax over j of the routing logits b_i·, so for each lower capsule i, the c_ij across all j sum to 1 — capsule i is distributing 100% of its "vote" across the higher capsules, more to whichever ones currently agree with it. The b_ij update is a dot product: it grows large and positive exactly when the prediction vector û_j|i points in nearly the same direction as the current output v_j, i.e., when the part's vote and the whole's consensus pose agree. Nothing here is a learned weight — b_ij is reset to zero at the start of every forward pass and rebuilt purely from that input's own agreement structure. Only W_ij is a parameter that persists across inputs and gets updated by gradient descent.

Worked example: tracing two routing iterations by hand

Take two primary capsules (i = 1, 2) voting for two candidate higher capsules — call them A ("6") and B ("9") — in a simplified 2-D pose space so every step is checkable by hand. Suppose the transformation matrices have already produced these prediction vectors:

u_hat(A|1) = [ 1.00,  0.00]      u_hat(B|1) = [ 0.00, 1.00]
u_hat(A|2) = [ 0.90,  0.10]      u_hat(B|2) = [-0.80, 0.20]

Capsule 1's and capsule 2's votes for A are nearly identical directions (high agreement); their votes for B point in unrelated directions (low agreement). Routing should discover this without being told.

Iteration 1. All b_ij start at 0, so every softmax is uniform: c_1A=c_1B=c_2A=c_2B=0.5. The weighted sums:

s_A = 0.5*[1.00,0.00] + 0.5*[0.90,0.10] = [0.950, 0.050]
s_B = 0.5*[0.00,1.00] + 0.5*[-0.80,0.20] = [-0.400, 0.600]

Squash each. ||s_A|| = √(0.95²+0.05²) = √0.905 = 0.9513, scale = 0.905/1.905 = 0.4751, giving v_A = 0.4751×[0.9986, 0.0526] = [0.4744, 0.0250]. For B: ||s_B|| = √0.52 = 0.7211, scale = 0.52/1.52 = 0.3421, giving v_B = [-0.1898, 0.2847]. Now update the logits by agreement (dot product of each vote with the output it fed):

b_1A += [1,0]·[0.4744,0.0250]     = 0.4744
b_2A += [0.9,0.1]·[0.4744,0.0250] = 0.4295
b_1B += [0,1]·[-0.1898,0.2847]   = 0.2847
b_2B += [-0.8,0.2]·[-0.1898,0.2847] = 0.2087

Iteration 2. Recompute softmaxes with these logits. For capsule 1: c_1A = e^0.4744/(e^0.4744+e^0.2847) = 1.607/(1.607+1.329) = 0.5473, c_1B = 0.4527. For capsule 2: c_2A = e^0.4295/(e^0.4295+e^0.2087) = 0.5550, c_2B = 0.4450. New sums:

s_A = 0.5473*[1,0] + 0.5550*[0.9,0.1] = [1.0468, 0.0555]
s_B = 0.4527*[0,1] + 0.4450*[-0.8,0.2] = [-0.3560, 0.5417]

Squashing: ||s_A|| = 1.0483, scale = 1.0989/2.0989 = 0.5236 → v_A = [0.5228, 0.0277], ||v_A|| = 0.5236. ||s_B|| = 0.6482, scale = 0.4201/1.4201 = 0.2958 → v_B = [-0.1625, 0.2472], ||v_B|| = 0.2958.

Compare the two iterations: ||v_A|| rose from 0.4751 to 0.5236 while ||v_B|| fell from 0.3421 to 0.2958. Capsule A's confidence is growing because its two incoming votes agree with each other; capsule B's is shrinking because its votes point in conflicting directions. That is dynamic routing doing its job with zero label information at inference time — purely by internal consistency between what the parts predict.

Routing in twenty lines of NumPy

import numpy as np

def squash(s):
    norm_sq = np.sum(s ** 2)
    norm = np.sqrt(norm_sq)
    scale = norm_sq / (1 + norm_sq)
    return scale * (s / norm)

u_hat = {
    ('A', 1): np.array([1.0, 0.0]),
    ('A', 2): np.array([0.9, 0.1]),
    ('B', 1): np.array([0.0, 1.0]),
    ('B', 2): np.array([-0.8, 0.2]),
}
b = {(j, i): 0.0 for j in ['A', 'B'] for i in [1, 2]}

for r in range(2):
    c = {}
    for i in [1, 2]:
        eA, eB = np.exp(b[('A', i)]), np.exp(b[('B', i)])
        c[('A', i)] = eA / (eA + eB)
        c[('B', i)] = eB / (eA + eB)
    s = {j: sum(c[(j, i)] * u_hat[(j, i)] for i in [1, 2]) for j in ['A', 'B']}
    v = {j: squash(s[j]) for j in ['A', 'B']}
    for i in [1, 2]:
        for j in ['A', 'B']:
            b[(j, i)] += float(np.dot(u_hat[(j, i)], v[j]))
    print(r, {j: np.round(v[j], 4) for j in v})

Running this prints, on iteration 0, {'A': [0.4744, 0.025], 'B': [-0.1898, 0.2847]}, and on iteration 1, {'A': [0.5228, 0.0277], 'B': [-0.1625, 0.2472]} — matching the hand trace to four decimal places, because the code is literally the four-line algorithm with nothing hidden.

The full CapsNet architecture, with real numbers

The original CapsNet for 28×28 MNIST-style digits (the direct ancestor of any 6-vs-9 digit-pose reader) chains three stages. Conv1: 256 filters of size 9×9, stride 1, ReLU, turning the 28×28×1 input into a 20×20×256 feature map (28−9+1 = 20) — an ordinary convolution, doing the low-level edge detection convolution is good at. PrimaryCaps: this feature map is reshaped into 32 separate "capsule types," each produced by its own 9×9, stride-2 convolution, giving spatial output size ⌊(20−9)/2⌋+1 = 6, so a 6×6×32 grid where every one of the 6×6×32 = 1152 positions is an 8-dimensional capsule (8 numbers, squashed together as one vector, instead of 8 independent scalar feature maps). DigitCaps: 10 capsules, one per digit class, each 16-dimensional, fully connected to all 1152 PrimaryCaps capsules via routing (3 iterations). Each of the 1152×10 = 11,520 connections has its own 16×8 transformation matrix, for 11,520×128 = 1,474,560 learned parameters in that one routing layer alone — more parameters than the entire Conv1 layer (256×9×9+256 ≈ 21,000), spent entirely on learning how each part's pose predicts each whole's pose.

Training does not use softmax cross-entropy over the ten DigitCaps lengths. It uses a margin loss per class k:

L_k = T_k * max(0, 0.9 - ||v_k||)^2 + 0.5 * (1 - T_k) * max(0, ||v_k|| - 0.1)^2

where T_k = 1 if digit k is the true label, else 0. The first term penalizes the correct capsule if its length falls below 0.9; the second penalizes any incorrect capsule if its length rises above 0.1, down-weighted by 0.5 so early training (when everything is near zero) doesn't shrink all ten capsules toward nothing at once. A small reconstruction sub-network (three fully-connected layers, 512 → 1024 → 784, reading only the winning 16-D capsule) is added as a regularizer, its loss scaled by 0.0005 — the network must be able to redraw the input digit from that one capsule's pose vector, which forces the vector to actually encode pose rather than just a class tag.

The misconception to kill

The mistake nearly every student makes on first exposure: treating a DigitCaps capsule's length as "the probability the network assigns to this digit," in the same sense as a softmax output. It is not. Softmax forces ten numbers to sum to exactly 1, encoding mutual exclusivity by construction. Capsule lengths carry no such constraint — each of the 10 capsules independently squashes its own s_k into [0,1), so in principle several capsules can report high length at once (this is deliberate: it is what lets a later, "MultiMNIST" extension of the same architecture detect two overlapping digits in one image, something a single softmax head structurally cannot do). That is precisely why margin loss, not cross-entropy, is used to train it — margin loss judges each capsule against fixed thresholds (0.9 to reward, 0.1 to punish) independently, with no competition term forcing the ten lengths to trade off against each other.

Diagram: one routing iteration, traced with the worked numbers

Active recall

Work these before reading the answers.

1. A capsule outputs v = [0.3, 0.4] after squashing. What does its length represent, and what does its direction represent?
2. Given raw capsule output s_j = [6, 8], compute v_j = squash(s_j), showing every step.
3. In dynamic routing, why does b_ij increase when û_j|i and v_j point in similar directions, and what effect does a larger b_ij have on c_ij in the next iteration?
4. Why can't CapsNet be trained with ordinary softmax cross-entropy over the ten DigitCaps lengths? Give the loss actually used, with its formula.
5. A PrimaryCaps layer has 32 capsule types over a 6×6 grid, each 8-D, feeding a DigitCaps layer of 10 capsules, each 16-D. How many transformation matrices W_ij does this routing layer need, and how many scalar parameters total?
6. True or false, with justification: "the coupling coefficients c_ij are learned during backpropagation the same way the transformation matrices are."

Answers.

1. Length = √(0.3²+0.4²) = √0.25 = 0.5 — a 50% confidence that whatever entity this capsule represents (e.g., "loop-and-tail digit shape") is present in the input. The direction of [0.3, 0.4] encodes that entity's pose/instantiation parameters in the capsule's learned pose subspace — not a probability, a description of "how" the entity appears (orientation, deformation, etc.), which is what makes the representation equivariant rather than purely invariant.

2. ||s_j|| = √(6²+8²) = √100 = 10. ||s_j||² = 100. Scale = 100/(1+100) = 100/101 ≈ 0.9901. Unit vector = [6,8]/10 = [0.6, 0.8]. v_j = 0.9901 × [0.6, 0.8] = [0.5941, 0.7921], with ||v_j|| ≈ 0.9901.

3. b_ij += û_j|i·v_j is a dot product, which is largest and most positive when the two vectors are closely aligned (high cosine similarity and reasonable magnitude) — that is the mathematical definition of "agreement" the algorithm uses. Since c_ij = softmax over j of b_i·, raising b_ij relative to capsule i's other logits increases c_ij after the softmax in the next iteration, routing more of capsule i's vote to capsule j. This is the entire mechanism of "routing by agreement": the routing weight is computed from agreement, not stored as a separate learned parameter.

4. Cross-entropy assumes the ten output values sum to 1 (mutual exclusivity between classes), but capsule lengths are ten independently squashed values with no such constraint — several could be large at once (the point of the architecture, since it also supports detecting overlapping objects). CapsNet instead uses margin loss per class k: L_k = T_k·max(0, 0.9−||v_k||)² + 0.5(1−T_k)·max(0, ||v_k||−0.1)², where T_k=1 for the true class.

5. Lower capsules = 6×6×32 = 1152. Each connects to all 10 DigitCaps capsules, so there are 1152×10 = 11,520 transformation matrices. Each matrix maps an 8-D vector to a 16-D vector, so it is 16×8 = 128 numbers. Total parameters = 11,520×128 = 1,474,560.

6. False. Only W_ij (plus the ordinary convolution filters and the reconstruction network's weights) are learned by gradient descent across the training set and persist between inputs. The coupling coefficients c_ij are recomputed from scratch for every single forward pass, starting from b_ij=0, purely as a function of that input's own prediction vectors and their mutual agreement — they are activations produced at inference time, not stored parameters.

Think About It

Think about this: How would you explain capsule networks: beyond convolutions to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind capsule networks: beyond convolutions, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.