Self-Supervised Learning: Beyond Contrastive

An agri-tech team building a crop-disease detector for an Indian farmer app has five million leaf photographs uploaded through the app over two years, and about four thousand of them hand-labelled by agronomists into disease categories. The obvious move is to pretrain a visual encoder on the five million unlabelled images with a self-supervised objective, then fine-tune the last layers on the four thousand labels. The team reaches for SimCLR, the contrastive method their deep-learning course covered: augment each image twice, pull the two augmented views together in embedding space, push every other image in the batch away as a negative. It works beautifully in the paper. It works badly on their hardware.

The reason is arithmetic, not bad luck. SimCLR's loss, InfoNCE, estimates a ratio between one positive pair and a sum over every negative pair in the batch. The larger the batch, the sharper and less noisy that ratio, which is why the original paper trains with batches in the thousands, spread across many accelerators. The team has one GPU and a batch size of 256. Two hundred fifty-five negatives per anchor sounds like plenty until you remember that in a random batch of natural leaf images, dozens of those "negatives" are near-duplicates of the anchor (same species, same lighting, same disease stage), so the denominator is full of accidental positives masquerading as negatives, and the gradient signal that is supposed to spread the embedding space out instead gets muddy and weak. Linear-probe accuracy on the pretrained encoder comes in well below what the SimCLR paper reports at large batch sizes. The team is not doing anything wrong; they have simply hit the structural dependency that contrastive learning has on batch size and negative-pair quality.

This chapter is about the two families of self-supervised methods that were built specifically to remove that dependency: non-contrastive Siamese networks (BYOL, SimSiam), which learn from positive pairs only and use no negatives at all, and masked prediction (MAE for images, and the BERT-style masked language modelling you have already seen for text), which sidesteps the negative-pair question entirely by turning self-supervision into a reconstruction problem. Both let the agri-tech team train on a single GPU with a batch size of 256 without their representations degrading. Understanding how they avoid the trap that killed the naive "just drop the negatives" idea is the real content of this chapter.

Why you cannot just delete the negative term

Suppose you take the SimCLR recipe and simplify it: keep the positive pair (two augmentations of the same image passed through the same encoder), but throw away the negative term in the loss. Train the encoder to output identical embeddings for both augmented views by minimising the squared distance between them, nothing else.

This objective has a global minimum you were not looking for. If the encoder ignores its input entirely and outputs the same constant vector for every image, the loss is exactly zero. Two augmented views of a leaf photo map to the same vector, but so do two augmented views of a completely different leaf, a rock, or noise, because the encoder never looked at the input in the first place. This failure mode is called representation collapse: the network satisfies the training objective perfectly while learning nothing useful, and downstream linear probes trained on the collapsed embeddings perform at chance.

It is worth seeing this collapse happen in numbers rather than taking the claim on faith, because the exact mechanism (not just the existence of collapse) is what BYOL and SimSiam are engineered to break.

Worked example, part 1: collapse is a geometric decay to zero

Strip the encoder down to a single scalar weight so every step is checkable by hand. Let the encoder be f(v) = w·v, and let two augmented views of one image be scalars v1 = 1.0 and v2 = 1.2 (an augmentation such as a brightness jitter that nudges the underlying value slightly). Train the same weight w on both branches (a true Siamese network: one set of weights, two forward passes) by minimising the naive loss

L(w) = (f(v1) - f(v2))^2 = w^2 * (v1 - v2)^2

Since (v1 - v2)^2 = (1.0 - 1.2)^2 = 0.04, this is L(w) = 0.04·w², a parabola with its unique minimum at w = 0. Its gradient is dL/dw = 0.08·w. Starting from w₀ = 0.5 and taking gradient-descent steps with learning rate lr = 0.5:

w_{n+1} = w_n - lr * (0.08 * w_n) = w_n * (1 - lr*0.08) = w_n * 0.96

Every step multiplies the weight by exactly 0.96. Three steps by hand:

w0 = 0.5
w1 = 0.5   * 0.96 = 0.48
w2 = 0.48  * 0.96 = 0.4608
w3 = 0.4608* 0.96 = 0.442368

Verifying with code that performs the identical arithmetic:

v1, v2 = 1.0, 1.2
w, lr = 0.5, 0.5
diff2 = (v1 - v2) ** 2          # 0.04
for step in range(3):
    grad = 2 * diff2 * w        # dL/dw
    w = w - lr * grad
    print(step, w)

# output:
# 0 0.48
# 1 0.4608
# 2 0.442368

The weight is on a geometric path w_n = 0.5 × 0.96ⁿ to zero, and it never stops until it gets there, because w = 0 is the only fixed point of a symmetric squared-distance loss between two views of the same shared encoder. Note something important: this had nothing to do with the specific values 1.0 and 1.2. As long as the two augmented views differ at all, the decay factor (1 − 2·lr·(v1−v2)²) is strictly less than 1, so the encoder always collapses toward the trivial constant solution. The naive symmetric objective is unsalvageable by tuning learning rate or augmentation strength; the fixed point is structural.

Common misconception

Students meeting BYOL and SimSiam for the first time usually describe them as "contrastive learning minus the negative term," as if the only change from SimCLR is deleting one piece of the InfoNCE sum. The worked example above shows why that description is wrong: deleting the negative term and training a plain Siamese network on positive pairs alone is exactly the recipe that collapses to zero, as just proven. BYOL and SimSiam are not "contrastive learning with negatives removed." They are architecturally different: an intentionally asymmetric pair of networks connected by a stop-gradient, which is a different mechanism from anything in contrastive learning, not a subtraction from it. If you remove the negatives from SimCLR without also introducing this asymmetry, you get collapse, not BYOL.

What actually prevents collapse: asymmetry and stop-gradient

BYOL (Bootstrap Your Own Latent, Grill et al., 2020) splits the single shared encoder into two networks that are no longer trained the same way:

• An online network: encoder f_θ → projector g_θ → an extra predictor head q_θ. This is the only branch that receives gradients.
• A target network: encoder f_ξ → projector g_ξ, structurally identical but with no predictor. Its weights ξ are never updated by gradient descent. Instead, after every online update, ξ is nudged toward θ by an exponential moving average: ξ ← τξ + (1−τ)θ, with τ close to 1 (BYOL uses values like 0.99, so the target moves slowly).

The loss compares the online network's prediction of the target's output to the target's actual output, with the target output passed through a stop-gradient operator sg(·) that blocks any gradient from flowing back through that branch:

p1 = q_theta(g_theta(f_theta(view1)))     # online prediction, has gradient
z2 = sg(g_xi(f_xi(view2)))                # target output, gradient blocked
Loss = || normalize(p1) - normalize(z2) ||^2

(The whole computation is symmetrised: swap view1 and view2, compute the loss again, and sum both.) SimSiam (Chen and He, 2021) strips this down further and shows the momentum encoder is not even necessary: use one shared encoder for both branches, but stop-gradient the target branch and keep the predictor on the online branch only. Collapse still does not happen. That single fact tells you where the anti-collapse power actually lives: it survives the removal of the momentum encoder, so it cannot be the momentum averaging that prevents collapse. It does not survive removal of the stop-gradient plus predictor asymmetry (delete those and you are back to the naive Siamese network from the previous section). The stop-gradient and the predictor are the load-bearing ingredients; the momentum encoder is a stability refinement on top.

Worked example, part 2: the same toy model, with stop-gradient added

Repeat the scalar experiment, but now give the online branch a predictor weight w_p and stop-gradient the target branch. Reuse the encoder weight w on both branches (this is the SimSiam simplification), but block gradients from flowing back through the branch that produces the target value:

c      = w * v2                 # target output this step, TREATED AS CONSTANT
online = w_p * w * v1           # online prediction, differentiable
Loss   = (online - c)^2

Crucially, when you differentiate this loss, c is held fixed (that is what stop-gradient means): the gradient with respect to w only sees w's appearance in the online branch, not its appearance inside c. Starting from w₀ = 0.5, w_p,0 = 1.0, learning rate 0.5, and the same v1 = 1.0, v2 = 1.2:

Step 0: c = 0.5*1.2 = 0.6, online = 1.0*0.5*1.0 = 0.5, loss = (0.5-0.6)^2 = 0.0100
  dLoss/dw_p = 2*(online-c)*(w*v1) = 2*(-0.1)*(0.5)  = -0.1000
  dLoss/dw   = 2*(online-c)*(w_p*v1) = 2*(-0.1)*(1.0) = -0.2000
  w_p <- 1.0 - 0.5*(-0.1000) = 1.0500
  w   <- 0.5 - 0.5*(-0.2000) = 0.6000

Step 1: c = 0.6*1.2 = 0.72, online = 1.05*0.6*1.0 = 0.63, loss = (0.63-0.72)^2 = 0.0081
  w_p <- 1.1040,  w <- 0.6945

Step 2: c = 0.6945*1.2 = 0.8334, online = 1.104*0.6945 = 0.7667, loss = 0.00445
  w_p <- 1.1503,  w <- 0.7681

Every line above was checked by running the corresponding Python (the update rules are exactly the two partial derivatives shown, applied with lr = 0.5):

v1, v2 = 1.0, 1.2
w, wp, lr = 0.5, 1.0, 0.5
for step in range(3):
    c = w * v2                       # stop-gradient: constant this step
    online = wp * w * v1
    loss = (online - c) ** 2
    dLdwp = 2 * (online - c) * (w * v1)
    dLdw  = 2 * (online - c) * (wp * v1)
    wp, w = wp - lr * dLdwp, w - lr * dLdw
    print(step, round(w, 4), round(wp, 4), round(loss, 4))

# output:
# 0 0.6    1.05   0.01
# 1 0.6945 1.104  0.0081
# 2 0.7681 1.1503 0.0044

Compare the two trajectories directly. Without stop-gradient, w went 0.5 → 0.48 → 0.4608 → 0.442368, shrinking toward zero every single step: total collapse. With stop-gradient and a predictor, w went 0.5 → 0.6 → 0.6945 → 0.7681, growing, while the loss still fell (0.01 → 0.0081 → 0.0044). The stop-gradient breaks the exact symmetry that made w = 0 a magnetic fixed point: in the naive version, shrinking w shrinks both the prediction and the target by the same factor, so the gradient always points toward zero. Once the target is treated as a momentarily fixed number rather than a live function of w, the online branch has to move toward a nonzero target rather than co-collapsing with it. This toy example is a simplified illustration, not the full published proof; the rigorous version (Tian, Chen and Ganguli, ICML 2021, analysing "DirectPred" dynamics) works through the eigenspace of a linear predictor in higher dimensions and shows collapse only occurs from a measure-zero set of adversarial initialisations. But the toy trace above already shows the qualitative mechanism correctly: it is the asymmetry, not the presence of a momentum encoder or a large batch, that keeps the weights away from the trivial solution.

The other route beyond contrastive: predict what you cannot see

BYOL and SimSiam still compare two embeddings of a whole image. A second, structurally different family of self-supervised methods sidesteps the collapse question altogether by never comparing two embeddings in the first place. You already met this idea for text: BERT's masked language modelling hides 15% of input tokens and trains the network to predict the missing token from the surrounding context, using ordinary cross-entropy loss against the true vocabulary index. Masked Autoencoders (MAE, He et al., 2021) transplant the same idea to images: split an image into patches, mask out a large fraction of them (MAE uses as much as 75%), encode only the small set of visible patches with a Vision Transformer, and have a lightweight decoder reconstruct the raw pixel values of the masked patches. The loss is mean-squared error between reconstructed and true pixels, computed only on the masked positions.

Ask why this family does not need a stop-gradient trick, and the answer follows directly from what caused collapse in the Siamese case. Collapse happened because the loss compared two network outputs to each other, and a constant network output was a valid, cost-free way to make those two outputs agree. Masked reconstruction compares a network's output to the actual pixel values of the masked patches, which are fixed, external, and vary from image to image (a green leaf patch, a brown lesion patch, a background patch are numerically different targets). A decoder that outputs a constant vector regardless of input incurs large, nonzero reconstruction error on every image whose masked patches are not themselves constant, so the trivial collapse solution is simply not a minimiser of this loss. The self-supervision here comes from the structure of the data itself (predict occluded regions of a physical scene from visible ones), not from making two views of a network agree, so there is no symmetric feedback loop to break in the first place. This is the deeper reason masked prediction and Siamese matching are treated as two separate "beyond contrastive" families rather than variations on one idea: they fail (or do not fail) for different structural reasons.

Choosing between the two families

For the agri-tech team's constraint (five million unlabelled leaf images, one GPU, batch size 256), BYOL or SimSiam is the more direct fix for the specific failure they hit, because that failure was about batch-size-dependent negative sampling, and both methods remove negatives from the objective entirely; batch size stops being a statistical-estimation parameter and becomes a pure compute-throughput knob. Masked patch reconstruction (an MAE-style pretraining) is also batch-size-insensitive for the same underlying reason (no negatives, no comparison across images in a batch at all) and is often cheaper per step because only the small visible-patch subset passes through the (expensive) encoder, with the decoder handling the masked majority. Between the two, the practical trade-off is architecture: MAE assumes a patch-based Vision Transformer backbone and a masking ratio tuned to the redundancy of the domain (natural images tolerate 75% masking because neighbouring patches are highly correlated; a domain with less spatial redundancy, like a single disease lesion crowding a whole leaf image, may need a lower masking ratio to leave enough of the diagnostic region visible), while BYOL and SimSiam are architecture-agnostic and slot behind any convolutional or transformer encoder with only a small projector-and-predictor head added on top.

Diagram: the asymmetric loop that stops collapse

Active recall

Attempt every question before reading the answer beneath it.

1. Why does training two copies of the same encoder to minimise the squared distance between embeddings of two augmented views (with ordinary shared gradients, no stop-gradient) always reach a global minimum of exactly zero regardless of what the input image is? What is this failure mode called?
2. In the scalar toy model with v1 = 1.0, v2 = 1.2, starting weight w₀ = 0.5, and learning rate 0.5, the naive symmetric loss shrinks w by a constant multiplicative factor every step. What is that factor, derived from the loss function, and why is it always strictly less than 1 whenever the two augmented views differ at all?
3. BYOL keeps a momentum (EMA) target encoder; SimSiam removes it and just stop-gradients a direct weight copy, yet both avoid collapse. What does this comparison tell you about which ingredient is doing the actual anti-collapse work: the momentum averaging, or the stop-gradient plus predictor asymmetry?
4. Why does masked image or token reconstruction (MAE, BERT-style masked language modelling) not need a stop-gradient trick to avoid collapse, when Siamese embedding-matching methods (BYOL, SimSiam) do?
5. The agri-tech team can only fit a batch size of 256 on their single GPU. Between SimCLR and BYOL, which is architecturally the better fit for this constraint, and why specifically, in terms of what each method's loss function depends on?
6. Suppose you initialise BYOL's online and target networks identically and then set the momentum coefficient to exactly τ = 1, so the target network never updates at all after initialisation. Using the stop-gradient argument from this chapter, would training still avoid collapse to a trivial constant, or would it behave more like the naive Siamese case? Reason about it before checking the answer.

Answers

1. Because the loss is a squared distance between two outputs of one shared function, and a function that outputs a fixed constant vector regardless of its input makes that squared distance zero for every possible pair of augmented views, of every image, with no dependence on actually encoding the image content. This is representation collapse: the network satisfies the loss perfectly while discarding all information about the input, so a linear probe trained on the resulting embeddings performs at chance.
2. The loss is L(w) = (v1-v2)²·w², so dL/dw = 2(v1-v2)²·w, and one gradient step gives w_{n+1} = w_n(1 - 2·lr·(v1-v2)²). With lr=0.5 and (v1-v2)²=0.04, the factor is 1 - 2(0.5)(0.04) = 0.96, matching the traced values 0.5 → 0.48 → 0.4608 → 0.442368. The factor is strictly below 1 whenever v1≠v2 because (v1-v2)² is then strictly positive, and for any positive learning rate small enough to keep training stable, 1 - 2·lr·(v1-v2)² < 1; the weight is pulled toward zero on every single step with no floor, which is exactly why the process never stabilises anywhere except w=0.
3. Since collapse is avoided with the momentum encoder removed (SimSiam) but reappears the moment stop-gradient and the predictor asymmetry are also removed (the naive Siamese case worked out in this chapter), the momentum encoder cannot be the essential anti-collapse ingredient; at most it improves training stability and final representation quality. The stop-gradient combined with the predictor head, which breaks the symmetric feedback loop where shrinking one branch automatically shrinks the other, is the mechanism actually responsible for avoiding collapse, as the second worked example demonstrated numerically.
4. Siamese matching methods compare two outputs produced by the network itself, so a constant output is a valid, zero-cost way to satisfy the loss, which is why an artificial asymmetry (stop-gradient) has to be introduced to break that shortcut. Masked reconstruction compares a network output to the actual, externally fixed pixel or token values at the masked positions, which vary across images and positions; a constant output cannot match varying external targets, so it incurs large loss and is never a minimiser. The anti-collapse property comes from the nature of the target (real, varying data) rather than from any architectural trick.
5. BYOL is the better fit. SimCLR's InfoNCE loss is a function of the batch, because its denominator sums over every negative in the batch, so linear-eval quality degrades as batch size shrinks (fewer, noisier negatives per anchor). BYOL's loss compares only the online prediction to its own target for the same image; it has no negatives and no batch-level term at all, so a batch of 256 changes only throughput, not the statistical quality of the training signal.
6. With τ=1 the target network is frozen at its random initial weights forever; it never moves and never receives gradients. The stop-gradient argument still applies: the online network is being trained to predict a fixed, externally-determined (if useless) target, not to co-collapse with a partner that shrinks alongside it. The stop-gradient still blocks the symmetric feedback loop that caused collapse in the naive case, so collapse is still avoided in principle; in practice, however, a target frozen at random initialisation provides only random, uninformative targets, so while representation collapse is avoided, the representations learned will be low quality, since the online network is chasing a target that carries no information about which augmentations correspond to the same image content in any semantically useful way. This is why BYOL uses a slowly-updating EMA target rather than either a fast-updating shared copy or a permanently frozen one: it needs the target to be stable enough to break the collapse loop but still informative enough to improve over training.

Think About It

Think about this: How would you explain self-supervised learning: beyond contrastive to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind self-supervised learning: beyond contrastive, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.