Self-Supervised Learning: Learning Without Labels

The labeling bottleneck

A Swiggy or Zomato-scale platform accumulates tens of millions of food photographs every year — uploaded by restaurants, by delivery partners confirming a pickup, by customers posting a review photo. Almost none of it carries a machine-usable label. Nobody tags each image with "biryani" or "dosa" or "not food" as it is uploaded. If the platform wants a model that can auto-tag dish category from an image — useful for search, for menu deduplication, for flagging mismatched photos — the standard recipe from a Grade 10 supervised-learning course says: collect a labeled dataset, train a classifier. But labeling is a human bottleneck. A team of annotators looking at a photo for five seconds each, working eight-hour days, can produce perhaps 5,000–6,000 labels a day. Getting to a training set of even 200,000 labeled dishes — modest by deep-learning standards — costs weeks of paid annotation labor before a single gradient step is taken. The Princeton team that built ImageNet (led by Fei-Fei Li, who moved to Stanford shortly after) ran into exactly this wall in the late 2000s: to label millions of images across thousands of categories, they had to crowdsource the work through Amazon Mechanical Turk over an extended multi-year effort, because no small team could hand-label at that scale.

Meanwhile the unlabeled photos keep arriving for free. The question self-supervised learning answers is: can a model learn useful visual (or linguistic) structure directly from that raw, label-free flood, so that only a small labeled set is needed afterward to point the model at the specific task you care about? The answer, established repeatedly since the mid-2010s in vision and language, is yes — and the mechanism is precise enough to derive by hand, which is what this chapter does.

A formal definition

Self-supervised learning (SSL) is not a new loss function or a new kind of neural network. It is a strategy for constructing the target y in an ordinary supervised training loop — cross-entropy, contrastive, or otherwise — out of the input x itself, instead of collecting y from a human annotator. A pretext task is the specific rule used to manufacture that target. Three common families:

• Transformation prediction. Apply a known transformation to an unlabeled image and train the network to recover which transformation was applied. Gidaris, Singh, and Komodakis (ICLR 2018) rotated images by 0°, 90°, 180°, or 270° and trained a CNN to predict the rotation as a 4-way classification problem — the label is free because you chose the rotation yourself.
• Masked / context prediction. Hide part of the input and train the network to reconstruct or predict it from the rest. Word2Vec (Mikolov, Chen, Corrado, and Dean, 2013) predicts a word from its surrounding context (or vice versa); BERT (Devlin, Chang, Lee, and Toutanova, NAACL 2019) masks out roughly 15% of input tokens and trains a Transformer encoder to predict the masked tokens from the unmasked ones. The "label" at every masked position is just the word that was already sitting there before you hid it.
• Contrastive / instance discrimination. Produce two different augmented views of the same underlying instance and train an encoder so that the two views' embeddings are pulled together while embeddings from different instances are pushed apart. SimCLR (Chen, Kornblith, Norouzi, and Hinton, ICML 2020) is the canonical example for images.

In every case the training loop is genuinely supervised — there is a loss function comparing a prediction to a target, and gradients flow through backpropagation exactly as in the Grade 11 deep-learning architectures you have already studied. What is absent is a human in the labeling loop. This distinction matters enough that it is worth stating as a misconception now, before the worked example, so you read the rest of the chapter with the right mental model.

Kill the misconception: SSL is not unsupervised learning

Students who have just learned k-means clustering or PCA in Grade 10 often fold self-supervised learning into the same bucket: "no labels, so it must be unsupervised learning." This is wrong, and the difference is not cosmetic.

Unsupervised learning has no target variable at all. K-means minimizes within-cluster distance; PCA minimizes reconstruction error against the data's own covariance structure. There is no notion of a "correct answer" being predicted — the algorithm is discovering structure, full stop.

Self-supervised learning has a target variable at every training step — a specific rotation angle, a specific masked token, a specific paired embedding — and a loss function that scores how wrong the prediction was, exactly like supervised classification. The only thing missing is a human writing that target down; the target is derived algorithmically from the unlabeled input. A rotation-prediction network is trained with ordinary cross-entropy against a 4-class label, precisely the same loss used to train a labeled cat-vs-dog classifier. This is why SSL pretraining slots directly into the same optimizer, the same backpropagation machinery, and the same overfitting/regularization concerns you already know from supervised deep learning — it borrows all of that machinery and changes only where the label comes from.

Worked example: computing a contrastive loss by hand

The most widely used SSL objective in vision today is the contrastive loss used in SimCLR, called NT-Xent (normalized temperature-scaled cross-entropy), which is a specific instance of the more general InfoNCE loss. The mechanics are worth deriving fully by hand once, because every contrastive SSL method — SimCLR, MoCo, and their descendants — is a variant of this same computation.

Setup: take a mini-batch of N = 2 unlabeled images, call them A and B. Apply two independent random augmentations (crop, flip, color jitter) to each, producing four augmented views: a₁, a₂ (both from image A) and b₁, b₂ (both from image B). Pass all four through a shared-weight encoder to get four embedding vectors, which SimCLR L2-normalizes so that cosine similarity is a plain dot product. Suppose the encoder has already learned enough to produce these (illustrative, already-normalized) embeddings:

a1 = ( 1.0,  0.0)
a2 = ( 0.8,  0.6)
b1 = ( 0.0,  1.0)
b2 = (-0.6,  0.8)

The positive pair for a₁ is a₂ (same source image); every other view in the batch — b₁ and b₂ — is treated as a negative. The pairwise cosine similarities (dot products, since every vector has unit length) are:

sim(a1,a2) =  0.80      sim(a1,b1) =  0.00
sim(a1,b2) = -0.60      sim(a2,b1) =  0.60
sim(a2,b2) =  0.00      sim(b1,b2) =  0.80

The NT-Xent loss for view i, with positive view j and temperature τ, is:

L_i = -log( exp(sim(z_i,z_j)/τ) / sum_{k != i} exp(sim(z_i,z_k)/τ) )

Note that the denominator sums over every other view in the batch (2N − 1 = 3 terms here), and the positive similarity appears in both the numerator and the denominator — the loss is a softmax cross-entropy where the "correct class" is the positive view among all 2N − 1 candidates. With τ = 0.5, work through a₁ by hand:

logits for a1: sim(a1,a2)/0.5 = 1.6   (positive)
               sim(a1,b1)/0.5 = 0.0
               sim(a1,b2)/0.5 = -1.2

exp(1.6) = 4.9530,  exp(0.0) = 1.0000,  exp(-1.2) = 0.3012
denominator = 4.9530 + 1.0000 + 0.3012 = 6.2542
ratio = 4.9530 / 6.2542 = 0.7919
L_a1  = -ln(0.7919) = 0.2333

Now a₂, whose negatives (b₁ at similarity 0.60, b₂ at similarity 0.00) are less separated from its positive similarity of 0.80 than a₁'s were:

logits for a2: sim(a2,a1)/0.5 = 1.6   (positive)
               sim(a2,b1)/0.5 = 1.2
               sim(a2,b2)/0.5 = 0.0

exp(1.6) = 4.9530,  exp(1.2) = 3.3201,  exp(0.0) = 1.0000
denominator = 4.9530 + 3.3201 + 1.0000 = 9.2731
ratio = 4.9530 / 9.2731 = 0.5341
L_a2  = -ln(0.5341) = 0.6271

By the symmetry of the similarity values, b₁ reproduces a₂'s computation exactly (its negatives are a₁ at 0.0 and a₂ at 0.6) and b₂ reproduces a₁'s (its negatives are a₁ at −0.6 and a₂ at 0.0). So L_b1 = 0.6271 and L_b2 = 0.2333. The batch loss is the mean over all four views:

L = (0.2333 + 0.6271 + 0.6271 + 0.2333) / 4 = 0.4302

Here is the same computation as runnable code, which should print the value just derived by hand:

import numpy as np

# Four L2-normalized embeddings: a1,a2 from image A; b1,b2 from image B
z = np.array([
    [ 1.0,  0.0],   # a1
    [ 0.8,  0.6],   # a2
    [ 0.0,  1.0],   # b1
    [-0.6,  0.8],   # b2
])

tau = 0.5
positive = {0: 1, 1: 0, 2: 3, 3: 2}  # each view's positive index

sim = z @ z.T                # cosine similarity matrix (rows are unit vectors)
logits = sim / tau

losses = []
for i in range(4):
    num = np.exp(logits[i, positive[i]])
    denom = sum(np.exp(logits[i, k]) for k in range(4) if k != i)
    losses.append(-np.log(num / denom))

print(round(float(np.mean(losses)), 4))   # -> 0.4302

Two things this derivation should make concrete. First, gradient descent on this loss literally rotates the encoder's output vectors: it increases sim(a1,a2) and decreases sim(a1,b1), sim(a1,b2) on every step, which geometrically pulls same-image views together and pushes different-image views apart on the unit sphere — the encoder is never told "this is biryani," only "these two crops came from the same photograph." Second, the whole objective is computable without a single human-provided category label; the only "supervision" is the bookkeeping fact that a₁ and a₂ came from the same source image, which the training pipeline knows automatically because it did the augmenting itself.

Complexity: why contrastive batches got so large, and how MoCo fixed it

Look again at the denominator: for a batch of N images (2N augmented views), every view's loss sums over the other 2N − 1 views, and computing the full similarity matrix costs O((2N)²) — quadratic in batch size. This is not just an inconvenience; it is the reason SimCLR needed batch sizes in the thousands to work well: the more negatives in a batch, the harder the network is forced to discriminate, and quality of the learned representation tracked batch size closely. But every one of those 2N views has to be forward-passed with gradients retained for backpropagation, so GPU memory for storing activations also scales with N — at large enough N, memory becomes the binding constraint, not compute.

He, Fan, Wu, Xie, and Girshick (Momentum Contrast, CVPR 2020) broke this coupling. Instead of drawing all negatives from the current mini-batch, MoCo maintains a queue of K previously computed key embeddings — produced by a second, momentum-updated copy of the encoder whose weights are an exponential moving average of the main encoder's, and whose outputs are stored with gradients detached. A training step needs only N query embeddings with gradients; comparing them against the K-entry queue costs O(N·K) — linear in the batch size N, with the number of negatives K decoupled entirely from it. Concretely, with a mini-batch of N = 256 and a queue of K = 65,536, MoCo performs 256 × 65,536 = 16,777,216 similarity comparisons per step — far more negatives than SimCLR's in-batch approach could offer at that batch size (a same-size SimCLR batch gives a 512 × 512 similarity matrix, roughly 261,632 usable off-diagonal comparisons) — while backpropagation still only has to carry activations for the 256 query images, because the queue's 65,536 keys never need a gradient. This is the standard move in self-supervised system design: separate "how many negatives does the loss see" from "how many samples does backprop have to hold in memory," because only the second one is expensive.

From pretext task to downstream task

A pretrained encoder is not the deliverable — it is an intermediate artifact. The standard two-stage pipeline is: (1) pretrain on the full unlabeled corpus with a pretext-task loss (rotation prediction, masked-token prediction, or contrastive loss), producing an encoder whose representations capture general structure; then (2) attach a small task-specific head and train on a much smaller labeled set. Two evaluation protocols are standard here and worth naming precisely, because papers and job postings use them as fixed vocabulary: linear probing freezes the pretrained encoder entirely and trains only a linear classifier on top of its fixed features — a direct measure of how linearly separable the pretrained representation already is; fine-tuning unfreezes the encoder and updates all its weights (typically at a smaller learning rate) jointly with the new head, usually giving higher final accuracy but requiring more labeled data to avoid overfitting the encoder itself. For the food-photo platform, Stage 1 (rotation prediction or SimCLR-style contrastive pretraining) runs over the full uploaded-photo corpus with zero labels; Stage 2 fine-tunes on perhaps 2,000 dish-labeled photos, which is a labeling job a small team finishes in a day rather than a multi-week crowdsourcing campaign. This exact pattern — self-supervised pretraining on unlabeled text, followed by small-labeled-set fine-tuning — is also what sits underneath the large language models the next stage of this curriculum covers: BERT's masked-language-model pretraining on raw web text is the direct linguistic analogue of the rotation-prediction and contrastive examples above.

A bad pretext task, and why it fails

Not every automatically-generated target is useful. Consider a pretext task of predicting the average RGB brightness of an image from a downsampled version of itself. The network can solve this almost perfectly by learning nothing about shapes, edges, or objects — it only needs a crude global-pooling operation over pixel intensities, a shortcut that trivially minimizes the loss while learning representations useless for, say, dish classification. This failure mode is called shortcut learning, and it is the real design constraint on choosing a pretext task: a good pretext task must be solvable only by extracting the kind of structure the downstream task actually needs. Rotation prediction works because recognizing "this photo is upside down" genuinely requires understanding gravity-consistent object structure (where a bowl's rim sits relative to its base, which way rice grains and gravy pool); predicting average brightness does not.

Active recall

Attempt every question before reading the worked answers below.

1. State, in one sentence, the formal definition of self-supervised learning, then name the pretext task and the downstream task for BERT.
2. True or false, with justification: "Self-supervised learning is just another name for unsupervised learning."
3. In the worked NT-Xent example, the batch had N = 2 images (4 views), and each loss term summed over 3 negatives/positives. If the batch instead held N = 5 images, how many views are in the batch, and how many terms does each view's denominator sum over?
4. Why does rotation prediction force a CNN to learn object-relevant structure, while "predict the average RGB brightness" does not? Name the general failure mode this second task falls into.
5. Recompute the four NT-Xent losses from the worked example using temperature τ = 0.1 instead of τ = 0.5, using the same similarity values. Trace the effect on all four losses, not only a₁'s, and state the qualitative lesson about what temperature controls.
6. A MoCo-style setup uses batch size N = 128 and queue size K = 8,192. How many similarity comparisons happen per step, and why does the queue not increase the memory needed for backpropagation?

Worked answers

1. Self-supervised learning trains a model with an ordinary supervised loss whose target is generated automatically from the input itself, rather than supplied by a human annotator. For BERT: the pretext task is masked-token prediction (recover randomly hidden words from surrounding context); the downstream task is whatever the fine-tuned model is later applied to — sentiment classification, question answering, named-entity recognition, and so on.

2. False. Unsupervised learning (k-means, PCA) has no target variable and no loss comparing a prediction to a "correct answer" — it only discovers structure. Self-supervised learning has a target and a loss at every step (a rotation class, a masked token, a paired embedding); the only thing absent is a human writing that target down. The training loop, loss function, and backpropagation are identical in form to ordinary supervised learning.

3. With N = 5 images, each augmented twice, the batch holds 2N = 10 views. Every view's denominator sums over all other views in the batch, i.e. 2N − 1 = 9 terms.

4. Recognizing whether a photograph is rotated 0°, 90°, 180°, or 270° requires understanding gravity-consistent object structure — where a bowl's rim sits relative to its base, which direction food naturally settles — so a network can only solve the task well by learning object shape and orientation cues transferable to real recognition tasks. Predicting average RGB brightness can be solved by a trivial global-pooling shortcut that requires no understanding of shape or objects at all. This is the general failure mode of shortcut learning: a pretext task solvable by a degenerate, task-irrelevant statistic teaches the encoder nothing useful.

5. Rescaling all four logit sets by 1/0.1 = ×10 instead of ×5 (since logit = similarity / τ):

a1 (positive 0.8, negatives 0.0,-0.6):  logits 8, 0, -6
   exp(8)=2980.96, exp(0)=1, exp(-6)=0.0025
   ratio = 2980.96 / 2981.96 = 0.99966 -> L_a1 = 0.00034

a2 (positive 0.8, negatives 0.6, 0.0):  logits 8, 6, 0
   exp(8)=2980.96, exp(6)=403.43, exp(0)=1
   ratio = 2980.96 / 3385.39 = 0.88053 -> L_a2 = 0.12723

b1 mirrors a2 -> L_b1 = 0.12723
b2 mirrors a1 -> L_b2 = 0.00034

mean loss = (0.00034 + 0.12723 + 0.12723 + 0.00034) / 4 = 0.0638

Every one of the four losses drops relative to τ = 0.5 (mean falls from 0.4302 to 0.0638) — lower temperature sharpens the softmax, so once the positive already holds the largest similarity, more of the probability mass concentrates onto it and the loss shrinks. But the drop is wildly uneven: a₁'s loss falls by a factor of about 690 (0.2333 → 0.00034), while a₂'s falls by only about 5× (0.6271 → 0.12723). At τ = 0.5 the hard case (a₂, whose nearest negative sits at similarity 0.6) contributes 2.7× the loss of the easy case (a₁, whose nearest negative sits at −0.6); at τ = 0.1 that ratio balloons to roughly 374×. The qualitative lesson: temperature does not just rescale the loss uniformly — a low temperature effectively concentrates almost all of the remaining loss (and therefore almost all of the gradient signal) onto the hardest negative pairs, while easy pairs stop contributing gradient almost entirely. This is why temperature is treated as one of the most sensitive hyperparameters in contrastive SSL: too low, and training is driven almost entirely by a handful of confusable hard negatives; too high, and the loss stops distinguishing hard negatives from easy ones at all.

6. Comparisons per step = N × K = 128 × 8,192 = 1,048,576. The queue does not add to backpropagation memory because its 8,192 key embeddings come from a momentum encoder and are stored with gradients detached — no activations need to be retained for them. Only the 128 query embeddings require gradient-carrying activations, so backpropagation memory stays O(N), independent of how large K is set.

Think About It

Think about this: How would you explain self-supervised learning: learning without labels to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where self-supervised learning: learning without labels is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting self-supervised learning: learning without labels to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind self-supervised learning: learning without labels, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.