Contrastive Learning: Learning from Similarities

Open a video-KYC flow on any Indian bank's app or a UPI app doing periodic re-verification, and it asks you to take a selfie. Within seconds it decides: same person as the PAN or Aadhaar photo on file, or not. Somewhere behind that decision sits a neural network that has never seen your face during training. It was trained before you ever opened that account, on a completely different set of people, and yet it correctly separates "you" from "not you" on the first try. A standard classifier cannot do this. This chapter is about the training objective that can — contrastive learning — and about the broader idea it belongs to: learning useful structure from data without needing a label for every example.

Why a classifier cannot do e-KYC

You already know how to train a classifier: a softmax output layer with one neuron per class, cross-entropy loss, and enough labelled examples per class for gradient descent to carve out decision regions. Try applying that recipe to face verification for a bank with 40 crore customers. The output layer would need 40 crore neurons, one per identity — computationally absurd, and pointless anyway, because tomorrow the bank onboards a few lakh new customers whose faces were never in the training set. A closed-set classifier has no neuron reserved for a person it has never met. Even ignoring scale, most identities have only one or two enrolment photos, which is nowhere near enough data to fit a reliable per-class decision boundary.

The actual requirement is different from classification. Nobody needs the network to say "this is person #4,829,113." They need it to say "this photo and that photo show the same person" — a same/different judgement between any two images, including images of people the network has never encountered. That is a metric-learning problem: learn a function that maps every face to a point in some vector space such that photos of the same person land close together and photos of different people land far apart, and make that property hold for identities outside the training set too. Verification then becomes arithmetic: embed both photos, measure the distance between the two points, and accept if the distance is below a threshold calibrated on a held-out validation set of genuine and impostor pairs (this threshold-setting step is itself a standard exercise — trace out the false-accept and false-reject rates as the threshold varies, and pick the operating point the bank's risk policy demands).

Contrastive learning is the family of training objectives built to produce exactly that kind of embedding space. The name says it directly: you train by contrasting pairs — pulling together pairs that should be similar, pushing apart pairs that should not.

The general recipe: anchor, positive, negative

Every contrastive method shares the same three ingredients:

• An encoder f_θ, a neural network that maps a raw input (an image, a sentence, a signal) to a fixed-length vector — the embedding.
• A similarity function on embeddings, usually squared Euclidean distance or cosine similarity.
• A sampling scheme that, for a chosen anchor example, produces a positive (something that should end up nearby in embedding space) and one or more negatives (things that should end up far away).

What differs across methods is how positives and negatives are obtained, and the exact loss shape. Three landmark formulations, roughly in historical order:

1. Pairwise contrastive loss (Chopra, Hadsell & LeCun, 2005): given a pair with label y = 1 (same) or y = 0 (different), L = y·d² + (1−y)·max(0, m−d)², where d is the embedding distance and m is a margin. Same-pairs are pulled together with no limit; different-pairs are pushed apart only until they clear the margin, then the loss for that pair is zero — the network stops wasting gradient on negatives that are already far enough.
2. Triplet loss (Schroff, Kalenichenko & Philbin, FaceNet, Google, 2015): sample an anchor a, a positive p (same identity), and a negative n (different identity) together, and require the anchor-positive distance to be smaller than the anchor-negative distance by at least a margin: L = max(0, d(a,p)² − d(a,n)² + margin). FaceNet is the direct ancestor of the face-embedding systems behind most e-KYC pipelines in production today.
3. InfoNCE / NT-Xent (used in SimCLR, Chen, Kornblith, Norouzi & Hinton, Google Research, 2020): instead of one negative at a time, treat the problem as a classification-over-a-batch task — pick out the true positive from among many candidates using a softmax, so every other item currently sitting in the minibatch supplies a negative for free, with no separate negative-sampling step required.

The first two need labels to know which pairs are positive and which are negative — that is supervised contrastive learning, and it is exactly what FaceNet uses (the label is identity). The third can be run with zero human labels at all, which is what makes it self-supervised, and it is the version this chapter's subject is really pointing at. We will work through both, starting with the labelled triplet case because its gradient is short enough to derive completely by hand, then move to the label-free case.

Worked example 1: one full gradient step of triplet loss

Take toy 2-D embeddings (real face embeddings are 128 or 512 dimensions; 2-D keeps the arithmetic checkable by hand while the mechanism is identical):

anchor  a = (1.0, 2.0)   # a selfie taken today
positive p = (1.3, 2.4)  # same person, an older enrolment photo
negative n = (0.6, 1.7)  # a different person, but a poor embedding places them nearby
margin  m = 0.2

Squared Euclidean distances:

d(a,p)^2 = (1.0-1.3)^2 + (2.0-2.4)^2 = 0.09 + 0.16 = 0.25
d(a,n)^2 = (1.0-0.6)^2 + (2.0-1.7)^2 = 0.16 + 0.09 = 0.25

The two distances are identical — as far as this embedding currently knows, the impostor is exactly as "close" as the real match, which is a genuinely dangerous state for a verification system. Plugging into the triplet loss:

L = max(0, d(a,p)^2 - d(a,n)^2 + margin)
  = max(0, 0.25 - 0.25 + 0.2)
  = 0.20

Now derive the gradient with respect to the anchor. Writing d(a,p)² = (a−p)·(a−p) and d(a,n)² = (a−n)·(a−n), and using ∂/∂a[(a−p)·(a−p)] = 2(a−p):

dL/da = 2(a - p) - 2(a - n)
      = 2a - 2p - 2a + 2n
      = 2(n - p)

This is worth pausing on: the a-terms cancel. The gradient pushing the anchor around does not depend on where the anchor currently sits — only on the vector from the negative to the positive. Geometrically, one gradient-descent step moves the anchor along the direction (p − n): straight from the impostor toward the genuine match. Plugging in numbers, n − p = (0.6−1.3, 1.7−2.4) = (−0.7, −0.7), so the gradient is 2(−0.7, −0.7) = (−1.4, −1.4). With learning rate 0.1:

a_new = a - 0.1 * (-1.4, -1.4) = (1.0 + 0.14, 2.0 + 0.14) = (1.14, 2.14)

Recomputing the distances at the new anchor position:

d(a_new,p)^2 = (1.14-1.3)^2 + (2.14-2.4)^2 = 0.0256 + 0.0676 = 0.0932
d(a_new,n)^2 = (1.14-0.6)^2 + (2.14-1.7)^2 = 0.2916 + 0.1936 = 0.4852
new L = max(0, 0.0932 - 0.4852 + 0.2) = max(0, -0.1920) = 0

A single step nearly halved the distance to the genuine match (0.25 → 0.093) and almost doubled the distance to the impostor (0.25 → 0.485), driving the loss to exactly zero. Every value here was computed independently in Python and matches the hand derivation to the displayed precision. This is the entire mechanism of supervised contrastive learning: identity labels tell you which pairs to pull and which to push, and the geometry does the rest, one triplet at a time, across millions of triplets sampled from the training set.

Removing the labels: SimCLR and self-supervised contrastive learning

Triplet loss still needed identity labels to pick p and n. The self-supervised version asks a sharper question: can you build a useful embedding space with no labels whatsoever — not even "these two photos are the same person"? SimCLR's answer is to manufacture positive pairs mechanically, from a single unlabelled image, using data augmentation.

Take one image x from an unlabelled pile — a Flipkart product photo, a satellite tile, anything. Apply two independently sampled random augmentations to it (random crop and resize, colour jitter, Gaussian blur, horizontal flip): you get two different-looking views, t(x) and t′(x). Because both views come from the same underlying image, they are declared a positive pair by construction — no human ever looked at them and confirmed "same." Every other image sitting in the same minibatch, and its own two augmented views, become negatives for this anchor automatically. Build a minibatch of N images, augment each one twice, and you get 2N embeddings containing exactly N positive pairs and 2N(N−1) negative pairs, all without a single label.

Architecturally: both views pass through the same encoder f_θ (identical weights — the "Siamese" arrangement) producing representations h₁, h₂, and then through a small additional network, the projection head g_θ (typically a two-layer MLP), producing the actual vectors z₁, z₂ that the loss operates on. A detail worth knowing precisely because it surprises people: the contrastive loss is computed in z-space, but once training finishes, the projection head is discarded and downstream tasks (classification, retrieval, detection) use h, the representation one layer earlier. The SimCLR authors found g_θ specializes z for the specific business of telling augmented views apart, which throws away information — colour, exact orientation — that other tasks actually need. h keeps that information; z does not.

The InfoNCE / NT-Xent loss: worked numeric example

For an anchor embedding z_i with positive z_j, temperature τ, and similarity function sim(u,v) = cosine similarity, the NT-Xent loss for that anchor is:

L_i = -log( exp(sim(z_i, z_j)/τ) / sum_{k != i} exp(sim(z_i, z_k)/τ) )

This is a softmax classification loss where the "correct class" is the true positive among all 2N−1 other embeddings in the batch. Work it through with a concrete batch of two images (A and B), each with two augmented views, using unit vectors so cosine similarity is a plain dot product:

z1 = (0.6, 0.8)    # image A, view 1
z2 = (0.8, 0.6)    # image A, view 2  -- positive pair with z1
z3 = (-0.6, 0.8)   # image B, view 1
z4 = (-0.8, -0.6)  # image B, view 2  -- positive pair with z3

All six pairwise cosine similarities:

sim(z1,z2) = 0.6*0.8 + 0.8*0.6  =  0.96
sim(z1,z3) = 0.6*(-0.6)+0.8*0.8 =  0.28
sim(z1,z4) = 0.6*(-0.8)+0.8*(-0.6) = -0.96
sim(z2,z3) = 0.8*(-0.6)+0.6*0.8 =  0.00
sim(z2,z4) = 0.8*(-0.8)+0.6*(-0.6) = -1.00
sim(z3,z4) = -0.6*(-0.8)+0.8*(-0.6) =  0.00

z1 and z2 (the true positive pair, same source image) score highest at 0.96; z1 and z4 (different images) score lowest at −0.96 — the space is already fairly well organised. Take τ = 0.5 and compute the loss for anchor z1 against positive z2, with z3 and z4 as its two negatives:

l12 = 0.96/0.5 = 1.92    exp(1.92) = 6.8210
l13 = 0.28/0.5 = 0.56    exp(0.56) = 1.7507
l14 = -0.96/0.5 = -1.92  exp(-1.92) = 0.1466

denominator = 6.8210 + 1.7507 + 0.1466 = 8.7182
ratio       = 6.8210 / 8.7182 = 0.7824
L1          = -log(0.7824) = 0.2454

The same computation for the other three anchors (z2 against z1, z3 against z4, z4 against z3) gives losses of 0.1540, 1.3219 and 0.2484; the batch loss is their average, 0.4924. z3's loss is by far the largest of the four — look back at the similarity table and you can see why: sim(z3,z4) = 0.00, its own positive, is no higher than sim(z2,z3) = 0.00, one of its negatives. The softmax genuinely cannot tell z3's true partner apart from a decoy yet, so that anchor still carries a large gradient signal pushing the encoder to fix exactly that confusion. Verified in code:

import math

def dot(a, b):
    return a[0]*b[0] + a[1]*b[1]

z = [(0.6, 0.8), (0.8, 0.6), (-0.6, 0.8), (-0.8, -0.6)]
tau = 0.5

def ntxent(i, j, z):
    num = math.exp(dot(z[i], z[j]) / tau)
    denom = sum(math.exp(dot(z[i], z[k]) / tau) for k in range(len(z)) if k != i)
    return -math.log(num / denom)

losses = [ntxent(0, 1, z), ntxent(1, 0, z), ntxent(2, 3, z), ntxent(3, 2, z)]
print(losses)                 # [0.2454, 0.1540, 1.3219, 0.2484]
print(sum(losses) / 4)        # 0.4924

Scale this from a toy batch of 2 images to a real SimCLR run — batch sizes of several thousand images are typical, precisely because a bigger batch means more negatives per anchor, which sharpens the softmax and gives the encoder a harder, more informative task to solve at every step.

Diagram: the SimCLR pipeline

The misconception: "pretraining gives you a classifier"

Students who first meet contrastive learning right after studying cross-entropy classifiers often assume the contrastive-trained network can already answer "what is this?" the moment training finishes. It cannot, and this is not a minor technicality — it is the central design point of self-supervised pretraining. NT-Xent, triplet loss and pairwise contrastive loss all optimise purely geometric structure: which points are near which. None of them ever compute a class probability, and none of them ever see a downstream label like "dog," "e-commerce product," or "genuine customer." What comes out of training is an encoder that produces well-organised embeddings — nothing more.

Turning that geometry into an actual decision needs one more step, and which step depends entirely on the task. For verification (the e-KYC case this chapter opened with), the step is a distance threshold, calibrated afterward on a labelled set of known genuine and impostor pairs by sweeping the threshold and reading off false-accept and false-reject rates. For classification (say, sorting product images by category), the step is training a small linear layer — a "linear probe" — on top of the frozen embeddings, using a labelled dataset that can be far smaller than what training the encoder from scratch would have needed, because the encoder has already done the hard work of organising the space; the probe only has to draw straight lines through it. This two-stage pattern — self-supervised pretraining on unlabelled data, then a lightweight labelled step on top — is precisely why the field calls it "pretraining followed by linear evaluation," never "self-supervised classification." Skipping that second step and expecting labelled predictions directly out of a contrastively-trained encoder is the misconception to drop.

Active recall

Attempt each question before reading its answer.

1. Why can't a bank train an ordinary softmax classifier to do face verification for its entire customer base?
2. Compute the triplet loss for anchor a = (0, 0), positive p = (0.6, 0.8), negative n = (0.1, 0.05), with margin 0.3.
3. In SimCLR, where do the "negative" examples for a given anchor come from? Does anyone label them as negative?
4. A classmate says: "I trained a SimCLR model on 2 lakh unlabeled Zomato dish photos, so it can now tell me which cuisine a new photo belongs to." What is wrong with this claim, and what extra step is actually needed?
5. In the triplet-loss gradient derivation, why does the gradient with respect to the anchor, 2(n − p), not depend on the anchor's own position a?

Answers

1. The customer base is open-set and constantly growing — the output layer would need one neuron per identity, retrained every time someone new signs up, and most identities have only one or two enrolment photos, far too few to fit a reliable class boundary. Verification needs a same/different judgement that generalises to identities never seen in training, which is a metric-learning problem, not a classification problem.
2. d(a,p)² = 0.6² + 0.8² = 0.36 + 0.64 = 1.0. d(a,n)² = 0.1² + 0.05² = 0.01 + 0.0025 = 0.0125. L = max(0, 1.0 − 0.0125 + 0.3) = 1.2875. The loss is large because the negative sits almost on top of the anchor while the true positive is comparatively far away — exactly the failure mode the loss is designed to penalise heavily.
3. Every other image currently in the same minibatch (and its own two augmented views) automatically serves as a negative for the anchor. Nobody labels anything "negative" — it is simply assumed, by construction, that two different source images are dissimilar. This is what makes the whole scheme label-free.
4. SimCLR pretraining alone never sees cuisine labels, so the encoder has no notion of "cuisine" as a category — it only knows how to place augmented views of the same photo near each other and different photos apart. To get cuisine predictions, a linear probe (or a fine-tuned head) must be trained on top of the frozen embeddings using a labelled set of dish-photo-to-cuisine pairs. The unlabeled pretraining reduces how much labelled data that second step needs, but it does not eliminate the need for it.
5. Because the loss is quadratic in a, and taking the derivative of a quadratic (a−p)·(a−p) − (a−n)·(a−n) with respect to a produces terms 2a − 2p − 2a + 2n; the two "2a" terms are equal and opposite and cancel algebraically, leaving only 2(n − p). This is a property of squared-Euclidean triplet loss specifically — it means the direction of the update step is fixed entirely by where the positive and negative currently sit, not by the anchor's own coordinates.

Think About It

Think about this: How would you explain contrastive learning: learning from similarities to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind contrastive learning: learning from similarities, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.