Contrastive Learning: Learning Representations

A Gallery That Sorts Itself

Open Google Photos, or almost any modern phone gallery app, and look at how it groups your pictures by the people in them. Every photo of your grandmother — festival portraits, blurry candids, a passport-style shot from years ago — ends up in one album, without you ever telling the app who she is. Nobody at Google has seen your family photos in advance. No one typed "this is Grandmother" under fourteen thousand images before the app shipped.

This is strange if you think about it the way you would think about ordinary supervised machine learning, which you have likely already studied: a model learns to map inputs to correct outputs by looking at thousands of labelled examples. Face-grouping on your phone cannot have worked that way for your specific family, because your grandmother was never a labelled class in anyone's training set. What the underlying model has learned instead is not "who is this," but something more general and, in a sense, more useful: whether two photographs show the same person at all, for any two people it has never encountered before. Hand it two face crops of a stranger and it will still say "same" or "different" correctly, most of the time. That skill — learned with no labels naming any individual — is the subject of this chapter, and the family of techniques that builds it is called contrastive learning.

What a Representation Actually Is

Every deep learning model that processes an image, a sentence, or a sound clip does so by first turning it into a list of numbers. An encoder, written as a function f, takes a raw input x — say, the pixels of a face crop — and produces a representation, or embedding, z = f(x): typically a vector of a few hundred numbers. Everything downstream operates on z, not on the raw pixels: classification, search, clustering, and the face-grouping in your gallery.

The entire game of representation learning is to make z useful. A useful embedding space has one defining property: inputs that are semantically alike end up near each other, and inputs that are semantically different end up far apart. When that property holds, even a very simple downstream tool — literally drawing a straight line through the space, or measuring plain distance — can tell people apart, find similar products, or match a song to its cover version. When it doesn't hold, no amount of downstream cleverness rescues a representation where two photos of the same person landed on opposite sides of the vector space.

The traditional way to build such a space is supervised: collect millions of labelled examples and train the encoder to predict the labels directly. This works, but labels are expensive. ImageNet, the dataset that drove much of the last decade's progress in computer vision, contains well over a million labelled training images, assembled over several years of crowdsourced human annotation. For most of the data that actually exists in the world — the billions of unlabelled photos sitting on phones, the audio recorded every day, the text already on the open web — nobody has done, or ever will do, that labelling, because it is too expensive at that scale. Self-supervised learning, and contrastive learning in particular, asks a different question: can the data label itself?

Pull Together, Push Apart

Here is the idea contrastive learning is built on. Take one image — a single photo of your friend Kabir — and produce two different, randomly altered versions of it: crop it differently, adjust the brightness, maybe flip it horizontally. These two versions form a positive pair: they differ at the pixel level but obviously depict the same underlying thing. Now bring in a second, unrelated photo, one of a different friend, Meera. Any view of Meera's photo is a negative relative to Kabir's views — a different underlying instance entirely. Whichever view is currently being compared against everything else is the anchor.

• Anchor — the view being compared against everything else in the batch.
• Positive — a different view of the same underlying instance as the anchor.
• Negative — a view of any other instance in the batch.

The training objective is simple to state: pull the embeddings of positive pairs close together, and push the embeddings of negative pairs far apart. Do this over millions of images, and the only way for the encoder to consistently win this "same or different" game — to reliably recognize that two crops came from one original photo, against a large pool of imposters — is to stop paying attention to superficial pixel arrangement and start representing the actual content of the scene: the identity of a face, the shape of an object, the category of an item. Nobody told the network what a face or a dog is. It discovered a representation that respects those categories purely because that was the only strategy that let it keep winning.

The trick that makes this entirely label-free is that the positive pair is manufactured by data augmentation, not supplied by a human annotator. The "same or different" answer is known for free, because the two augmented views were created by the training pipeline itself. This is the defining move of self-supervised learning: turning a property of the data pipeline into a supervisory signal.

Cosine Similarity

To talk about pulling embeddings together or pushing them apart, we need a number that measures how close two vectors are. Contrastive learning almost always uses cosine similarity:

sim(u, v) = (u · v) / (||u|| ||v||)

where u · v is the dot product of the two vectors and ||u|| is the vector's length, its L2 norm. Cosine similarity measures the angle between two vectors while ignoring how long either one is: it equals 1 when the vectors point in exactly the same direction, 0 when they are perpendicular, and −1 when they point in opposite directions. Encoders used in contrastive learning typically normalize every embedding to unit length before computing similarity. Once every vector sits on the surface of the same unit hypersphere, cosine similarity reduces to a plain dot product — no division required — which is cheap to compute across an entire batch and is the simplification the worked example below relies on.

The NT-Xent Loss

"Pull together, push apart" is an intuition, not yet something you can back-propagate through. SimCLR, the paper that popularized this exact framework in 2020, turns it into a precise loss called NT-Xent (Normalized Temperature-scaled Cross Entropy), a specific case of a more general objective from earlier work called InfoNCE.

Take a batch of N images. Apply two independent random augmentations to each, producing 2N views in total. For any view i in that batch, there is exactly one other view that is its positive — the other augmentation of the same original image — and 2N − 2 views that are negatives, drawn from every other image in the batch. The loss for one such positive pair (i, j) is:

L(i, j) = -log ( exp(sim(z_i, z_j) / τ) / D )

where D = Σ exp(sim(z_i, z_k) / τ), summed over every other
view k in the batch of 2N views (k ≠ i)

Read the fraction inside the log as a softmax: the numerator scores how similar the anchor is to its true positive; the denominator sums that same score over the positive and every negative. The whole expression is exactly the cross-entropy loss used for an ordinary classification problem, except the "class" being predicted is "which one of these 2N − 1 candidates is the real match for the anchor" — a classification problem with no human-written labels, because the correct answer is simply "the other augmented copy of the same image," which the training pipeline already knows.

The symbol τ (tau) is the temperature, a small positive number, typically well below 1, that rescales similarities before the softmax. Dividing by a small τ stretches the differences between similarity scores before exponentiating them, so the loss punishes the model far more harshly for confusing the anchor with a close, "hard" negative than with an obviously unrelated one. A larger τ flattens the distribution and treats all negatives more equally. Temperature is one of the most sensitive hyperparameters in contrastive learning: too low, and training chases a handful of hard negatives unstably; too high, and the loss stops distinguishing hard cases from easy ones at all.

The negative terms in the denominator are not a minor detail — they are what keeps the objective from being cheated. If a loss only rewarded pulling positive pairs together, with no penalty for where negatives end up, the encoder could satisfy it perfectly by mapping every input, regardless of content, to the exact same point: every positive-pair distance would be zero, minimizing the loss completely, while the representation carried no information at all. This failure mode is called representation collapse, and it is the reason contrastive losses always include an explicit repulsive term. A newer family of self-supervised methods, including BYOL and SimSiam, manages to avoid collapse without any negative pairs, using asymmetric network tricks instead — but that is a story for a different chapter; everything here relies on negatives doing real work.

Worked Example: Four Views, One Loss

Suppose a (very small, very much toy) encoder has already processed two photos, each augmented twice, producing four embeddings. A1 and A2 are two crops of the same photo of Meera; B1 and B2 are two crops of the same photo of Kabir. To keep the arithmetic exact, use an embedding size of just 2, with every vector already normalized to unit length:

• A1 = (1.0, 0.0)
• A2 = (0.8, 0.6)
• B1 = (0.0, 1.0)
• B2 = (−0.6, 0.8)

Check that each vector really is unit length before going further — for A2: 0.8² + 0.6² = 0.64 + 0.36 = 1.0. The other three check out the same way. Since every embedding lies on the unit circle, cosine similarity between any two of them is just their dot product; there is no division left to do.

Take A1 as the anchor, with batch size N = 2 so there are 2N = 4 views total, temperature τ = 0.5, and positive partner A2. First compute the three similarities A1 needs — against its positive A2, and its two negatives, B1 and B2:

sim(A1, A2) = (1.0)(0.8) + (0.0)(0.6) = 0.80
sim(A1, B1) = (1.0)(0.0) + (0.0)(1.0) = 0.00
sim(A1, B2) = (1.0)(−0.6) + (0.0)(0.8) = −0.60

Divide each by the temperature, τ = 0.5, and exponentiate:

0.80 / 0.5 =  1.60   →  exp( 1.60) ≈ 4.9530
0.00 / 0.5 =  0.00   →  exp( 0.00) = 1.0000
−0.60 / 0.5 = −1.20  →  exp(−1.20) ≈ 0.3012

Sum the three exponentials to get the denominator of the softmax. Notice the positive pair's own exponential, exp(1.60), is included in this sum along with the negatives — that is what makes it a softmax over all the candidates, not just a comparison against one:

D = 4.9530 + 1.0000 + 0.3012 = 6.2542

The loss for this one positive pair is the negative log of the positive's share of that sum:

L(A1, A2) = −log(4.9530 / 6.2542) = −log(0.7920) ≈ 0.233

A loss of 0.233 corresponds to the model assigning roughly 79% of its "probability mass" to the correct match among three candidates — a good sign, and it matches the geometry: A1 and A2 have a cosine similarity of 0.80, clearly higher than A1's similarity to either of Kabir's views (0.00 and −0.60). NT-Xent is symmetric, so a full pass over this batch also computes L(A2, A1), L(B1, B2), and L(B2, B1), using A2, B1, and B2 in turn as the anchor, and the batch loss handed to the optimizer is the average of all four.

Checking It in Code

The same computation, carried out in full for all four views, is only a few lines of NumPy — and it is worth running, because it is easy to make a sign error or an off-by-one mistake doing this arithmetic by hand:

import numpy as np

A1 = np.array([1.0, 0.0])
A2 = np.array([0.8, 0.6])
B1 = np.array([0.0, 1.0])
B2 = np.array([-0.6, 0.8])

views = np.stack([A1, A2, B1, B2])   # shape (4, 2)
tau = 0.5

def nt_xent_loss(views, positive_idx, tau):
    n = views.shape[0]
    sims = (views @ views.T) / tau      # cosine similarity, since rows are unit vectors
    losses = []
    for i in range(n):
        j = positive_idx[i]
        numerator = np.exp(sims[i, j])
        mask = np.ones(n, dtype=bool)
        mask[i] = False                 # exclude the anchor itself
        denominator = np.exp(sims[i][mask]).sum()
        losses.append(-np.log(numerator / denominator))
    return np.array(losses)

# A1 pairs with A2 (indices 0, 1); B1 pairs with B2 (indices 2, 3)
positive_idx = [1, 0, 3, 2]
losses = nt_xent_loss(views, positive_idx, tau)

for name, loss in zip(["A1", "A2", "B1", "B2"], losses):
    print(f"{name}: {loss:.3f}")
print(f"batch loss: {losses.mean():.3f}")

Running this prints A1: 0.233 and B2: 0.233, matching the hand calculation above, alongside A2: 0.627 and B1: 0.627 — noticeably higher. That asymmetry is not a bug; it is the geometry of the four points telling you something real. A2 sits closer to B1 (similarity 0.60) than A1 sits to either of Kabir's crops, so when A2 is the anchor, its correct match has to compete harder against a genuinely confusable negative. The final batch loss — (0.233 + 0.627 + 0.627 + 0.233) / 4 ≈ 0.430 — is what gradient descent actually minimizes, and driving it down means the optimizer will specifically push A2 and B1 further apart, since that pair contributes the most error. This is why contrastive learning is often described as doing its own hard-negative mining for free: pairs that are close but wrong contribute more loss, and therefore a larger gradient, than pairs that were never going to be confused in the first place.

From Faces to Foundation Models: SimCLR, MoCo, and CLIP

The idea of learning similarity directly, rather than learning to name a fixed list of identities, predates the NT-Xent formula above. FaceNet (Schroff, Kalenichenko, and Philbin, Google, 2015) trained a face-embedding network using a triplet loss — anchor, positive, negative, exactly the roles defined earlier in this chapter — to do precisely the identity-matching your phone's gallery performs today. SimCLR's contribution, five years later, was to reframe the same pull-together-push-apart logic as a batch-wide softmax classification problem and show it scales cleanly to general-purpose image representations, not just faces.

The toy example above used two images and four views; real systems use batches of thousands of images and embeddings with hundreds of dimensions, but the loss is the exact same formula. SimCLR (Chen, Kornblith, Norouzi, and Hinton, Google Research, 2020) showed that this framework, applied to a standard convolutional network with no other architectural tricks, could learn image representations good enough that a single linear classifier trained on top of the frozen embeddings — no fine-tuning of the encoder at all — reached 76.5% top-1 accuracy on ImageNet, matching a fully supervised ResNet-50 trained end-to-end with every label available. This train-a-linear-layer-on-frozen-features test is called linear probing, and it is the standard way the field checks whether a self-supervised representation is actually good: if classes are linearly separable in the embedding space, the encoder has already done the hard work of untangling them.

SimCLR's paper also isolated a detail worth remembering: which augmentations get composed matters enormously. Random cropping alone lets the network cheat, because two crops of the same photo usually share similar color statistics, so the encoder can match pairs by color histogram instead of by content. Only once random cropping was combined with random color distortion did the representations become meaningfully better, because color-matching stopped being a reliable shortcut and the network was forced to fall back on shape and structure — closer to how you would recognize Kabir in a dim room as readily as in daylight. A second detail is architectural: SimCLR passes the encoder's output through a small additional network, a projection head, before computing the contrastive loss, then discards the projection head and keeps only the encoder's own output for downstream use, since that turned out to be the more broadly useful representation of the two.

One practical problem with the NT-Xent formula is that its pool of negatives is exactly the rest of the current batch, and more negatives generally make the task harder and the representation better — SimCLR's results kept improving as batch sizes grew into the thousands, which is expensive. MoCo (He, Fan, Wu, Xie, and Girshick, Facebook AI Research, 2020) solved the same problem differently: instead of relying on a huge batch, it maintains a running queue of negative embeddings computed by a second copy of the encoder, one whose weights are updated slowly, as a moving average of the main encoder's weights, rather than by gradient descent directly. This momentum encoder keeps the queued negatives reasonably consistent with the current encoder even though most of them were computed many training steps earlier, giving MoCo access to tens of thousands of negatives without needing a correspondingly enormous batch.

CLIP (Radford et al., OpenAI, 2021) generalizes the idea in a different direction: instead of two augmented views of an image, the positive pair is an image and its own caption, drawn from about 400 million image-text pairs collected from the public internet. Two separate encoders, one for images and one for text, are trained jointly so that an image's embedding lands close to its caption's embedding and far from unrelated captions, using precisely the same batch-softmax structure as NT-Xent, just across two modalities instead of two augmentations of one. The result is a model that was never trained on a single hand-labelled ImageNet image, yet matches the zero-shot classification accuracy of a supervised ResNet-50 on that exact benchmark — because "knowing what an image is close to in meaning" turned out to transfer directly into "knowing what class it belongs to," once the classes themselves are described in words rather than looked up in a fixed label list.

All of these systems share the one idea this chapter has been building toward: manufacture a positive pair using some correspondence that already exists for free in the data — two crops of one photo, an image and its caption, one video frame and the next — and let a large pool of negatives do the rest. Contrastive learning is not really a technique for images specifically; it is a way to turn any naturally occurring pairing into a supervisory signal.

Why This Matters for Indian AI

Labelled data is the most expensive ingredient in machine learning, and the expense is not spread evenly across languages or domains. A dataset like ImageNet exists in globally sourced, English-labelled form; a comparably large, carefully labelled dataset for diabetic retinopathy scans from Indian hospitals, or spoken commands in a language like Bhojpuri or Marathi, or crop-disease photographs from Indian farms, usually does not exist at that scale, because someone has to pay skilled annotators to build it from nothing. What India's AI-heavy sectors — agritech, healthtech, fintech, and speech and vision systems for Indian languages — tend to have in abundance instead is unlabelled data: millions of crop photographs taken on farmers' phones, hours of call-centre audio, satellite imagery of Indian farmland.

Contrastive pretraining is precisely the technique that turns that abundance into an advantage instead of a bottleneck. An encoder can be pretrained contrastively on a large pool of unlabelled crop images — no human ever says which photo shows a diseased leaf — and only afterward fine-tuned, or simply linear-probed, on a much smaller labelled set that agronomists have checked by hand. Because the encoder has already learned to organize its embedding space around real visual structure, a few thousand labelled examples, sometimes even a few hundred, are enough to reach strong accuracy where training from scratch would have needed many times more. The same pattern — pretrain on cheap unlabelled data, adapt on a small labelled set — underlies most of the speech and vision systems now being built for India's twenty-two scheduled languages and its long tail of regional dialects, where large hand-labelled corpora may never be economically built, but raw audio and images are being generated constantly.

Back to the Gallery

Return to the phone gallery this chapter opened with. The face-embedding model behind that grouping feature was very likely never told, during its own training, whose face was whose — it was shown enormous numbers of face crops with only one piece of free information available: which crops came from the same photo, or the same short video, as which others. Pulling those together and pushing everything else apart, over enough data, was sufficient to produce an embedding space where distance means identity. Once that space exists, sorting your particular photos into "Grandmother" and "not Grandmother" is no longer a machine learning problem at all — it is ordinary clustering on a handful of points, something a much simpler algorithm finishes in milliseconds.

That is the real generality of contrastive learning, and the reason it sits at the center of modern self-supervised learning rather than at its edge. It does not need to know in advance what a face, a diseased leaf, or a fraudulent transaction looks like. It only needs a rule for manufacturing a positive pair — two crops, an image and its caption, one moment of a sequence and the next — and a large enough pool of negatives to make matching them hard enough that the encoder is forced to learn something real. Labels tell a model what humans already know. Contrastive learning asks the data to tell the model what is true about itself, and lets that be enough.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where contrastive learning: learning representations is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting contrastive learning: learning representations to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind contrastive learning: learning representations, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.