CLIP: Vision-Language Pre-training

Open Google Photos on your phone, type "chai stall in the rain," and photos you took two years ago in Manali surface in under a second — even though nobody ever typed the words "chai," "stall," or "rain" next to those files. No human tagged them. No folder was named for them. The search bar accepted a sentence; the result was a set of images. Somewhere between your query and the gallery, a system had to decide, for tens of thousands of untagged photographs, which ones a sentence in English was "about." That is not a search problem in the classical database sense — there is no column called caption to run WHERE caption LIKE '%chai stall%' against. It is a problem of putting pictures and sentences into the same measurable space, so that "close" can mean the same thing whether the two things being compared are pixels or words. CLIP — Contrastive Language-Image Pre-training, introduced by OpenAI in 2021 — is the model that made this workable at internet scale, and it is worth understanding exactly how, because the trick it uses (a shared embedding space trained by contrast rather than by labels) now sits underneath image search, the text-conditioning stage of diffusion models like Stable Diffusion, and automatic image-caption-quality scoring.

Two towers, one space

CLIP is built from two separate neural networks that never talk to each other during a forward pass, only during training. One is an image encoder — the original paper used both ResNets and Vision Transformers (ViT-B/32, ViT-B/16, ViT-L/14) — that takes a fixed-resolution image and outputs a single vector summarizing it. The other is a text encoder, a standard Transformer operating on byte-pair-encoded tokens, that takes a caption and outputs a single vector summarizing it. Left alone, these two vectors would live in unrelated spaces: nothing forces "a photograph of an orange cat" (text) to point in a similar direction to a photograph of an orange cat (image), because the two networks have completely different architectures, different input types, and no shared parameters.

CLIP fixes this with two small linear projection layers, one bolted onto each encoder's output, that map both vectors into a common dimensionality — say 512 numbers regardless of whether the input was pixels or tokens. After projection, both vectors are divided by their own Euclidean norm, so every embedding — image or text — ends up as a point on the surface of the same unit hypersphere. This normalization step matters more than it looks: once every vector has length exactly 1, the dot product between any two of them is no longer influenced by how "confident" or "large-magnitude" one encoder's output happens to be — it reduces to the cosine of the angle between them, a quantity bounded in [-1, 1] that has an identical meaning no matter which two vectors you pick. Without this step, an image encoder that happens to produce larger-magnitude vectors than the text encoder would systematically win every similarity comparison regardless of actual semantic closeness, and the training signal described next would be measuring encoder calibration instead of meaning.

The contrastive objective — training without a single label

Here is the central design decision that makes CLIP different from a classifier. A classifier trained on ImageNet needs a human to have assigned each of 1.28 million images one of 1,000 fixed category names. CLIP was trained on 400 million (image, caption) pairs scraped from public internet sources — alt-text, captions, surrounding page text — where the "label" for an image is whatever unstructured sentence happened to sit next to it. There is no fixed vocabulary of classes anywhere in this data. The question CLIP's training loop has to answer is not "which of 1,000 classes is this," but something much more general: "out of the captions in front of me right now, which one actually goes with this image?"

Concretely, take a mini-batch of N (image, caption) pairs. Run all N images through the image encoder and all N captions through the text encoder, producing N normalized image vectors I₁…Iₙ and N normalized text vectors T₁…Tₙ. Compute every pairwise cosine similarity between an image and a text — all N×N of them — and arrange them in a similarity matrix. Because the batch was assembled from N genuinely matched pairs, the diagonal entries (Iᵢ against Tᵢ) are the correct matches; every one of the N²−N off-diagonal entries is an incorrect pairing that the model has never been told is wrong except by omission — image 3's caption is never explicitly labeled "not about" image 7, it is simply absent from image 7's row. The model learns entirely from this pattern of "this one goes together, these others in the same batch don't," which is exactly what the word contrastive means here: no external labels, only relative comparison within a batch.

Before turning the similarity matrix into a loss, CLIP scales every entry by a learned temperature. Rather than fixing this scale, the paper parameterizes it as a single learnable scalar t, with the multiplier applied to the logits equal to exp(t), initialized so that the effective temperature starts at 0.07 (a multiplier of about 14.3) and clipped during training so the multiplier never exceeds 100 — otherwise the softmax below would saturate so hard the gradients would vanish. This one learned number turns out to matter a great deal: too low a scale and the model can't express confident distinctions; too high and training becomes unstable.

The loss itself is applied twice, in two directions, and averaged. Read each row of the scaled similarity matrix as logits for "given this image, which of the N captions is correct" — apply softmax across the row and take cross-entropy against the true diagonal index. That is the image→text loss. Then read each column the same way — "given this caption, which of the N images is correct" — that is the text→image loss. The two are averaged into the final batch loss. This is the InfoNCE loss shared with contrastive self-supervised methods like SimCLR, applied here across two different modalities instead of two augmented views of one image.

Reading the training loop as a diagram

Worked example — computing the loss by hand

Numbers make this concrete. Take a tiny batch of N = 3 image–caption pairs, with embeddings already projected and normalized to 2 dimensions (real CLIP uses 512+, but 2D lets every dot product be checked by hand and plotted as an angle). Suppose the image encoder places the three images at angles 0°, 90°, and 180° on the unit circle:

I1 = (1.000,  0.000)   # 0 degrees
I2 = (0.000,  1.000)   # 90 degrees
I3 = (-1.000, 0.000)   # 180 degrees

and the text encoder places the three matching captions nearby but not identically — 20°, 80°, and 160° — reflecting that a caption and its image are related but never encode to the exact same point:

T1 = (0.940, 0.342)    # 20 degrees
T2 = (0.174, 0.985)    # 80 degrees
T3 = (-0.940, 0.342)   # 160 degrees

Because every vector already has unit length, each dot product Iᵢ · Tⱼ is directly the cosine similarity — no separate normalization step needed at this point. Computing all nine gives the similarity matrix:

          T1       T2       T3
I1     0.9397   0.1736  -0.9397
I2     0.3420   0.9848   0.3420
I3    -0.9397  -0.1736   0.9397

The diagonal (I1–T1, I2–T2, I3–T3) is highest in its row and its column, as it should be — those are the true pairs, 20° away from each other, versus 70–160° for the mismatches. Now apply a temperature scale of 10 (a round number for this example; real CLIP's learned scale ranges from about 14.3 up to the cap of 100) by multiplying every entry:

          T1       T2       T3
I1      9.397    1.736   -9.397
I2      3.420    9.848    3.420
I3     -9.397   -1.736    9.397

Softmax each row (image→text direction). For row I1, the exponentials are e^9.397 ≈ 12054, e^1.736 ≈ 5.67, e^-9.397 ≈ 0.0000829; dividing each by their sum (≈ 12060) gives probabilities (0.99953, 0.00047, ~0). The cross-entropy loss for this row against the correct index (T1, position 1) is −ln(0.99953) ≈ 0.00047 — the model is essentially certain, and it is right to be. Running the same softmax-then-cross-entropy on all three rows and averaging gives a mean image→text loss of 0.001237. Doing the identical operation down each column (text→image direction) gives losses of 0.002534, 0.000309, and 0.002534, averaging to 0.001792. CLIP's final batch loss is the mean of the two directions:

total_loss = 0.5 * (0.001237 + 0.001792) = 0.001515

This number was computed independently with NumPy to confirm the hand arithmetic:

import numpy as np

I = np.array([[1.0, 0.0], [0.0, 1.0], [-1.0, 0.0]])
T = np.array([[0.9397, 0.3420], [0.1736, 0.9848], [-0.9397, 0.3420]])

S = I @ T.T                     # cosine similarities (already normalized)
logits = S * 10.0                # temperature scale

def softmax(x, axis):
    e = np.exp(x - x.max(axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

row_loss = -np.log(softmax(logits, axis=1)[np.arange(3), np.arange(3)])
col_loss = -np.log(softmax(logits, axis=0)[np.arange(3), np.arange(3)])
total = 0.5 * (row_loss.mean() + col_loss.mean())
print(total)   # 0.0015148828034117...

The number itself is less important than what it demonstrates: a loss near zero means the diagonal already dominates every row and column, which only happens when true pairs land close together in embedding space and everything else lands far away. It is worth holding that number against a second one. For a freshly initialized network, before any training, the embeddings are essentially random and unrelated to content, so every entry of the similarity matrix looks like noise of similar magnitude — the softmax over each row (or column) is then close to uniform, 1/N everywhere, and the cross-entropy loss reduces to −ln(1/N) = ln(N). For N = 3, that baseline is ln(3) ≈ 1.0986 — roughly 725 times larger than the trained loss of 0.0015 computed above. Watching the batch loss fall from near ln(N) toward near zero over training is literally watching the diagonal of the similarity matrix pull apart from the rest of the grid.

Zero-shot classification: the payoff

Training produces two encoders that share a space; the practical use of that space is zero-shot classification, and it is worth seeing that it is the exact same computation as one row of the matrix above, just relabeled. Suppose, instead of a batch of true captions, you hand CLIP a fixed set of class-name prompts — "a photo of a {label}" filled in with candidate class names — and one query image. Embed all the prompts with the text encoder, embed the image with the image encoder, and compute the row of cosine similarities between that one image and every prompt. That row is precisely what row I1 was in the worked example above: I1 against T1, T2, T3 gave probabilities (0.9995, 0.00047, ~0) after the softmax. Relabel T1, T2, T3 as three candidate classes — say "auto-rickshaw," "bicycle," "bus" — and that same computation is a full zero-shot classification: the image is 99.95% "auto-rickshaw," essentially 0% the other two, with no gradient update, no fine-tuning, and no example of an auto-rickshaw ever having been labeled as such during CLIP's own training — the model only ever needed to have encountered the phrase "auto-rickshaw" used near relevant images somewhere in its 400-million-pair training set.

In real code this looks like the snippet below. The output is a probability vector of the same shape and character as the row you just traced by hand — one entry per class prompt, softmaxed, summing to 1 — so no new arithmetic is required to know what it represents:

import torch, clip
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

image = preprocess(Image.open("street.jpg")).unsqueeze(0).to(device)
labels = ["auto-rickshaw", "bicycle", "bus"]
text = clip.tokenize([f"a photo of a {c}" for c in labels]).to(device)

with torch.no_grad():
    img_f = model.encode_image(image)
    txt_f = model.encode_text(text)
    img_f /= img_f.norm(dim=-1, keepdim=True)
    txt_f /= txt_f.norm(dim=-1, keepdim=True)
    probs = (100.0 * img_f @ txt_f.T).softmax(dim=-1)

Two details connect this back to the loss derivation. First, the 100.0 multiplier is exactly the temperature scale from training — CLIP's public checkpoints were trained with the scale capped at 100, so inference code reuses that same number rather than the softer 10 used in the hand example, which makes the model's predictions sharper and more confident. Second, the class list labels can be swapped for any set of English phrases at inference time with zero retraining — this is the entire mechanism behind the headline result of the original paper: zero-shot CLIP, evaluated on ImageNet's 1,000 classes it was never fine-tuned on, matched the accuracy of the original ResNet-50 trained with full supervision on all 1.28 million labeled ImageNet training images.

The misconception worth correcting

The single most common misunderstanding students bring to CLIP is confusing it with an image-captioning model — expecting it to look at a photo and generate the sentence "a red auto-rickshaw waiting at a traffic signal." CLIP does no such thing and cannot: it has no decoder, no mechanism for producing novel text token-by-token, nothing analogous to the generation loop in a language model or a sequence-to-sequence captioner. CLIP only ever does one operation — given an image and a set of candidate texts you supply, score how well each candidate matches. If none of the candidate texts you hand it describes the image correctly, CLIP has no way to tell you what would; it will simply return whichever of your bad candidates scores least badly. Every "zero-shot" capability of CLIP is really "zero-shot discrimination among options you provide," not zero-shot generation — the class list in the code above has to already contain the right answer as one of its entries, or the model has nothing correct to point to. This distinction is exactly why CLIP's text encoder later became the standard way to condition generative image models like Stable Diffusion — CLIP supplies the semantic target for a separate generative network to aim at, but the actual pixel generation is done by that other network, not by CLIP.

Active recall

Attempt each question before reading its answer.

1. Why does CLIP L2-normalize both embeddings before taking their dot product, instead of using the raw encoder outputs directly?
2. In a training batch of N = 256 image–caption pairs, how many positive pairs and how many negative pairs does the similarity matrix contain?
3. Two already-normalized embeddings have a dot product of 0.6. With a learned temperature scale of 20 (i.e. logit = 20 × cosine similarity), what raw logit is fed into the softmax?
4. A completely untrained CLIP model (random encoder weights) is evaluated on a batch of N = 5 pairs. What contrastive loss value should you expect, and why?
5. Why would CLIP's zero-shot classifier likely fail on a proprietary internal part code like "XJ-4471B" as a class label, even if you had real photos of that part?
6. Why does CLIP average an image→text loss and a text→image loss, rather than training on just one direction?

Answers

1. Normalizing pins every embedding to the surface of the same unit hypersphere, so the dot product between any two vectors reduces exactly to the cosine of the angle between them — a quantity bounded in [-1, 1] with a fixed, comparable meaning. Without normalization, whichever encoder happens to produce larger-magnitude outputs would dominate every similarity score regardless of actual semantic alignment, and the learned temperature scale (which assumes bounded logit inputs) would have no stable reference point to calibrate against.
2. The diagonal of an N×N matrix has N entries, so there are 256 positive pairs. The remaining entries are N² − N = 256² − 256 = 65536 − 256 = 65,280 negative pairs — every batch this size supplies over 65,000 free negative examples from just 256 labeled pairs, which is why the contrastive setup is so data-efficient compared to per-image manual negative mining.
3. logit = 20 × 0.6 = 12.
4. Roughly ln(5) ≈ 1.609. With random, untrained encoders, embeddings carry no reliable content signal, so all N similarities in a row (or column) are approximately equal noise; the softmax over them is close to uniform at 1/N each, and cross-entropy against the true index becomes −ln(1/N) = ln(N) = ln(5).
5. The text encoder's embedding for a prompt is only meaningful to the extent the phrase (or its subword pieces) appeared in a semantically grounded context somewhere in the 400-million-pair training data. "XJ-4471B" is very unlikely to have appeared anywhere near a relevant photo online, so its embedding is driven by arbitrary subword fragments rather than genuine visual meaning — the resulting similarity scores against real images of the part would be close to noise, not a reliable classification signal. Zero-shot CLIP works for classes describable in the natural language it has actually seen, not for opaque internal codes.
6. Optimizing only the image→text direction teaches each image to prefer its own caption among the batch's captions, but says nothing about the reverse — a caption's text embedding could still turn into a generic "attractor" that scores well against many different images without being uniquely discriminative when that caption is the fixed reference. Averaging in the text→image direction forces every caption to also prefer its own image among the batch's images, which regularizes both encoders jointly and is what gives the objective its InfoNCE-style mutual-information guarantee rather than a one-sided approximation of it.

Think About It

Think about this: How would you explain clip: vision-language pre-training to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where clip: vision-language pre-training is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting clip: vision-language pre-training to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind clip: vision-language pre-training, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.