Multimodal AI: GPT-4V, CLIP, and Beyond

Open Myntra or Flipkart, tap the camera icon, and photograph a saree you saw someone wearing on the metro. Within a second the app returns a grid of visually similar sarees from a catalog of tens of millions of items — none of which were ever manually tagged "green Banarasi silk with gold zari border." No human wrote that description into a database, and no classifier was trained with "saree" as an output category. The system instead converts your photo into a point in a high-dimensional space, converts every catalog description into a point in the same space, and returns whichever catalog points sit closest to your photo's point. That is CLIP-style retrieval — the mechanism behind Radford et al.'s 2021 paper "Learning Transferable Visual Models From Natural Language Supervision" (ICML 2021), trained by OpenAI on roughly 400 million image-text pairs scraped from the public web, which produced, almost as a side effect, one of the most reused components in modern AI.

Now try something CLIP cannot do at all: photograph the same saree and ask "what's this embroidery style called, and does it need dry-cleaning?" A system like GPT-4V will look at the image and write a sentence answering both questions — something no similarity search can produce, because a similarity search has no mechanism for generating a new sentence. It can only rank things that already exist against each other. GPT-4V-style models solve a different problem with a different architecture, trained on a different objective, and inheriting different failure modes from CLIP. This chapter is about that difference: what CLIP's contrastive objective actually computes, worked out numerically end to end, and how a generative vision-language model is wired so differently underneath that treating both as "the same kind of multimodal AI" is the single biggest misconception students carry into this topic.

Two very different jobs wearing the same "sees images" label

Before touching either architecture, fix the vocabulary. An embedding is a vector — a list of numbers — that a neural network produces to represent something (a word, an image, a sentence) such that geometric distance in that vector space corresponds to semantic similarity: two sentences with similar meaning should map to nearby vectors, two unrelated ones to distant vectors. Every model discussed in this chapter is in the business of producing and using embeddings, but the two families use them for entirely different purposes.

CLIP is a dual-encoder model: one neural network (a Vision Transformer, or a ResNet in CLIP's original smaller variants) turns an image into an embedding vector; a completely separate neural network (a text Transformer) turns a caption into an embedding vector of the same dimensionality. The two networks never see each other's inputs during a forward pass — the image encoder has no idea what text exists, and vice versa. The only place they interact is a single dot product computed after both embeddings already exist: cosine similarity. CLIP has no mechanism to generate a sentence, because it has no decoder — no component that predicts the next word of anything.

A GPT-4V-style model is fused and generative. An image is chopped into patches, each patch is embedded by a vision encoder (frequently a CLIP vision tower, reused precisely because its contrastive pretraining already produces useful visual features), and those patch embeddings are projected into the same numerical space that the language model's token embeddings live in. The result — "soft tokens" that are not any actual word but sit in the same coordinate system as words — is spliced directly into the sequence the decoder reads, alongside the actual text prompt. The decoder then does exactly what a text-only LLM does: predicts the next token, one at a time, conditioning on everything before it, including the image patches now embedded in that sequence. There is no dot product, no similarity score. Every output token is a full generative decode step through billions of parameters.

This distinction is not cosmetic. It determines what each model can be asked to do, how expensive each query is, and how each one fails.

Inside CLIP: the contrastive objective, precisely

CLIP trains on batches of N image-text pairs scraped from the web (OpenAI used batches of 32,768 pairs). Within one training batch, every image is paired with exactly one caption that actually describes it — call these the N positive pairs — and every image is also compared against every other caption in the batch, none of which describe it. That produces an N × N grid of image-text combinations: N correct pairings on the diagonal, and N² − N incorrect pairings everywhere else.

The training step: encode all N images and all N texts, L2-normalize every embedding to unit length (so the dot product between any two embeddings equals their cosine similarity, bounded between −1 and 1), then compute the full N × N similarity matrix. Divide every entry by a learned temperature τ to get logits, treat each row of that matrix as a classification problem — "which of these N texts is the right one for this image?" — and compute the cross-entropy loss against the diagonal label. Then do the same thing column-wise — "which of these N images is the right one for this text?" Average the two directions. That symmetric average is the CLIP loss, exactly the InfoNCE (Information Noise-Contrastive Estimation) objective used in earlier single-modality representation-learning work, now applied across two modalities at once.

The two directions matter for different downstream uses. The image→text direction is what makes zero-shot image classification work: CLIP never sees a fixed label set during training, so to classify a photo into "cat" vs "dog" vs "bus" at inference, you embed the strings "a photo of a cat," "a photo of a dog," "a photo of a bus" as text, embed the query image, and pick whichever text has the highest cosine similarity to the image. No classification head, no fine-tuning, no gradient updates at inference — the entire "classifier" is a handful of dot products against embeddings generated on the fly. The saree example from the opening of this chapter is actually another instance of the image→text direction — the same operation as zero-shot classification, just with millions of catalog-description candidates instead of three class names, since the photo is the single query and the catalog descriptions are the ranked candidates. The text→image direction instead powers the mirror-image product: typing a natural-language query ("green Banarasi silk saree with gold zari border") into a search bar and getting back matching photos — one text anchor, many image candidates.

Worked example: computing the CLIP batch loss by hand

Take a toy batch of N = 3 image-text pairs. Suppose the already-normalized vision and text encoders have produced these cosine similarities — a 3×3 matrix where entry (i, j) is sim(image i, text j):

          T1     T2     T3
  I1     0.90   0.20   0.10
  I2     0.30   0.80   0.40
  I3     0.10   0.30   0.70

The diagonal (I1–T1, I2–T2, I3–T3) holds the true pairs, each around 0.7–0.9 — not 1.0, because even a well-trained encoder rarely drives matched embeddings to perfect alignment (the reason why is covered later in this chapter). Fix the temperature at τ = 0.5. Real CLIP learns τ as a parameter, initialized so 1/τ ≈ 14.3, and clips it at a maximum of 100 to keep training stable; this worked example uses a gentler 2× scale so the arithmetic stays traceable by hand. Logits are similarity divided by τ, i.e. multiplied by 2 here:

          T1     T2     T3
  I1     1.80   0.40   0.20
  I2     0.60   1.60   0.80
  I3     0.20   0.60   1.40

Take row I1: logits [1.80, 0.40, 0.20]. Exponentiate: e^1.80 = 6.0496, e^0.40 = 1.4918, e^0.20 = 1.2214, summing to 8.7629. The softmax probability assigned to the correct answer, T1, is 6.0496 / 8.7629 = 0.6903. The per-example cross-entropy loss is −ln(0.6903) = 0.3705 — low, because the model is already fairly confident in the right answer.

Repeating this for every row (image→text) and every column (text→image, using the same matrix transposed) gives six per-example losses:

image->text:  I1 = 0.3705   I2 = 0.5973   I3 = 0.5599   mean = 0.5092
text->image:  T1 = 0.4075   T2 = 0.5123   T3 = 0.6152   mean = 0.5117

CLIP loss = (0.5092 + 0.5117) / 2 = 0.5105

Verify this in code rather than trusting the hand arithmetic:

import numpy as np

sim = np.array([
    [0.9, 0.2, 0.1],
    [0.3, 0.8, 0.4],
    [0.1, 0.3, 0.7],
])

tau = 0.5
logits = sim / tau

def cross_entropy_rows(logits):
    exp = np.exp(logits)
    probs = exp / exp.sum(axis=1, keepdims=True)
    correct = np.diag(probs)
    return -np.log(correct)

loss_i2t = cross_entropy_rows(logits).mean()           # image -> text
loss_t2i = cross_entropy_rows(logits.T).mean()          # text -> image
clip_loss = (loss_i2t + loss_t2i) / 2

print(round(loss_i2t, 3), round(loss_t2i, 3), round(clip_loss, 3))

This prints exactly 0.509 0.512 0.51, matching the hand computation. Notice that cross_entropy_rows(logits.T) is doing real work, not a formality: transposing the matrix turns each column of the original into a row, so the same row-softmax function computes the text→image direction by reusing the image→text logic on the transpose. That single transpose is why the CLIP loss is called symmetric — one function, applied twice, from both directions of the same matrix.

Beyond CLIP: how a generative vision-language model actually reads an image

OpenAI has never published GPT-4V's internal architecture — the September 2023 "GPT-4V(ision) System Card" documents its behavior and safety evaluation in detail but not its weights or wiring. What is documented in the open literature is the general architecture family GPT-4V belongs to, demonstrated concretely by systems like LLaVA (Liu, Li, Wu, and Lee, "Visual Instruction Tuning," NeurIPS 2023) and Flamingo (Alayrac et al., DeepMind, "Flamingo: a Visual Language Model for Few-Shot Learning," NeurIPS 2022). Both make the same core move: take a vision encoder — often literally CLIP's ViT, reused because its contrastive pretraining already produces features that transfer well — and connect it to a decoder-only language model through a small trainable bridge.

LLaVA's bridge is almost embarrassingly simple: a single linear layer (later versions use a small MLP) that projects each patch embedding from the vision encoder's dimensionality into the language model's token embedding dimensionality. A 336×336 image split into 14×14-pixel patches produces 576 patch embeddings; the linear layer turns each into a vector indistinguishable, dimensionally, from a word embedding. Those 576 "soft tokens" get concatenated with the tokenized text prompt and fed straight into the language model, which was never told which positions are real words and which are projected image patches — it just runs causal self-attention over the whole sequence. Flamingo's bridge is more elaborate: a Perceiver Resampler compresses a variable number of patch embeddings down to a fixed 64 visual tokens, which then interact with the (frozen) language model through newly inserted gated cross-attention layers rather than being spliced directly into the input sequence. Both approaches solve the same problem — getting continuous visual information into a system built for discrete text tokens — with different tradeoffs between architectural simplicity and how much of the pretrained language model has to be disturbed.

Whichever bridge is used, training is generative: given an image and a partial caption or answer, predict the next text token, and back-propagate ordinary language-modeling cross-entropy loss through the projection layer and, depending on the recipe, into the vision encoder itself. This is a completely different loss from CLIP's InfoNCE — there is no batch of negatives, no similarity matrix, no symmetric averaging. It is next-token prediction, exactly like training a text-only LLM, except that some of the "tokens" in the context happen to have come from a projected image patch instead of a vocabulary lookup.

The production consequence a systems-minded student should be able to reason through: images are expensive in token terms, and that cost is documented, not hidden. OpenAI's vision API guide specifies that a "low-detail" image costs a flat 85 tokens regardless of resolution, while "high-detail" mode resizes the image to fit within 2048×2048, scales the shorter side to 768px, then tiles the result into 512×512 blocks, charging 170 tokens per tile plus the 85-token base. A 1024×1024 photo at high detail tiles into four 512×512 blocks: 85 + 4 × 170 = 765 tokens — before the model has generated a single word of its answer. A visual-search product built on CLIP pays a one-time embedding cost per catalog image and then runs cheap dot products forever; a product built on GPT-4V-style VQA pays this token cost on every single query. That is precisely why production systems route "does this image roughly match" questions to CLIP-style retrieval and reserve GPT-4V-style generation for questions that genuinely require producing new language.

Generation also opens a failure mode retrieval does not have: object hallucination. Li et al.'s "Evaluating Object Hallucination in Large Vision-Language Models" (EMNLP 2023) introduced the POPE benchmark specifically to measure how often a vision-language model confidently names an object, attribute, or relationship that is not actually present in an image — a spoon on a table that has no spoon, a second person in a photo that has only one. This happens because next-token prediction is only loosely grounded in the visual input; the language model's strong prior over "what usually appears in a kitchen photo" can outvote the actual pixels, especially for objects that co-occur frequently in training data. CLIP cannot fail this way in principle, because CLIP never asserts that anything is present — it only ever reports a similarity score between an image and a piece of text supplied to it. It can rank a wrong caption highest, but it cannot invent a caption from nothing.

One more result worth knowing precisely, because the worked example above already demonstrated it without naming it: even a well-trained CLIP model does not push matched image-text pairs to cosine similarity 1.0, or anywhere near it, and this is not merely undertraining. Liang et al.'s "Mind the Gap: Understanding the Modality Gap in Multimodal Contrastive Representation Learning" (NeurIPS 2022, Stanford) showed that image embeddings and text embeddings from CLIP occupy two distinct, non-overlapping cones of the shared embedding space, separated by a persistent gap that traces back to the geometry of the encoders' random initialization interacting with the contrastive objective — not to insufficient training data. The diagonal similarities in the worked-example matrix (0.90, 0.80, 0.70) reflect exactly this pattern: those are strong matches by CLIP's own standards, and none of them approach 1.0. The gap's size is itself a measurable, manipulable property of a trained model, with documented downstream effects on zero-shot classification and retrieval accuracy — not an artifact of an unlucky training run.

The misconception to correct directly

Students almost universally arrive at this topic treating CLIP and GPT-4V as two versions of "the AI that understands pictures," differing only in how good they are. They are not comparable on a single scale — they solve different problems. CLIP has no decoder and cannot generate a sentence about anything; ask it "what is unusual about this image" and there is no operation to perform, because CLIP was never trained to produce open-ended text, only to score how well a given piece of text matches a given image. GPT-4V-style models have no built-in similarity-scoring interface at all; asking one to "return a similarity score between this image and this caption" only works if you prompt it to generate a number in words, which it may do inconsistently and at far higher cost, whereas CLIP returns an exact, cheap, deterministic float from a single dot product. And critically, these are not competitors: CLIP is very often a literal component sitting inside a GPT-4V-style system, since its contrastively trained vision encoder produces exactly the kind of general-purpose visual features that a projection layer can hand off to a language model. The relationship is closer to "engine" and "car" than "two brands of car."

Active recall

Attempt every question before reading the answers below.

1. Why can CLIP perform "zero-shot" image classification on a set of class names it has never seen labeled during training, with no classifier head and no fine-tuning?
2. The worked example computed the CLIP loss two ways — once as cross_entropy_rows(logits) and once as cross_entropy_rows(logits.T). What real modeling distinction does taking the transpose encode, and why would training on the image→text direction alone produce a worse text embedding space for the zero-shot classification described in question 1?
3. Starting from the worked example's similarity matrix, suppose the I2–T3 entry rises from 0.40 to 0.75 (image 2's true caption becomes easy to confuse with image 3's caption), while every other entry stays fixed. Which of the six per-example losses change, and which stay exactly the same? Reason from the structure of the softmax, not just by re-running the numbers.
4. The worked example used τ = 0.5. If τ were instead lowered to 0.1 on the same, unmodified similarity matrix, what happens to the total CLIP loss — and is a lower τ always a safe choice during training?
5. A LLaVA-style model projects each image patch embedding with a single linear layer before handing it to the language model. Why must that projection layer's output width exactly equal the language model's token embedding width, and what happens mechanically if it doesn't?
6. A colleague says: "GPT-4V is just CLIP with better prompting." Give the one-sentence architectural fact that makes this false.

Answers

1. CLIP's contrastive loss trains it to place any image near the embedding of text that truly describes it, and far from text that doesn't, inside one shared metric space. "Classifying" an image is then nothing but nearest-neighbour search in that pretrained space: embed each candidate class name as a sentence, embed the image, and pick the class whose text embedding has the highest cosine similarity. Because this operation is just a dot product against whatever text you supply at inference time, it works for any class name expressible in language — including ones that never appeared during training — with no dedicated classification head and no gradient update.

2. The transpose encodes the text→image retrieval direction: "given this caption, which image in the batch does it actually describe?" — the mirror image of the image→text question. Training only on the image→text direction optimizes each image's row-softmax to prefer its own caption over the other captions in the batch, but nothing in that objective stops two different images from ending up needing embeddings close to the same popular caption, since no term ever normalizes across images for a fixed piece of text. Left unchecked, that lets the text embedding space collapse toward a smaller number of generic "attractor" points. Zero-shot classification depends on class-name text embeddings staying well separated from each other in exactly this way, so the missing text→image term directly damages the capability question 1 depends on — even though, on the surface, that capability only ever seems to use the image→text direction at inference time.

3. Row I2's softmax denominator increases (its logit for T3 rose), shifting probability mass away from the correct T2 and toward T3, so the image→text loss for I2 rises from 0.5973 to 0.8210. Column T3's softmax denominator increases for the same reason (I2's competing logit is now in that column too), shifting probability mass away from the correct I3, so the text→image loss for T3 rises from 0.6152 to 0.8781. Every other loss — I1, I3 (image→text) and T1, T2 (text→image) — is computed from a row or column that never contains the changed cell sim[1,2], so those four softmax inputs are numerically identical to before and their losses do not move at all. The overall CLIP loss rises from 0.5105 to 0.5916. A single similarity entry always touches exactly one row-softmax and one column-softmax — never more, never fewer.

4. Lowering τ from 0.5 to 0.1 multiplies every logit by 10 instead of 2, and because the diagonal already holds the largest similarity in every row and column of this matrix, sharpening the softmax pushes almost all remaining probability mass onto the already-correct answer: the total loss collapses from 0.5105 to roughly 0.0183. That is not universally safe, though — the same sharpening amplifies the effect of any wrong ranking exactly as aggressively as it rewards a right one. An off-diagonal entry only slightly above the true diagonal would, after a 10× scale-up, get turned into an overconfident wrong prediction with a correspondingly huge loss and gradient. That asymmetric risk is exactly why real CLIP caps its learned temperature so 1/τ never exceeds 100 — an unbounded sharpening can blow up training whenever the network is briefly wrong about a hard pair, even while it helps on the easy pairs it already gets right.

5. The decoder's self-attention layers apply the same weight matrices to every position in the sequence regardless of whether that position came from a text token embedding or a projected image patch, and matrix multiplication requires the two operands' inner dimensions to match. If the projection's output width didn't equal the model's hidden size, the image vectors could not be concatenated into the same sequence or multiplied against the same attention weight matrices as the text embeddings at all — the forward pass would fail with a shape-mismatch error before any attention was computed, not silently produce a degraded answer.

6. CLIP is a dual-encoder model trained with a contrastive similarity loss and has no decoder, so it cannot generate text under any prompt; GPT-4V-style models are fused, autoregressive decoders trained on next-token prediction that may use a CLIP-style encoder only as one internal component feeding the decoder, not as the whole system.

Think About It

Think about this: How would you explain multimodal ai: gpt-4v, clip, and beyond to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where multimodal ai: gpt-4v, clip, and beyond is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting multimodal ai: gpt-4v, clip, and beyond to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind multimodal ai: gpt-4v, clip, and beyond, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.