Myntra and Flipkart run visual search at production scale: a shopper photographs a kurta they liked in a shop window, or types "maroon silk saree with gold zari border," and the system must return matching catalogue items in under a second, out of tens of millions of product photos. Neither query type is special-cased. Both the photo and the text sentence are converted into vectors in the same geometric space, and the catalogue image whose vector sits closest to the query vector wins. That single design decision — force images and text into one shared coordinate system so that "close in space" means "close in meaning" — is called cross-modal alignment, and it is the actual training problem this chapter is about. The sibling chapter on vision-language models covers what these systems can do once built; this one covers how the training loop that builds them actually works: the loss function that pulls matching image-text pairs together and pushes non-matching pairs apart, the batch-size economics that make that loss tractable at scale, and the two-stage recipe (freeze, project, then instruct) that modern systems like LLaVA use to turn a pretrained vision encoder and a pretrained language model into one unified system without retraining either from scratch.
Why a shared embedding space needs a contrastive loss
Suppose you have a vision encoder (a Vision Transformer, ViT) that turns an image into a vector, and a text encoder (a Transformer) that turns a sentence into a vector. Trained independently — the ViT on ImageNet classification, the text encoder on a language-modelling objective — nothing forces the two vector spaces to line up. The image vector for "red saree" and the text vector for the string "red saree" could point in completely unrelated directions, because the two encoders never saw each other during training. Cosine similarity between them would be meaningless.
Alignment training fixes this by training the two encoders jointly against pairs of (image, caption) that are known to match, using a loss that has one very specific property: it does not just reward matching pairs for being similar, it simultaneously penalizes every non-matching pair in the same training batch for being similar. This second half is what makes it "contrastive" rather than merely "similar." Radford et al. (Learning Transferable Visual Models From Natural Language Supervision, ICML 2021 — the CLIP paper) formalized this using the InfoNCE objective, treating alignment as a batch-wise classification problem: given an image, correctly pick out its matching caption from among all the other captions in the batch, and vice versa.
The InfoNCE / CLIP objective, derived
Take a batch of N image-text pairs. Run every image through the vision encoder and L2-normalize the output to a unit vector; do the same for every caption through the text encoder. You now have image embeddings I₁...I_N and text embeddings T₁...T_N, all unit length. Because both are unit vectors, their dot product Iᵢ · Tⱼ is exactly the cosine similarity between image i and text j — ranging from -1 (opposite) to 1 (identical direction).
Compute the full N × N similarity matrix S, where S[i][j] = Iᵢ · Tⱼ. Divide every entry by a temperature τ to get logits, then treat row i of that logit matrix as a classification problem: "out of the N candidate captions, which one is the true caption for image i?" The answer is always index i (the diagonal), so this is ordinary softmax cross-entropy with the diagonal as ground truth. Do the same thing column-wise, treating each text embedding as needing to pick out its matching image. Average the row-wise (image→text) and column-wise (text→image) cross-entropy losses to get the final symmetric CLIP loss:
L = ½ [ CrossEntropy(S/τ, labels=diag, axis=rows)
+ CrossEntropy(S/τ, labels=diag, axis=cols) ]The two directions matter separately because they penalize different failure modes. Image→text failing means the image's vector is closer to some wrong caption than to its own; text→image failing means a caption's vector is closer to some wrong image. A model can fail one direction while passing the other, so both terms are needed.
Worked example: computing the loss by hand for a batch of three
Take N = 3 pairs with 2-dimensional embeddings (real CLIP uses 512 or 768 dimensions; 2D keeps the arithmetic checkable by hand while showing every mechanic exactly). All vectors below are already unit length.
Images: I1=(1.0, 0.0) I2=(0.0, 1.0) I3=(-1.0, 0.0)
Texts: T1=(0.8, 0.6) T2=(0.6, 0.8) T3=(-0.8,-0.6)Check unit length: 0.8² + 0.6² = 0.64 + 0.36 = 1.0 — confirmed for T1, T2, T3 by the same arithmetic. Each Tᵢ is meant to be the caption matching Iᵢ, but deliberately imperfect: T2 sits only about 0.28 radians (~16°) of "direction" away from I1's caption T1, making it a hard negative for the I1 row.
Compute the 3×3 similarity matrix via dot products (e.g. I1·T2 = 1.0×0.6 + 0.0×0.8 = 0.6):
T1 T2 T3
I1 [ 0.8 0.6 -0.8 ]
I2 [ 0.6 0.8 -0.6 ]
I3 [ -0.8 -0.6 0.8 ]The diagonal (0.8, 0.8, 0.8) holds the true pairs. Row 1 shows the hard negative clearly: T2 scores 0.6 against I1, well above T3's -0.8, so the model has real work to do separating T1 from T2 for this image.
Set τ = 0.1 (CLIP's learned temperature typically settles in this range) and divide: row 1 logits become [8, 6, -8]. Row-wise softmax cross-entropy against the diagonal label, computed exactly as ln(Σⱼ exp(logits_j − logit_correct)):
Row 1 (correct=T1): ln(1 + e^(6-8) + e^(-8-8)) = ln(1 + e^-2 + e^-16) = 0.12693
Row 2 (correct=T2): ln(1 + e^(6-8) + e^(-6-8)) = ln(1 + e^-2 + e^-14) = 0.12693
Row 3 (correct=T3): ln(1 + e^(-8-8) + e^(-6-8)) = ln(1 + e^-16 + e^-14) ≈ 0.00000094Mean image→text loss: (0.12693 + 0.12693 + 0.00000094) / 3 = 0.08462. Because S is symmetric here, the column-wise (text→image) losses come out identical, so the final symmetric loss is L ≈ 0.0846 nats. I verified this arithmetic in code rather than trusting hand calculation:
import numpy as np
I = np.array([[1.0, 0.0], [0.0, 1.0], [-1.0, 0.0]])
T = np.array([[0.8, 0.6], [0.6, 0.8], [-0.8, -0.6]])
S = I @ T.T
tau = 0.1
logits = S / tau
row_max = logits.max(axis=1, keepdims=True)
row_probs = np.exp(logits - row_max)
row_probs /= row_probs.sum(axis=1, keepdims=True)
i2t = -np.log(np.diag(row_probs))
col_max = logits.max(axis=0, keepdims=True)
col_probs = np.exp(logits - col_max)
col_probs /= col_probs.sum(axis=0, keepdims=True)
t2i = -np.log(np.diag(col_probs))
loss = (i2t.mean() + t2i.mean()) / 2
print(loss) # 0.084619265892613
Running this prints 0.084619265892613, matching the hand derivation. Notice what row 3's near-zero loss (0.00000094) versus row 1's 0.12693 is telling the optimizer: row 3 is already solved (its correct match beats both alternatives by a wide margin after scaling by τ), so its gradient contribution is tiny; row 1 is where the hard negative T2 lives, so almost all of the gradient signal in this batch comes from pushing I1 away from T2 and toward T1. This is the mechanism by which contrastive training concentrates learning on the pairs that are actually confusable — exactly the property that makes it effective for real catalogues where near-duplicate products (two very similar sarees) are the hard cases that matter.
Batch size is not a hyperparameter here — it is part of the loss
In ordinary supervised training, batch size mainly affects gradient noise and throughput. In contrastive alignment it changes what the loss function is: every other sample in the batch becomes a negative for every anchor. A batch of 3 gives each image only 2 negatives to discriminate against; a batch of 32,768 — the batch size CLIP was actually trained with (Radford et al., 2021) — gives each image 32,767 negatives per step. More negatives per step means a harder, more informative classification problem and a tighter embedding space, which is why CLIP-style training is reported to degrade sharply at small batch sizes and why production teams training their own catalogue-alignment encoders fight hard for large effective batch sizes.
This creates a specific GPU memory problem distinct from ordinary large-batch training: the N × N similarity matrix and the two encoders' activations for all N samples must be materialized simultaneously to compute the contrastive loss, so memory scales roughly quadratically in the number of negatives you want, not just linearly in batch size the way a classification loss would. Production systems handle this three ways: (1) gradient checkpointing inside each encoder to trade recomputation for activation memory, letting a bigger batch fit; (2) sharding the similarity-matrix computation across GPUs and all-gathering only the embeddings (not full activations) before computing the matrix, so each device holds one encoder's activations for its local shard but sees the global negative set; (3) a memory bank / momentum-encoder trick (used in MoCo-style contrastive learning) that reuses embeddings from recent past batches as extra negatives without recomputing them, decoupling negative count from GPU memory at the cost of slightly stale negatives. Understanding which constraint you are actually up against — activation memory versus negative-set size — determines which fix is worth applying.
Building a unified VLM: freeze, project, instruct
CLIP-style contrastive training produces two aligned encoders good at retrieval — "does this image match this text" — but it does not produce a system that can converse about an image, answer a question about it, or generate a paragraph describing it. For that, the vision encoder's output has to be fed into a decoder-only LLM as if it were more text tokens, and that requires a second, architecturally different training stage. LLaVA (Liu et al., "Visual Instruction Tuning," NeurIPS 2023) is the clearest published recipe for this and illustrates a design choice students consistently underestimate: almost nothing in the pipeline is trained from scratch, and most of it is frozen throughout.
The pipeline: a frozen, already-trained CLIP ViT-L/14 encoder converts an image into a grid of patch-level feature vectors (not one pooled vector — the spatial grid is kept, because the LLM needs fine-grained visual detail, not just a global gist). A single trainable linear layer, the projector, maps each patch vector from the vision encoder's dimensionality into the LLM's token-embedding dimensionality, so the visual features become indistinguishable in shape from a sequence of word-embedding vectors. Those projected visual tokens are concatenated with the tokenized text prompt and fed straight into the LLM, which is trained (or partly trained) with the same next-token cross-entropy loss used for ordinary language modelling — except now some of the "tokens" it is conditioning on are visual.
Training happens in two stages with different frozen/trainable splits: Stage 1 (feature alignment) freezes both the vision encoder and the LLM and trains only the projector, on roughly 595K filtered image-caption pairs, teaching the projector to place visual tokens somewhere the LLM's existing embedding space can already interpret — this is cheap, since the trainable parameter count is a single linear layer. Stage 2 (visual instruction tuning) keeps the vision encoder frozen but now unfreezes the LLM alongside the projector, training end-to-end on roughly 158K multimodal instruction-following examples so the model learns to actually follow visual questions and instructions, not merely describe images. BLIP-2 (Li et al., ICML 2023) solves the same connection problem differently, inserting a small trainable Querying Transformer (Q-Former) with 32 learnable query vectors between a frozen image encoder and a frozen LLM, trained first with a mixed contrastive/matching/generation objective and then to produce soft prompts the frozen LLM can consume — a heavier bridge than LLaVA's single linear layer, but one that compresses each image down to a fixed 32 tokens regardless of resolution.
Traced code: shapes through the LLaVA pipeline
The following traces exact tensor shapes for a single image feeding a Vicuna-7B-class LLM (hidden dimension 4096), using ViT-L/14 (patch size 14, hidden width 1024) on a 224×224 image, plus a 40-token text prompt:
import numpy as np
img_size, patch_size = 224, 14
grid = img_size // patch_size # 16
num_patches = grid * grid # 256
vit_dim, llm_dim = 1024, 4096
rng = np.random.default_rng(0)
visual_tokens = rng.standard_normal((num_patches, vit_dim)) # (256, 1024)
W_proj = rng.standard_normal((vit_dim, llm_dim)) * 0.02 # (1024, 4096)
projected = visual_tokens @ W_proj # (256, 4096)
text_tokens = rng.standard_normal((40, llm_dim)) # (40, 4096)
sequence = np.concatenate([projected, text_tokens], axis=0) # (296, 4096)
print(sequence.shape)
Running this prints (296, 4096): 224/14 = 16 patches per side, 16×16 = 256 patches, each projected from the vision encoder's 1024 dimensions into the LLM's 4096, then concatenated with 40 text tokens (already in 4096-dim LLM embedding space) to give a 296-token sequence the causal decoder processes exactly like a pure-text 296-token prompt. The frozen ViT never sees this concatenation — its job ends at producing the (256, 1024) grid; the trainable projector is the only new arithmetic that touches vision-space numbers before the LLM takes over.
Common misconception: "training a VLM" means one joint loss over both modalities from scratch
The natural mental model — reasonable, but wrong — is that a vision-language model is trained the way a plain classifier is: assemble a big multimodal dataset, wire up a vision branch and a language branch, and back-propagate one combined loss through the whole thing end-to-end from randomly initialized weights. Almost no production system does this, and it is not merely an engineering shortcut — it would be worse on the merits even with unlimited compute. Vision encoders need enormous image datasets to learn good low-level visual features (edges, textures, object parts); LLMs need enormous text corpora to learn grammar, world knowledge, and reasoning. Multimodal paired data (an image next to a well-formed caption or instruction) is orders of magnitude scarcer than either of those unimodal sources alone. Training everything jointly from scratch on the scarce paired data would starve both branches of the unimodal signal they need to become competent in the first place, and would also throw away two enormously expensive pretraining runs whose outputs (CLIP, Vicuna, LLaMA, etc.) already exist and are public. Every real recipe covered here — CLIP's contrastive stage, LLaVA's projector stage, BLIP-2's Q-Former stage — instead starts from independently pretrained, often frozen, unimodal components and trains only a small bridge (a linear layer, a Q-Former, or in CLIP's case the encoders themselves but on a comparatively narrow contrastive objective) to connect spaces that are each already well-formed on their own. When you next read that a VLM "was trained on N million image-text pairs," check what was actually frozen during that training — it is rarely everything, and it is almost never nothing.
Diagram: the two-stage unified-VLM training pipeline
Active recall
Q1. In the worked 3×3 example, why does row 3 (image I3, correct caption T3) produce a loss of essentially zero while row 1 produces a loss of about 0.127, even though both rows have the same correct-pair similarity of 0.8?
Q2. The chapter increased temperature from τ=0.1 to τ=1.0 on the same similarity matrix S. Recompute what happens to the loss, and trace the full ripple effect: does this only change the loss's numeric value, or does it change anything about which pairs dominate the gradient?
Q3. A team wants to double their contrastive training batch size from 8,192 to 16,384 image-text pairs to get more negatives per step, using the same GPU cluster and no other changes. Name two distinct memory costs this increase causes (not just "more memory") and one architectural technique from the chapter that addresses each.
Q4. In LLaVA's Stage 1, why is the LLM kept frozen and only the projector trained, rather than training the LLM too, given that Stage 1 already has fewer training pairs (595K) than would be needed to train an LLM from scratch?
Q5. A classmate claims: "Since BLIP-2's Q-Former also connects a frozen vision encoder to a frozen LLM, it must be doing exactly the same job as LLaVA's linear projector, just with a fancier name." What is the concrete difference in what the two produce as output, and why does that difference matter for images of varying resolution?
Q6. Using the LLaVA shape-trace code, if the vision encoder is swapped from ViT-L/14 (patch size 14) to ViT-L/16 (patch size 16) on the same 224×224 image, and the LLM is swapped from a 4096-dimensional model to a 5120-dimensional one, what are the new shapes of visual_tokens, W_proj, and the final concatenated sequence (still with 40 text tokens)?
Worked answers
A1. The loss is not a function of the correct pair's similarity alone — it is a function of how much the correct pair's similarity exceeds its competitors, after the sharpening effect of dividing by τ. Row 3's competitors are -0.8 and -0.6 (both far below the correct 0.8), so after division by τ=0.1 the correct logit (8) dominates the softmax completely (softmax probability ≈0.9999991). Row 1's nearest competitor is T2 at 0.6, only 0.2 below the correct 0.8; after the same division that gap becomes only 2 in logit space (8 vs 6), leaving real competing probability mass on the wrong answer. Same correct-pair strength, very different loss, because contrastive loss is inherently about relative separation from negatives, not absolute similarity.
A2. Recomputing (verified in code): at τ=1.0 the per-row losses become approximately 0.7034, 0.7253, and 0.3705, averaging to a total symmetric loss of about 0.600 — roughly 7× larger than the τ=0.1 loss of 0.0846. The ripple is not just "the number got bigger." At τ=0.1 the softmax was nearly saturated (row 3's correct probability was 0.9999991), so that pair contributed almost no gradient — training was effectively driven only by the hard row-1/row-2 negatives. At τ=1.0 the softmax is far less peaked, so even the "easy" row 3 pair, which contributed near-zero gradient at τ=0.1, now contributes a real gradient (loss 0.3705, not near zero) alongside rows 1 and 2. Raising τ therefore redistributes gradient signal more evenly across easy and hard pairs instead of concentrating it on hard negatives, at the cost of less sharp per-step discrimination. This is precisely why CLIP treats τ as a learned parameter clipped to a maximum logit scale (equivalently a minimum τ) rather than a fixed constant — the right sharpness shifts as embeddings separate over training.
A3. First cost: activation memory inside each encoder scales with the number of samples processed per step, so doubling batch size roughly doubles per-device activation memory even before the loss is computed — addressed by gradient checkpointing (recompute activations during the backward pass instead of storing them). Second cost: the similarity matrix itself is N × N, so doubling N quadruples the matrix's memory footprint and the memory needed to hold all-gathered embeddings from every device if the batch is sharded — addressed by sharding the similarity-matrix computation across GPUs and only communicating the (much smaller) embedding vectors, not full activations, between devices.
A4. Freezing the LLM in Stage 1 is not primarily about the pair count being too small (though 595K pairs would indeed be nowhere near enough to train an LLM's language competence from scratch) — it is about what Stage 1 is trying to teach. Stage 1's only job is to place visual tokens somewhere the LLM's existing, already-competent embedding space can interpret them; that is a property of the projector alone. If the LLM's weights were also allowed to move during Stage 1, the model could "cheat" by warping the LLM's language representations to accommodate a badly-placed projector, degrading the LLM's pretrained language ability while appearing to reduce the training loss — a route Stage 1 specifically closes off by freezing the LLM, reserving joint fine-tuning for Stage 2 where enough higher-quality instruction data (158K examples, generated with GPT-4 assistance) exists to adjust the LLM safely.
A5. LLaVA's projector produces one output vector per input patch — for a 224×224 image with 14×14 patches that is 256 vectors, so the number of visual tokens fed to the LLM scales with image resolution and patch count. BLIP-2's Q-Former instead uses a fixed set of 32 learnable query vectors that cross-attend into the frozen image encoder's features and always output exactly 32 vectors, regardless of the input image's resolution or the vision encoder's patch grid size. This matters because LLM context length is a hard budget: LLaVA's approach lets sequence length grow (and therefore inference compute grow) with image resolution, while BLIP-2's fixed 32-token bottleneck keeps the visual footprint on the LLM's context constant no matter how large or detailed the source image is, at the cost of compressing all of that image's information through a fixed-size bottleneck.
A6. Patch size 16 on a 224×224 image gives a grid of 224/16 = 14 patches per side, so 14×14 = 196 patches — visual_tokens becomes shape (196, 1024) (the vision hidden width, 1024, is unaffected by patch size). The projector must now map into a 5120-dimensional LLM space, so W_proj becomes shape (1024, 5120), and the projected visual tokens become (196, 5120). Text tokens must also live in the new 5120-dimensional space, so text_tokens is (40, 5120), and the final concatenated sequence is (196 + 40, 5120) = (236, 5120) — shorter than the original 296-token sequence (fewer, coarser patches from the larger patch size) but wider in every embedding (larger LLM hidden dimension).
Think About It
Think about this: How would you explain multimodal training: unified vision-language models and cross-modal alignment to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where multimodal training: unified vision-language models and cross-modal alignment is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting multimodal training: unified vision-language models and cross-modal alignment to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind multimodal training: unified vision-language models and cross-modal alignment, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.