Multimodal AI: Vision-Language Models

Point a phone camera at a wilting cotton leaf, ask "what disease is this and what do I do about it" in Marathi or Hindi, and a model like Gemini or GPT-4V will answer in a full sentence, in the same language, referencing the specific lesion pattern on the leaf. Nothing about that exchange is one system doing two separate jobs — a captioning model handing off to a translator handing off to a chatbot. It is a single network that took in pixels and a question together and produced language that is actually grounded in what the pixels show. That capability is the subject of this chapter: how do you build a model that reasons over an image and a sentence in the same computation, when an image is a grid of continuous-valued pixels and a sentence is a sequence of discrete tokens — two representations that, on the face of it, have nothing in common?

The answer, worked out across a sequence of papers from 2020 to 2023, is to force both modalities through the same kind of object: a sequence of vectors in one shared embedding space. Once an image and a sentence are both "just" sequences of vectors of the same dimension, the transformer architecture you already know from language modelling does not care which modality a vector came from. Everything in this chapter is the engineering of that conversion — how pixels become vectors, how those vectors get aligned with text vectors, and how a language model is taught to read them.

From pixels to tokens: the vision encoder

A transformer operates on a sequence of vectors; a photograph is a 2-D grid of pixel intensities. The bridge is patch tokenization, introduced by Dosovitskiy et al. in "An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale" (2020), which gave us the Vision Transformer, or ViT. The idea is deliberately unglamorous: chop the image into a regular grid of small square patches, flatten each patch into a single long vector, and linearly project that vector into the model's working dimension. From that point on, a patch is a token, indistinguishable in kind from a word-piece token in an LLM.

Trace the arithmetic for a standard input size. A ViT-B/16 model (the "16" names the patch size) takes a 224×224×3 image. Dividing each spatial dimension by the patch size gives 224 / 16 = 14 patches per side, so the image becomes a 14×14 grid — 196 patches in total. Each patch is 16×16 pixels across 3 colour channels, so flattening one patch gives a vector of length 16 × 16 × 3 = 768 raw numbers. A learned linear layer projects that 768-length raw vector into the model's hidden dimension — which, for ViT-B, is also 768, purely by the base model's design choice, not because the numbers have to match. A single extra learnable vector, the classification ([CLS]) token, is prepended to carry a pooled, whole-image summary. The encoder's input is therefore a sequence of 197 vectors (196 patches + 1 CLS), each of dimension 768, and a stack of ordinary transformer self-attention blocks — 12 of them in the base model — lets every patch attend to every other patch, exactly as word tokens attend to each other in a text transformer. The output is 197 vectors of dimension 768: one per patch, each now informed by the whole image, plus the pooled CLS vector.

The code below traces the reshape arithmetic exactly, so the token count above is not just asserted — it is derived by running the actual splitting operation on a real array.

import numpy as np

def patchify(image, patch_size=16):
    """image: (H, W, C) array. Returns (num_patches, patch_size*patch_size*C)."""
    H, W, C = image.shape
    assert H % patch_size == 0 and W % patch_size == 0
    n_side = H // patch_size
    patches = image.reshape(n_side, patch_size, n_side, patch_size, C)
    patches = patches.transpose(0, 2, 1, 3, 4)   # group patch-grid axes together
    patches = patches.reshape(n_side * n_side, patch_size * patch_size * C)
    return patches

image = np.zeros((224, 224, 3), dtype=np.float32)
patches = patchify(image, patch_size=16)
print(patches.shape)          # (196, 768)

cls_token = np.zeros((1, 768), dtype=np.float32)
projected = patches @ np.eye(768)          # stand-in for the learned linear projection
sequence = np.concatenate([cls_token, projected], axis=0)
print(sequence.shape)          # (197, 768)

Running this: patchify reshapes the (224, 224, 3) array into (14, 16, 14, 16, 3), which splits the height axis into "which patch row" and "offset within that row", and the width axis likewise, without reordering any pixel data — the reshape is valid precisely because 14 × 16 = 224 on both axes. The transpose then groups the two "which patch" axes next to each other and the two "offset within patch" axes next to each other, so the final reshape correctly flattens each 16×16×3 patch into one contiguous 768-length row. The result, patches.shape, is exactly (196, 768) — 196 patches, each 768 raw numbers. After the (stand-in) linear projection and prepending the CLS token, sequence.shape is (197, 768), matching the derivation above term for term.

Aligning two modalities: CLIP and contrastive pretraining

A ViT alone gives you image vectors; it says nothing about language. The paper that made those vectors speak the same language as text is Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (OpenAI, ICML 2021) — CLIP, for Contrastive Language-Image Pre-training. The insight is that the internet already contains hundreds of millions of naturally occurring (image, caption) pairs — a product photo on an e-commerce listing next to its title, a photo next to its alt-text — and you never need to hand-label a single one of them with a fixed category. CLIP trained a ViT image encoder and a Transformer text encoder jointly on 400 million such pairs, with one objective: make the vector for an image and the vector for its matching caption point in the same direction, and make the vectors for mismatched image-caption pairs point apart.

This is done with a contrastive loss called InfoNCE. Take a batch of N images and N captions. Encode every image and every caption into a vector of the same dimension, normalize each vector to unit length, and compute the cosine similarity between every image and every caption in the batch — an N×N matrix where the diagonal holds the true pairs and everything off-diagonal is a mismatch. Divide every similarity by a temperature τ (a small positive number that sharpens the distribution) to get logits, then treat each row of that matrix as a classification problem: "given image i, which of the N captions is the real one?", solved with ordinary softmax cross-entropy. Do the same down each column ("given caption j, which image is real?"). Average the two directions, and that is the loss CLIP minimizes.

Work it through on a concrete 3-image, 3-caption batch. Suppose the pretrained encoders currently produce these cosine similarities (each row is one image, each column one caption, diagonal = correct pairing):

          T1    T2    T3
I1      0.90  0.20  0.15
I2      0.25  0.85  0.30
I3      0.10  0.28  0.88

With temperature τ = 0.2, the logits (similarity ÷ τ) for row I1 are [4.5, 1.0, 0.75]. Exponentiating gives [e^4.5, e^1.0, e^0.75] = [90.017, 2.718, 2.117], which sum to 94.852. Dividing each term by that sum gives the softmax row [0.949, 0.029, 0.022] — the model assigns 94.9% probability to the correct caption T1, and the cross-entropy loss for that one row is −ln(0.949) = 0.0523. Carrying the same computation through all three rows (image→text) and all three columns (text→image) and averaging both directions gives a symmetric batch loss of 0.0756 — small, because the diagonal similarities (0.85–0.90) are comfortably higher than the off-diagonal ones (0.10–0.30) in this example, meaning the encoders are already separating correct from incorrect pairs well.

Now change one thing and trace the ripple: sharpen the temperature from τ = 0.2 to τ = 0.1, with the exact same similarity matrix. Every logit doubles (row I1 becomes [9.0, 2.0, 1.5] instead of [4.5, 1.0, 0.75]), which exaggerates the gap between the largest logit and the rest. Recomputing the softmax for row I1 gives [0.9985, 0.0009, 0.0006] — essentially certainty on the correct caption — and the full symmetric loss over the batch drops from 0.0756 to 0.0035. A lower temperature makes an already-correct model's loss shrink faster, because it rewards confidence more steeply; the real cost shows up early in training, before the encoders have separated correct from incorrect pairs, when an overly sharp temperature turns small, noisy similarity differences into huge, unstable gradients. This is exactly why CLIP does not fix τ by hand — it makes it a learned parameter, clamped so the effective logit scale (1/τ) never exceeds 100, trading away some of the theoretical benefit of an aggressively low temperature for training stability.

The output of CLIP pretraining is not a classifier for any fixed label set — it is a pair of encoders that map images and text into one shared vector space, where semantic similarity is geometric closeness. That shared space is the substrate every architecture in the next section builds on.

From alignment to generation: bridging vision encoders into language models

CLIP can tell you which caption best matches an image (that is exactly the mechanism behind "search my photos for a photo of a red saree" or a visual-search bar on an e-commerce app), but it cannot hold a conversation about an image, count objects in it, or answer an open-ended question — it only ranks. Producing fluent, open-ended answers requires connecting a vision encoder to a full autoregressive language model, and three architectures from 2022–2023 show three different ways to build that connection.

Flamingo (Alayrac et al., DeepMind, NeurIPS 2022) keeps a large frozen language model (built on Chinchilla) almost entirely untouched, and instead inserts new, trainable gated cross-attention layers between the LLM's existing layers. A Perceiver Resampler first compresses a variable number of vision-encoder features per image into a small fixed number of vectors, and the new cross-attention layers let every LLM layer look back at those vision vectors while generating text. Each inserted layer is gated by a learned scalar initialized near zero, so at the start of training the model behaves exactly like the original frozen LLM and only gradually learns to lean on the visual signal — a stability trick that lets a huge pretrained language model be adapted without catastrophic forgetting of its language ability.

BLIP-2 (Li, Li, Savarese and Hoi, Salesforce Research, ICML 2023) takes a different route: it freezes both the vision encoder and the LLM completely, and trains only a small bridge module called the Q-Former. The Q-Former holds 32 learnable query vectors that cross-attend into the frozen image encoder's patch features and self-attend with text during a first alignment stage, distilling the (up to) 197 patch vectors down to a fixed 32 vectors regardless of image resolution. Those 32 vectors are then projected once more and handed to the frozen LLM as if they were a short prefix of input tokens. Because the two enormous networks never update, BLIP-2 trains roughly 190 million parameters (a BERT-base-initialized Q-Former plus its added cross-attention layers and output projection) to connect billion-parameter vision and language models.

LLaVA (Liu, Li, Wu and Lee, NeurIPS 2023) is the simplest of the three, and it is the architecture the diagram below follows. Take the sequence of patch embeddings straight out of a frozen CLIP ViT encoder, pass every one of them through a small trainable projection — in the original LLaVA paper, a single linear layer; in the follow-up LLaVA-1.5, a 2-layer MLP — so that each 768-dimensional patch vector becomes a vector of whatever dimension the target LLM's own token embeddings use (4096, for a 7-billion-parameter Vicuna model). Those projected vision vectors are then simply concatenated, as a literal prefix, in front of the tokenized text prompt, and the whole sequence — vision vectors and text vectors side by side — is fed into the language model exactly as if it were all text. No architectural change to the LLM is required at all; from the decoder's point of view, "image tokens" and "word tokens" are indistinguishable rows in the same input matrix, and ordinary causal self-attention lets later text tokens attend back over the earlier vision tokens when generating an answer.

All three share one economic pattern worth naming explicitly: the two largest networks in the system — the vision encoder (hundreds of millions of parameters) and the language model (billions) — are pretrained once, separately, on separate objectives, and then frozen. Only a small connector is trained to align them, sometimes followed by a lightweight fine-tuning pass (often via LoRA, adding low-rank trainable matrices to the frozen LLM rather than updating all its weights). A projector with on the order of tens of millions of parameters, trained against a frozen 7-billion-parameter LLM, is a training job that fits on a handful of GPUs rather than the thousand-plus-GPU cluster required to pretrain either giant network from scratch — which is exactly why the VLM boom of 2023 onward was led as much by small academic labs as by frontier AI companies.

Architecture: how an image becomes an answer

The diagram traces the LLaVA-style pipeline end to end, from a raw image to a generated answer, with every intermediate tensor's shape labelled.

Production considerations: the token budget an image actually costs

The concatenation step in the diagram exposes a fact that matters directly for anyone serving a VLM at scale: an image is expensive in the exact currency an LLM cares about — context-window tokens. A single 224×224 image at ViT-B/16 resolution costs 197 tokens before the user has typed a single word of their question. Real LLaVA and LLaVA-1.5 deployments actually use the larger CLIP ViT-L/14 encoder (patch size 14, not the ViT-B/16 used for the arithmetic above) at 336×336 resolution: 336 / 14 = 24 patches per side, so 24×24 = 576 patches per image — nearly three times the 196-patch, ViT-B/16 figure this chapter has been using for clean arithmetic — so one image alone can cost more tokens than several paragraphs of text. A chat interface that allows a user to attach multiple images in one conversation — comparing two product photos, or scrolling through a multi-page scanned document — multiplies that cost per image, and every one of those tokens participates in the LLM's self-attention, whose compute cost grows quadratically with sequence length. This is precisely why BLIP-2's Q-Former design — compressing any image down to a fixed 32 query vectors regardless of input resolution — is attractive for high-throughput serving despite adding an extra trained module: it decouples token cost from image resolution, capping the worst case, whereas LLaVA's simpler design ties token cost directly to how finely the image is patchified. Production systems handling high-resolution or multi-image inputs (LLaVA-NeXT's "AnyRes" tiling, or Gemini's native high-resolution handling) manage this the same way long-context text serving does — by budgeting the vision-token allowance the way you would budget an API rate limit, because it directly sets both GPU memory for the attention cache and per-request latency.

The misconception worth correcting directly

The fluency of a VLM's output makes it easy to assume that if the model can describe a photo in a grammatically perfect sentence, it has genuinely resolved every visual detail in that photo — precise counts, exact spatial layout, fine-grained differences between similar-looking objects. This is false, and the reason is specific to how these models are built, not a vague disclaimer. CLIP's contrastive objective only ever has to make an image's embedding closer to its matching caption's embedding than to any other caption's in the batch. Web captions almost never encode exact counts ("three people standing near a gate," not "3.0000 people"), precise spatial relationships, or fine visual detail that isn't needed to tell one photo apart from a plausible alternative caption — so nothing in the training signal ever forces the embedding to preserve that information. Tong, Liu, LeCun and Xie's 2024 paper "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs" (CVPR 2024) demonstrates this directly: they found pairs of images that look obviously different to a human — different textures, different object orientations, different counts — but that produce nearly identical CLIP embeddings, which they call "CLIP-blind pairs." Built into a benchmark (MMVP), these pairs cause state-of-the-art VLMs including GPT-4V and LLaVA to perform close to random chance, even though each model can describe either image individually in perfectly fluent, confident-sounding language. The fluency comes from the LLM's strong language-modelling prior finishing the sentence in a plausible way; it is not evidence that the vision pathway actually resolved the distinction the question is asking about. When a VLM's answer depends on counting, exact geometry, or a visual detail too fine-grained to have mattered for web-caption matching, treat the answer as a plausible guess dressed in confident language, not a verified observation.

Active recall

Attempt each question before reading its answer.

1. A ViT-style vision encoder processes a 224×224×3 image with patch size 32 instead of 16. How many patches result, how many tokens total including the CLS token, and what is the flattened dimension of each patch before projection?
2. In the worked InfoNCE example, the batch loss was 0.0756 at temperature τ = 0.2 and 0.0035 at τ = 0.1, using the same similarity matrix. Explain why the loss dropped, and name the concrete training risk of pushing τ even lower, early in training when the encoders have not yet separated correct from incorrect pairs.
3. In BLIP-2 and LLaVA, why is it economically significant that only the projector (or Q-Former) is trained while the vision encoder and LLM stay frozen? Give an order-of-magnitude comparison of parameter counts.
4. Looking at the concatenated-sequence stage of the diagram, why do vision tokens vastly outnumber text tokens for a typical single-sentence question, and what does that imply for a production chat app that allows several images per conversation?
5. A VLM confidently states "there are 3 people in this photo" when there are actually 5. Using what you know about CLIP's contrastive pretraining objective, explain why counting is a specific, predictable weak point rather than a random failure.
6. Flamingo re-injects vision information at every LLM layer via gated cross-attention; LLaVA injects it once, as a prefix of concatenated tokens. State one architectural consequence of each design choice.

Answers.

1. Patches per side = 224 / 32 = 7, so the grid is 7×7 = 49 patches. Adding the CLS token gives 50 tokens total. Each patch is 32 × 32 × 3 = 3,072 raw values before the linear projection maps it into the model's hidden dimension.

2. Halving τ doubles every logit (row I1's logits go from [4.5, 1.0, 0.75] to [9.0, 2.0, 1.5]), which sharpens the softmax around whichever entry was already largest. Since the diagonal (correct-pair) similarities were already higher than the off-diagonal ones, sharpening pushes the correct-pair probability even closer to 1 and the loss falls, here from 0.0756 to 0.0035. The risk is symmetric: early in training, when the model has not yet learned to separate correct from incorrect pairs and similarities are close together, the same sharpening amplifies small, noisy differences into large, unstable gradients, which is why CLIP learns τ as a parameter and caps it (logit scale ≤ 100) rather than fixing it low by hand.

3. A CLIP-scale ViT vision encoder has on the order of a few hundred million parameters and a 7-billion-parameter LLM decoder has roughly twenty times more than that; the connector trained from scratch is small either way, but not uniformly so across architectures. LLaVA's linear or 2-layer-MLP projector is a few million to about 20 million parameters — "tens of millions" at most. BLIP-2's Q-Former is initialized from BERT-base and then has cross-attention layers inserted to let its 32 query vectors attend to the frozen image encoder's features, which pushes its trainable parameter count to roughly 190 million — the better part of an order of magnitude larger than LLaVA's projector. Even the larger of the two connectors is still two-plus orders of magnitude below the 7-billion-parameter LLM, so training only the connector (optionally with LoRA adapters on the LLM) means updating a small fraction of the total system's parameters, which is the difference between a training run that needs a large pretraining cluster and one that fits on a handful of GPUs.

4. A single image contributes a fixed, resolution-determined number of tokens — 197 for a 224×224 ViT-B/16 image — regardless of how short the accompanying question is, whereas a typical single-sentence question is only 10–20 tokens; the image dominates because tokenization of an image is tied to pixel grid size, not to semantic content the way text tokenization is. For a chat app allowing multiple images per conversation, every additional image multiplies that fixed per-image token cost, so the context window fills up — and inference latency and attention-cache memory grow — from images alone long before the conversation's actual text content does, which is why production systems cap image count or resolution, or use a fixed-size compressor like BLIP-2's Q-Former to decouple token cost from resolution.

5. CLIP's contrastive pretraining only requires an image's embedding to be closer to its true caption's embedding than to other captions' embeddings in the batch; web alt-text captions rarely state exact counts, so nothing in that training signal ever forces the embedding to preserve precise numerosity. A vision-language model built on a CLIP-style encoder therefore was never trained to encode "exactly 5," only enough visual information to distinguish this photo's caption from a handful of plausible alternatives — the confident wrong count comes from the LLM decoder's fluent language generation, not a verified visual tally, which is exactly the CLIP-blind-pair failure mode described above.

6. Flamingo's per-layer gated cross-attention lets vision information be reconsidered at every stage of language generation and scales naturally to interleaved sequences of multiple images and text (useful for video or multi-image few-shot prompting), at the cost of a more complex architecture with new layers inserted throughout the frozen LLM. LLaVA's single prefix-concatenation is architecturally simpler — any off-the-shelf decoder-only LLM can be used unmodified — but it ties total context length directly and linearly to the number and resolution of images supplied, since every vision token permanently occupies a slot in the one shared sequence the LLM attends over.

Think About It

Think about this: How would you explain multimodal ai: vision-language models to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where multimodal ai: vision-language models is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting multimodal ai: vision-language models to at least 3 other topics you have studied.