The bridging problem production teams actually face
Picture a WhatsApp-based crop-advisory service built for farmers across Maharashtra and Punjab. A farmer photographs a diseased cotton leaf and sends it to the bot. The backend calls a large, frozen language model through a paid API — frozen because the team cannot afford to fine-tune an 11-billion-parameter model, does not own its weights, and does not want to risk the model forgetting the fluent Hindi and Marathi agricultural vocabulary it already knows. The request also carries the last few turns of conversation (the farmer already mentioned the crop is BT cotton and it rained three days ago) and a system prompt describing the bot's role. All of this has to fit inside a fixed context window, and every token in that window is billed and adds latency.
This is the real engineering problem behind "combining vision and language": a vision encoder like a Vision Transformer does not output a single vector for an image — it outputs one embedding per image patch. A standard ViT-g/14 run on a 224×224 photo splits the image into a 16×16 grid of 14-pixel patches (224/14 = 16 per side, 16×16 = 256 patches), prepends a [CLS] token, and returns 257 vectors of dimension 1408. Somehow those 257 vectors have to reach the language model. Other fusion approaches include CLIP's contrastive pretraining, which aligns a single pooled image vector with a single pooled text vector so the two live in a shared space, and Flamingo/LLaVA-style architectures, where the language model's own transformer layers grow new cross-attention sub-layers that reach directly into the image encoder's patch features. Both of those approaches change what happens inside the language model's forward pass, or work with a single global vector rather than the image's fine-grained detail. This chapter goes deep on a third, different fusion mechanism — the one that let the field build vision-language systems without touching either the vision encoder or the language model's own weights at all: the Querying Transformer (Q-Former) introduced in BLIP-2 (Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models," ICML 2023).
A learned bottleneck instead of a wired-in connection
The Q-Former's idea is stubbornly simple to state and subtle to get right: insert a small, separately trained module between the frozen vision encoder and the frozen language model whose entire job is to compress 257 patch embeddings down into a fixed, small number of vectors — 32 in the original paper — that the language model can consume exactly the way it consumes ordinary word embeddings. Nothing about the language model's architecture changes. It never learns a new attention pattern, never grows a layer, never sees a patch grid. It just receives 32 extra "words" at the front of its input sequence, prepended before the text prompt, and generates a response the same way it always has.
Structurally, the Q-Former is a 12-block transformer, initialized from BERT-base weights, built around 32 learnable query embeddings of dimension 768 — parameters that exist independently of any particular input image, the same way a CNN's convolutional filters exist independently of any particular photo. Each block runs self-attention among the 32 queries (so they can coordinate and avoid duplicating each other), and every second block additionally runs cross-attention from the queries into the frozen image encoder's 257 patch features. Because BERT-base has no cross-attention sub-layers to inherit weights from, those cross-attention layers are the one part of the Q-Former initialized randomly; everything else starts from pretrained language weights, which gives the module a head start at handling text before it ever sees a training gradient. After the final block, the 32 queries have become 32 output vectors, still dimension 768. A single linear layer then projects each of those 768-dimensional vectors into whatever embedding dimension the target frozen language model expects — 2560 for OPT-2.7B, for instance — producing 32 "soft visual tokens" that get concatenated in front of the text tokens before the frozen language model runs its ordinary forward pass.
The consequence worth sitting with: during training, gradients flow backward from the language model's loss, through the projection layer, through the Q-Former's cross-attention and self-attention weights, and into the 32 query embeddings themselves — but they stop there. They never reach the frozen ViT's convolution-like patch embedding or attention weights, and they never reach a single weight inside the frozen language model. Only the Q-Former (about 188 million parameters) and the small projection layer ever update.
Two training stages, and why both exist
BLIP-2 trains the Q-Former in two stages, and the reason for the split is itself an engineering lesson. In stage 1 (vision-language representation learning), the image encoder is already frozen, but there is no language model in the loop yet. The Q-Former is trained against three losses simultaneously, reusing the objective set from the original BLIP paper: image-text contrastive learning (pull matching image/caption pairs together, push mismatched pairs apart — similar in spirit to CLIP's objective, but here applied to the 32 query outputs rather than a single pooled vector), image-text matching (a binary classifier deciding whether an image and caption truly correspond), and image-grounded text generation (predicting caption tokens autoregressively, conditioned on the queries). These three objectives share the same Q-Former weights but use different self-attention masks — the contrastive loss blocks queries from seeing the caption text entirely so they can't cheat by copying it, the matching loss lets queries and text attend to each other freely, and the generation loss uses a causal mask so caption tokens can only see earlier caption tokens and all of the queries. Running all three together forces the queries to extract information that is genuinely tied to the image, not generic filler, before the module ever meets a language model.
In stage 2 (vision-to-language generative learning), a frozen large language model is attached via the projection layer, and training continues with an ordinary language-modeling loss: given the 32 projected visual tokens plus a text prompt, generate the correct caption or answer. Only the Q-Former and the projection layer continue to update — the language model stays frozen throughout.
Stage 1 is not a convenience; it is load-bearing. If a team skipped straight to stage 2 — attaching randomly initialized queries directly to a frozen language model and training only against its generative loss — the queries would have no dedicated pressure to encode visual content specifically, only pressure to produce something that helps the language model's next-token prediction. Since the language model is frozen and already fluent, the easiest way to reduce that loss early in training is to fall back on generic, image-independent patterns rather than genuinely visual ones — this is a sound inference from the architecture's incentive structure, not a claim about a specific ablation number. Stage 1's contrastive and matching losses exist precisely to prevent that shortcut before the harder generative stage begins. The paper reports the payoff: BLIP-2 outperforms Flamingo-80B by roughly 8.7 points of zero-shot accuracy on VQAv2 while training 54 times fewer trainable parameters — the entire trainable footprint is the ~188M-parameter Q-Former plus a small projection, against a multi-billion-parameter vision encoder and language model that never move.
Worked example: tracing the cross-attention step by hand
The real Q-Former runs 32 queries of dimension 768 attending over 257 patches of dimension 1408, with learned key/value projection matrices splitting the computation across multiple attention heads — too large to trace by hand. The mechanism itself, though, is nothing more than scaled dot-product attention, and it is small enough to compute exactly with a toy example: 2 queries, 3 "patches," dimension 4, and identity projection matrices (so we attend directly on the raw vectors, which isolates exactly what cross-attention does without the extra bookkeeping of learned Q/K/V weights).
import numpy as np
# Frozen image encoder output: 3 patch embeddings, dim 4 (toy-sized;
# the real Q-Former sees 257 patches of dim 1408)
X = np.array([
[1.0, 0.0, 1.0, 0.0], # patch 1
[0.0, 2.0, 0.0, 2.0], # patch 2
[1.0, 1.0, 1.0, 1.0], # patch 3
])
# Two learnable Q-Former query vectors, dim 4
# (the real Q-Former uses 32 queries, dim 768)
Q = np.array([
[1.0, 1.0, 0.0, 0.0], # query 1
[2.0, 0.0, 0.0, 1.0], # query 2
])
d = X.shape[1] # 4
scores = (Q @ X.T) / np.sqrt(d) # (2, 3) attention logits
weights = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True) # row-wise softmax
output = weights @ X # (2, 4) one row per query
print(np.round(scores, 3))
print(np.round(weights, 3))
print(np.round(output, 3))Trace it by hand to check the code before running it. Query 1, [1,1,0,0], dotted with each patch: with patch 1 [1,0,1,0] gives 1, with patch 2 [0,2,0,2] gives 2, with patch 3 [1,1,1,1] gives 2. Divide by √4 = 2: scores = [0.5, 1.0, 1.0]. Query 2, [2,0,0,1], dotted with the same patches: with patch 1 gives 2, with patch 2 gives 2, with patch 3 gives 2·1 + 0 + 0 + 1·1 = 3. Divided by 2: scores = [1.0, 1.0, 1.5]. So scores prints as [[0.5, 1.0, 1.0], [1.0, 1.0, 1.5]].
Softmax of [0.5, 1.0, 1.0]: e^0.5 ≈ 1.6487, e^1.0 ≈ 2.7183 (twice), summing to ≈7.0853, giving weights ≈ [0.233, 0.384, 0.384]. Softmax of [1.0, 1.0, 1.5]: e^1.0 ≈ 2.7183 (twice), e^1.5 ≈ 4.4817, summing to ≈9.9182, giving weights ≈ [0.274, 0.274, 0.452]. So weights prints as [[0.233, 0.384, 0.384], [0.274, 0.274, 0.452]] — each row sums to 1.0, as any softmax output must.
Finally, each query's output is the weighted sum of the three patch vectors using its own row of weights. For query 1: 0.233·[1,0,1,0] + 0.384·[0,2,0,2] + 0.384·[1,1,1,1] ≈ [0.616, 1.151, 0.616, 1.151]. For query 2: 0.274·[1,0,1,0] + 0.274·[0,2,0,2] + 0.452·[1,1,1,1] ≈ [0.726, 1.000, 0.726, 1.000]. So output prints as approximately [[0.616, 1.151, 0.616, 1.151], [0.726, 1.0, 0.726, 1.0]]. Two different learnable queries, seeing the same three patches, produced two genuinely different output vectors — because their raw dot products with the patches differed, which shifted where each query's softmax placed its attention mass (query 2 leaned harder on patch 3). This is the entire mechanism, repeated 32 times with 768-dimensional learned vectors and real key/value projections in the actual model: a small number of trainable "questions" (the queries), each independently deciding how to blend a large number of fixed visual facts (the patches) into one compact answer.
Why compression is the point, not a limitation
Return to the WhatsApp bot. Suppose the frozen language model has a 4096-token context window, the running conversation history averages 800 tokens, and the system prompt is 150 tokens. Feeding all 257 patch tokens directly — the naive approach some early vision-language systems used — leaves 4096 − 257 − 800 − 150 = 2889 tokens for the model's own generated diagnosis. The Q-Former's 32 tokens leave 4096 − 32 − 800 − 150 = 3114 tokens, about 225 more, roughly 7.8% additional headroom for a longer explanation in the farmer's own language. At the scale the bot actually runs — say 2,000,000 images processed per day — the naive approach bills (257 − 32) × 2,000,000 = 450,000,000 extra input tokens every single day, purely to represent images the model then mostly ignores in favor of the handful of patches actually relevant to the diseased leaf. That gap is precisely the pressure the Q-Former's fixed 32-token bottleneck is designed to relieve: it forces the extraction to happen once, in a small trainable module, instead of forcing every downstream language-model call to pay for 257 raw patches.
The common misconception
Students who follow the training description above often conclude: "so the language model learns to read images." It doesn't — not in the sense of its own weights changing. The frozen language model's parameters are never touched in either stage of BLIP-2 training. What happens instead is that the Q-Former and its projection layer are trained to land the 32 output vectors inside whatever region of embedding space the already-frozen language model already knows how to interpret as meaningful context — the same way a well-chosen prompt can steer a frozen model's behavior without changing a single one of its weights. The language model is not being taught to see; it is being handed 32 unusually-shaped "words" that were manufactured, through training pressure applied entirely outside the language model, to sit somewhere in its vocabulary space that it already knows how to use. This is also exactly why the frozen language model can be swapped for a different one (OPT-2.7B, OPT-6.7B, or FlanT5-XXL, as the BLIP-2 paper demonstrates) simply by retraining the projection layer and Q-Former against the new target — no change to the language model itself is ever required.
The diagram
Active recall
Attempt each question before reading its answer.
Q1. Why does BLIP-2 compress the image down to 32 learnable query outputs instead of feeding all 257 patch embeddings straight into the language model, the way an LLaVA-style architecture feeds projected patches as tokens?
Q2. During stage 1 training, the image encoder is already frozen. Explain how the Q-Former still learns anything useful about images if it can never adjust the encoder that produces the features it attends over.
Q3. Suppose you increase the number of learnable queries from 32 to 64, keeping the image encoder, the language model, the projection layer's output dimension, and the training data all fixed. Trace the full effect on: (a) context tokens consumed per image call to the frozen language model, (b) the parameter count of the linear projection layer, (c) the compute cost of self-attention among the queries, (d) representational capacity versus redundancy, and (e) any required change to the frozen language model's weights.
Q4. True or false: during BLIP-2's stage 2 training, gradients update the frozen language model's attention weights so it learns to interpret the visual tokens better. Justify your answer.
Q5. The self-attention cost among the Q-Former's query tokens scales roughly as (number of queries)² × embedding dimension per layer, ignoring constant factors. Compute this quantity for 32 queries (dim 768) and for 64 queries (dim 768), and confirm the ratio matches the 4× factor claimed in Q3(c).
Worked answers
A1. Context budget and compute cost. A frozen language model has a fixed context window shared by the image, conversation history, and system prompt, and every extra token costs money and latency on a paid API and adds quadratic self-attention cost inside the language model itself. Feeding 257 raw patches burns roughly 8× the context budget that 32 compressed tokens do, and — as the WhatsApp bot example showed — leaves far less room for the response and for conversation history. The 32-query bottleneck also forces the Q-Former to discard redundant or irrelevant patch information during training rather than pushing that filtering work onto the frozen language model at inference time.
A2. The frozen encoder is a fixed feature extractor, not a blank slate — it was itself pretrained (as an EVA-CLIP-style vision transformer) to produce features that already carry rich visual structure. Freezing it during stage 1 doesn't mean gradients disappear; gradients from the three losses (contrastive, matching, generative) still flow backward through the cross-attention layers into the query embeddings and the Q-Former's own weights, teaching the queries how to selectively read the encoder's existing features. What's frozen is the feature extractor's job of turning pixels into features; what's trainable is the Q-Former's job of turning those (unchanging) features into something useful. This is also why a strong, already well-pretrained encoder matters — the Q-Former is never given the chance to fix a weak one.
A3. (a) Context tokens consumed doubles, from 32 to 64 — in a 4096-token window, from about 0.78% to about 1.56% of the budget. (b) The projection layer's weight matrix maps each 768-dimensional query output to the language model's embedding dimension independently — the same matrix is applied to every query vector, so its parameter count is unchanged; only the number of times it is applied per forward pass doubles, doubling its FLOPs, not its size. (c) Self-attention among queries scales quadratically in the number of queries, so cost roughly quadruples (64²/32² = 4), while the cross-attention into the (unchanged) 257 image patches scales only linearly in the number of queries, so that part merely doubles. (d) More queries can capture finer-grained, more distinct visual concepts, but risk redundant or overlapping attention patterns if the image doesn't contain 64 genuinely distinct things worth separately encoding — a diminishing-returns tradeoff, which is why the BLIP-2 authors settled on 32 rather than a much larger number. (e) None — the frozen language model's architecture doesn't know or care how many soft tokens precede the text prompt; it just processes a longer input sequence. This is the entire appeal of the prompt-level bridge: doubling the visual token count requires zero architectural change to the frozen model, only more compute for a longer sequence.
A4. False. In both stages of BLIP-2 training, the language model's weights never update — only the Q-Former (~188M parameters) and the small linear projection train. What actually happens is the reverse of the misconception: the Q-Former and projection are trained to produce 32 vectors that land inside whatever embedding-space region the already-frozen, already-fluent language model already knows how to condition on, similar in spirit to how a soft prompt steers a frozen model's output without touching its weights. The language model doesn't learn to see; the visual encoder-plus-Q-Former pipeline learns to speak the language model's existing embedding language.
A5. For 32 queries: 32² × 768 = 1024 × 768 = 786,432. For 64 queries: 64² × 768 = 4096 × 768 = 3,145,728. The ratio is 3,145,728 / 786,432 = 4.0 exactly, confirming the 4× scaling: doubling the number of queries quadruples n² while the embedding dimension d stays fixed at 768, so the self-attention cost among queries scales purely with the square of the query count.
Think About It
Think about this: How would you explain multimodal models: combining vision and language to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where multimodal models: combining vision and language is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting multimodal models: combining vision and language to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind multimodal models: combining vision and language, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.