A Swiggy support ticket lands with three attachments: a blurry photo of a spilled biryani box, a 14-second voice note switching between Hindi and English ("bhai dabba tilt ho gaya tha, half spilled"), and a typed line, "requesting refund, order #4417". A human agent reads all three at once and decides in seconds. To automate that decision with one model, you need a transformer whose attention layers can look at pixels, waveform-derived features, and text tokens in the same forward pass and reason across all three jointly — not three separate models whose outputs get glued together with if-else logic afterward. That single requirement — one shared computation, three unrelated native representations — is the actual engineering problem this chapter is about. It sits one level below "can the model answer questions about a picture," which is where a vision-language chapter typically stops. Here we open up the two architectures that production multimodal foundation models actually use to solve it, trace their compute costs numerically, and show why the choice between them determines not just how well a model understands other modalities, but whether it can generate them at all.
Why you can't just staple encoders together
The lazy design is obvious: take a pretrained image encoder, take a pretrained text-generating LM, concatenate the image encoder's output embedding in front of the text tokens, and let the LM attend over both. This fails for a specific, measurable reason. Encoders trained with a contrastive objective — CLIP is the canonical example — do not place matching image and text embeddings at the same point in the shared space; they place them close by cosine-similarity ranking, but the two modalities' embeddings still occupy geometrically separate regions, forming what Liang et al. (NeurIPS 2022, "Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning") call the modality gap — two distinct cones in embedding space rather than one shared manifold. A frozen LM's attention and MLP weights were never trained on vectors from the image cone. Feeding them in without adaptation gives the LM something closer to structured noise than to a token it can reason over. Whatever bridge you build has to do real work: either teach the LM's attention layers to interpret vectors from that foreign region, or eliminate the foreign region entirely by forcing every modality through a common discretization before the transformer ever sees it. Those are exactly the two production routes.
Two routes, laid out structurally
Route A keeps modalities in continuous vector space and inserts a trained bridge that lets a frozen language model attend into that space. Route B eliminates continuous space at the model's input/output boundary altogether — every modality gets converted into discrete integer codes drawn from one shared vocabulary, and a single transformer runs ordinary causal self-attention over the resulting sequence, with no separate cross-attention machinery at all. The diagram below lays out both, with grey boxes marking components that stay frozen (pretrained, never updated during multimodal training) and green boxes marking components that are newly trained.
Route A in detail: resampling before you attend
Flamingo (Alayrac et al., NeurIPS 2022) keeps a frozen pretrained vision encoder and a frozen pretrained Chinchilla-class LM entirely untouched, and trains only two new pieces. The first is the Perceiver Resampler: a small set of learned query vectors (Flamingo uses 64 of them) that cross-attend into however many patch features the vision encoder produced — regardless of whether that's one image or an eight-frame video clip — and always emit exactly 64 output latents. The resampler's own internal cost scales with however many input tokens it has to attend over, but everything downstream of it sees a fixed-size bottleneck. The second new piece is the gated cross-attention dense (GATED XATTN-DENSE) block, inserted between the frozen LM's existing self-attention layers. Text tokens act as queries; the 64 resampled visual latents act as keys and values. A learned scalar gate, passed through tanh and initialized to zero, multiplies this cross-attention block's output before it's added back into the residual stream — so at the start of training the new layers contribute nothing and the model behaves exactly like the untouched frozen LM, which is what makes training stable enough to only need to learn the new bridge rather than re-learn language modelling from scratch.
The resampler's fixed output size is also what makes cross-attention affordable. For one gated cross-attention layer, the dominant cost is computing attention scores between every text query and every visual key, then the weighted sum over visual values — roughly 4 × T × N_kv × d multiply-adds, where T is the text sequence length, N_kv is the number of visual keys/values, and d is the hidden dimension. Trace it for a video few-shot prompt:
T = 2048 # text context length
d = 2048 # hidden dimension
N_full = 8 * 256 # 8 video frames x 256 patches each, no resampling
R = 64 # Perceiver Resampler output tokens (fixed regardless of frame count)
def xattn_macs(T, N_kv, d):
return 4 * T * N_kv * d
full_cost = xattn_macs(T, N_full, d)
resampled_cost = xattn_macs(T, R, d)
speedup = full_cost / resampled_cost
print(f"visual tokens without resampling: {N_full}")
print(f"full cross-attention MACs: {full_cost:,}")
print(f"resampled cross-attention MACs: {resampled_cost:,}")
print(f"speedup: {speedup:.0f}x")Tracing it by hand: N_full = 8 × 256 = 2048. Without resampling, full_cost = 4 × 2048 × 2048 × 2048 = 34,359,738,368 multiply-adds for that one layer. With resampling to 64 latents, resampled_cost = 4 × 2048 × 64 × 2048 = 1,073,741,824. The printed speedup is 34,359,738,368 / 1,073,741,824 = 32 — exactly N_full / R = 2048 / 64, because the formula is linear in N_kv and every other term is unchanged. Every gated cross-attention layer in the stack gets this same 32× reduction, and — this is the detail worth holding onto — the reduction is completely independent of how many frames the video had, because the resampler always compresses down to 64 regardless of whether N started at 256 (one image) or 2048 (eight frames) or larger still.
Route B in detail: one vocabulary, no bridge at all
Chameleon (Chameleon Team, Meta AI, 2024, "Chameleon: Mixed-Modal Early-Fusion Foundation Models") takes the opposite bet: instead of building a bridge into a frozen LM's continuous space, eliminate the need for a bridge by converting every modality into discrete tokens from the start, and train one ordinary transformer with standard causal self-attention over the resulting mixed sequence — text tokens and image tokens sit in the exact same embedding table and get predicted by the exact same output softmax. The mechanism that makes an image into "tokens" a transformer can consume is vector quantization: an encoder compresses the image into a spatial grid of continuous latent vectors, and each vector is then snapped to the nearest entry in a learned, finite codebook of size K (van den Oord et al., NeurIPS 2017, "Neural Discrete Representation Learning" — the original VQ-VAE). The index of the matching codebook entry, an integer from 0 to K−1, is the "image token." Audio can join the same unified vocabulary through the identical trick applied to waveforms instead of pixels — neural audio codecs like SoundStream (Zeghidour et al., 2021) and EnCodec (Défossez et al., 2022) use residual vector quantization to turn audio into a short sequence of discrete codes the same way. This is the real payoff of early fusion over a vision-language-specific bridge: once a modality has a working tokenizer, it slots into the same transformer with zero new architecture.
Discretization has a concrete bit cost, and it's worth deriving rather than asserting. Suppose an image encoder downsamples a 256×256 image by a factor of 16 in each spatial dimension, producing a 16×16 grid of latent codes, and the codebook holds K = 8192 entries:
patch_grid = 16 # 16x16 grid of latent codes for a 256x256 image
num_patches = patch_grid * patch_grid # 256
codebook_size = 8192
bits_per_code = codebook_size.bit_length() - 1 # log2(8192) = 13
tokenized_bits = num_patches * bits_per_code
raw_bits = 256 * 256 * 3 * 8 # RGB, 8 bits/channel
compression_ratio = raw_bits / tokenized_bits
print(f"tokens per image: {num_patches}")
print(f"bits per token: {bits_per_code}")
print(f"tokenized size: {tokenized_bits} bits ({tokenized_bits/8:.0f} bytes)")
print(f"raw size: {raw_bits} bits ({raw_bits/8:.0f} bytes)")
print(f"compression ratio: {compression_ratio:.1f}x")Tracing it: num_patches = 256, and since 8192 = 2^13, bits_per_code = 13. So tokenized_bits = 256 × 13 = 3328 bits, or 416 bytes. The raw image is 256 × 256 × 3 × 8 = 1,572,864 bits, or 196,608 bytes. The printed compression ratio is 1,572,864 / 3,328 ≈ 472.6. That 416-byte token sequence is what actually enters the transformer's embedding table — a length-256 slice of integers indexing into a size-8192 lookup, indistinguishable in shape from 256 text tokens.
This unification is not free at training time. Chameleon's authors report that mixing modalities inside one softmax destabilizes optimization — image-code and text-token embeddings can develop very different activation and gradient statistics under a shared loss, which without intervention produces loss spikes and divergence — and the paper introduces query-key normalization and adjusted layer-norm placement specifically to control this. A related hybrid, Transfusion (Zhou et al., 2024, "Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model"), keeps text autoregressive and discrete but replaces the image branch's cross-entropy loss with a diffusion loss inside the same transformer, using causal attention for text positions and bidirectional attention within each image's patch block — splitting the difference between full discretization and continuous generation while still sharing one set of weights end to end.
Why the wiring decides the capability set
The common misconception is that "multimodal" is a single capability — the model can look at pictures — and that any multimodal model is roughly interchangeable with any other. It isn't, and the gap is architectural, not a matter of scale or training data. A cross-attention adapter model like Flamingo has a frozen LM output head that only ever emits text-token logits; there is no mechanism by which cross-attending on visual latents could cause that head to emit a pixel. These models are structurally understanding-only — they can describe, answer questions about, and reason over images and video, but generating an image is outside what their output layer can express at all. An early-fusion model like Chameleon has no such asymmetry: its output softmax spans the same combined vocabulary — text tokens and image codes together — that its input embedding table does, so producing an image is nothing more than the model choosing to sample from image-code positions instead of text positions for a stretch of the sequence, then a decoder reassembles those codes back into pixels. The architecture, not the training recipe, is what puts image generation on or off the table. When you see a model advertised as capable of both understanding and generating images natively in one pass, the safe architectural inference is early fusion (or a Transfusion-style hybrid); when a model accepts images but only ever outputs text, a frozen-backbone adapter is the more likely design.
What each route costs in production
The two routes trade training cost against inference cost differently. Flamingo-style adapters are cheap to train precisely because the backbone is frozen — you're only updating the resampler and gated cross-attention weights, a small fraction of total parameters, while the expensively pretrained LM's language capability is inherited for free and can't be damaged by multimodal fine-tuning. The price shows up at serving time in context management: even after 32× compression, every image or video clip in a conversation still adds 64 tokens' worth of keys and values that every subsequent gated cross-attention layer must hold and attend over, and a long multi-turn conversation with several images accumulates this linearly. Early-fusion models pay the opposite way. Training is a single unified objective with no frozen components to lean on, requiring the stability engineering described above, but a trained early-fusion model's inference cost for understanding tasks is just standard transformer decoding over a slightly longer sequence — no separate cross-attention subsystem to maintain. Generation flips this again: producing a 512×512 image at the same 16× downsampling and K=8192 codebook needs a 32×32 = 1024-token grid, four times the 256 tokens a 256×256 image needs, and because early-fusion image tokens are sampled autoregressively one at a time, that's roughly four times as many sequential decode steps — a real latency cost that a parallel ViT forward pass in the cross-attention route simply doesn't incur, because that route was never asked to generate pixels in the first place.
Active recall
Attempt these before reading the worked answers.
- In the VQ-VAE worked example, the codebook size doubles from 8192 to 16384 while image resolution and the 16× downsampling factor stay the same. What happens to (a) bits per token, (b) total tokenized bits per image, (c) the combined vocabulary size if the text vocabulary is 32,000, and (d) the number of image tokens per image?
- Why does concatenating a frozen CLIP image embedding in front of a frozen GPT-style LM's token embeddings tend to perform badly without further training, even though the embeddings are supposedly in a "shared" space?
- Using the cross-attention cost formula
4 × T × N_kv × d, compute the multiply-adds for a gated cross-attention layer with T = 4096 (doubled text context), R = 64, d = 2048. Compare to the original T = 2048 case (1,073,741,824 MACs). What does this reveal about what resampling does and doesn't control? - A Chameleon-style model uses the same tokenizer (16× downsampling, K = 8192) to generate a 512×512 image instead of a 256×256 one. How many image tokens must it sample, how does that compare to the 256×256 case, and what's the serving consequence?
- If the Perceiver Resampler's input changes from 8 video frames to 16, does its output token count change? What property guarantees your answer, and what happens to the downstream gated cross-attention cost (T = 2048, d = 2048)?
- Why does Chameleon need extra stability engineering (QK-normalization) that Flamingo does not, given what each architecture actually trains end-to-end?
Worked answers.
1. (a) log2(16384) = 14 bits per token, up from 13. (b) Total tokenized bits = 256 × 14 = 3584 bits (448 bytes), up from 3328 bits (416 bytes) — a 7.7% increase, from the 256 tokens each costing one extra bit. (c) Combined vocabulary = 32,000 + 16,384 = 48,384, up from 40,192. (d) The number of image tokens per image is unchanged at 256 — codebook size only changes how many bits index each code; the token count is fixed by the spatial downsampling factor, an independent design knob. This is the trap: it's tempting to assume a bigger codebook means more tokens, but codebook size and token count are set by two different parts of the tokenizer.
2. Contrastively trained encoders like CLIP place matching image and text embeddings close by cosine similarity but not at the same point — the two modalities occupy geometrically distinct regions of the space (the modality gap, Liang et al. 2022). A frozen LM was trained only on text-token embeddings; it has never seen vectors from the image region and has no learned behavior for them. Without joint fine-tuning — via a cross-attention adapter or joint pretraining — the LM effectively treats the pasted-in image vector as out-of-distribution noise rather than as informative content.
3. 4 × 4096 × 64 × 2048 = 2,147,483,648 MACs — exactly double the T = 2048 case, since only T changed and the formula is linear in T. This shows resampling controls only the image-side dimension (N_kv); it does nothing to the text-side cost. Doubling the text context still doubles cross-attention cost regardless of how aggressively the visual tokens are compressed.
4. 512 / 16 = 32, so a 32×32 = 1024-token grid, four times the 256 tokens needed for 256×256 (matching the area scaling (512/256)^2 = 4). Since early-fusion image generation samples tokens autoregressively one at a time, this means roughly four times as many sequential decode steps — a direct latency cost at serving time that a route using a parallel vision-encoder forward pass never pays, because it was never built to generate pixels.
5. No — the Resampler's defining property is that its learned query vectors always produce a fixed R = 64 output latents regardless of how many input keys/values (N) they cross-attend over. Going from 8 to 16 frames only increases the resampler's own internal attention cost (it must attend over more input tokens once); the downstream gated cross-attention cost stays exactly 4 × 2048 × 64 × 2048 = 1,073,741,824 MACs, unchanged, because everything past the resampler only ever sees 64 latents.
6. Flamingo freezes both the vision encoder and the LM backbone and trains only the new resampler and gated cross-attention weights, with the gate initialized near zero so the new modules start as a near-identity perturbation on a well-conditioned pretrained model. Chameleon has no frozen anchor — it trains one transformer end to end over a mixed vocabulary where image-code and text-token embeddings can develop very different norm and gradient statistics, and without extra normalization those statistics diverge and destabilize training. Unifying modalities into one set of shared weights inherits a harder joint-optimization problem that a frozen-backbone adapter sidesteps by construction.
Think About It
Think about this: How would you explain multimodal foundation models: architecture and capabilities to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where multimodal foundation models: architecture and capabilities is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting multimodal foundation models: architecture and capabilities to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind multimodal foundation models: architecture and capabilities, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.