Type "a red sports car parked outside a Bengaluru flyover at dusk, cinematic lighting" into DALL-E or Stable Diffusion and within seconds you get a photograph-quality image that was never photographed. Nobody drew that car. No database of Bengaluru flyover photos was searched. A sentence, a sequence of words with no pixels in it anywhere, produced a 512x512 grid of RGB values that a human eye reads as a specific, coherent scene.
That sentence-to-pixel jump is the actual engineering problem this chapter solves. You already know two of the three pieces it is built from. From the deep learning unit, you know that a convolutional network turns an image into a stack of feature vectors. From the NLP unit, you know that a transformer turns a sentence into a sequence of token vectors. Neither vector space has anything to do with the other by default: an image encoder and a text encoder trained separately would learn completely different, mutually unreadable coordinate systems, the way Hindi and Japanese assign unrelated sound patterns to the same idea. The missing third piece is a shared vocabulary between "the sentence means this" and "the picture looks like this." Everything below, CLIP, the two rival image-generation architectures, cross-attention, and classifier-free guidance, exists to build and exploit that shared vocabulary.
Figure: the text-to-image pipeline
Keep this map in view. Each labeled block is explained, with a worked numerical example, in the section that follows it.
Two vector spaces that do not speak the same language
CLIP (Contrastive Language-Image Pre-training, Radford et al., 2021) is the component that builds the shared vocabulary. It trains two encoders at once, an image encoder (a vision transformer or CNN) and a text encoder (a transformer), on roughly 400 million (image, caption) pairs scraped from the internet. It never learns to generate anything. Its only job is to place a matching image and caption near each other in a shared embedding space, and to place every non-matching pair far apart.
The mechanism is contrastive learning. Take a batch of N image-text pairs. Run every image through the image encoder and every text through the text encoder, then L2-normalize every output vector so only its direction matters, not its length, which is exactly what makes a dot product equal a cosine similarity. Compute the full N-by-N matrix of similarities between every image and every text in the batch, multiply by a learned temperature (logit scale), and run a softmax cross-entropy loss along each row (does image i pick out its own caption among all N captions?) and along each column (does caption i pick out its own image?). Average the two directions. Gradient descent on that loss pulls matching pairs' vectors together and pushes every other pair in the batch apart, purely from the co-occurrence signal, no manual labels required beyond "this caption was posted with this image."
Work it through on a toy batch of N = 3 pairs (cat, dog, car) with 2-dimensional embeddings, small enough to compute by hand. Raw encoder outputs, already picked to have length 5 so normalizing is just dividing by 5:
I1=(3,4) -> n(I1)=(0.6, 0.8) T1=(4,3) -> n(T1)=(0.8, 0.6) [cat]
I2=(-4,3)-> n(I2)=(-0.8,0.6) T2=(-3,4) -> n(T2)=(-0.6,0.8) [dog]
I3=(0,-5)-> n(I3)=(0,-1) T3=(3,-4) -> n(T3)=(0.6,-0.8) [car]
Dot every normalized image vector with every normalized text vector to get the raw similarity matrix S, then scale by a temperature of 10 (a stand-in for CLIP's learned logit scale):
S (raw cosine sim) S x 10 (scaled logits)
T1 T2 T3 T1 T2 T3
I1 [ 0.96 0.28 -0.28 ] I1 [ 9.6 2.8 -2.8 ]
I2 [ -0.28 0.96 -0.96 ] I2 [ -2.8 9.6 -9.6 ]
I3 [ -0.60 -0.80 0.80 ] I3 [ -6.0 -8.0 8.0 ]
Row I1's softmax: e^9.6 = 14766.2, e^2.8 = 16.44, e^-2.8 = 0.061, sum = 14782.7, so P(T1 | I1) = 14766.2 / 14782.7 = 0.99888, and the cross-entropy loss for that row is -ln(0.99888) = 0.00112. Rows I2 and I3 work out to losses of about 0.000004 and 0.0000009 the same way (their correct entries, 9.6 and 8.0, dominate their rows even more heavily). Doing the same computation column-wise (does each text pick out its matching image?) gives a nearly identical set of tiny losses. Average all six numbers and the batch loss is about 0.0004, close to zero, because this toy embedding space already separates the three categories almost perfectly.
Contrast that with what the loss looks like before any training, when encoder weights are random and unrelated vectors have cosine similarity near 0 in every cell of S. A uniform distribution over N = 3 candidates gives every candidate probability 1/3, and the loss is -ln(1/3) = ln(3) = 1.099. That single number, ln(N), is the loss ceiling contrastive training starts from and drives toward the near-zero value above. The gap between 1.099 and 0.0004 is what "the model learned to align text and images" looks like as a number.
Once trained, CLIP's text tower is frozen and reused purely as a feature extractor: no image-generation model computes CLIP embeddings for the picture it hasn't drawn yet, only for the prompt it was given (and, in DALL-E 2's case, for real training images during training). What DALL-E and Stable Diffusion inherit from CLIP is a text embedding sequence that already lives in a space where "red," "sports," and "car" sit near the visual concepts they describe. That embedding sequence is the "e1..e5" row in the diagram above, and it is what everything downstream conditions on.
Two rival designs for turning that vector into pixels
Having a good text embedding does not by itself specify how to produce an image from it. The original DALL-E (Ramesh et al., 2021) and Stable Diffusion answer that question in fundamentally different ways.
DALL-E 1 treats image generation as sequence prediction, the same operation you already know from GPT-style language models, just extended to a second, discrete "pixel vocabulary." A discrete variational autoencoder (dVAE) is trained separately to compress a 256x256x3 image into a 32x32 grid of tokens, each token an index into a fixed codebook of 8192 learned visual "patches." The text prompt (up to 256 BPE tokens) and the image's 1024 tokens are concatenated into one sequence, and a 12-billion-parameter transformer is trained autoregressively: predict image token k from every text token and every image token before it, exactly like predicting the next word in a sentence. Generation runs the same way GPT generates text, one token at a time, sampling image token 1, then image token 2 conditioned on token 1, and so on for all 1024 tokens, before the dVAE decoder turns the finished token grid back into pixels.
Why tokenize the image at all, rather than autoregress directly over raw pixel intensities? Self-attention costs O(n^2) in sequence length, knowledge you already have from asymptotic analysis. A 256x256 image has 65,536 pixel positions; modeling those directly would need attention over roughly 65536^2 = 4.29 x 10^9 position pairs per layer. Compressing to a 1024-token image grid plus 256 text tokens gives a sequence length of 1280, and 1280^2 = 1.64 x 10^6 pairs, a reduction of about 2,600x. That factor is the entire reason DALL-E 1 needed the dVAE step before a transformer could touch the problem at all. Separately, the token grid itself is a real compression: 1024 tokens from an 8192-entry codebook carry 1024 x 13 = 13,312 bits (8192 = 2^13), versus 256x256x3x8 = 1,572,864 bits for the raw 8-bit pixels, a compression of about 118x.
Stable Diffusion abandons the autoregressive, one-token-at-a-time approach entirely and instead runs a diffusion process, the same forward-noise-then-learn-to-reverse-it idea from the DDPM chapter, but conditioned on the text embedding at every step. Rather than committing to pixels or tokens one at a time in a fixed left-to-right order, it starts from pure Gaussian noise occupying every position in the image at once and repeatedly, jointly, nudges every position toward a more coherent image, all positions updating together on every pass.
Latent diffusion: denoising where it is cheap
Running that denoising loop directly on 512x512x3 pixels, dozens of times, through a large U-Net, is expensive: that is 786,432 numbers per image, per step. Stable Diffusion (Rombach et al., 2022) instead trains a VAE once, ahead of time, whose encoder compresses a 512x512x3 image down to a 64x64x4 latent tensor, and whose decoder reconstructs pixels from that latent. Every denoising step then runs on the 64x64x4 latent, not the 512x512x3 image, and only the very last step's output is passed through the decoder to produce the final picture.
The size of that saving is exact, not approximate: 512x512x3 = 786,432 values, 64x64x4 = 16,384 values, and 786432 / 16384 = 48. Spatially, the compression is 8x per side (512/64 = 8), so 8x8 = 64 times fewer spatial positions for the U-Net's convolutions and attention layers to touch; that 64x reduction is partly offset by going from 3 channels to 4, giving a net 64 x (3/4) = 48x reduction in raw numbers the network processes at every one of its ~20 to 50 forward passes. That 48x is the entire reason "latent diffusion" made high-resolution text-to-image generation practical on consumer GPUs instead of only in a well-funded lab.
The denoising loop itself, briefly: training adds Gaussian noise to a real image's latent in T increasing steps until step T is pure noise, and trains the U-Net to predict the noise that was added at every step so it can be subtracted back out. Sampling runs that in reverse, starting from pure noise at step T and repeatedly calling the U-Net to predict and remove a bit of noise, arriving at a clean latent by step 0. What makes this a text-to-image model rather than a plain image generator is that the U-Net does not just see the noisy latent at each step, it also sees the text embedding sequence, through cross-attention, at every single one of those steps.
Cross-attention: how the prompt steers every step
Cross-attention is the mechanism that actually injects "red sports car" into the denoising process. At every resolution level inside the U-Net, the current latent's spatial features generate query vectors Q, and the frozen text embedding sequence e1..e5 generates key and value vectors K and V (via learned linear projections, one set of projections per attention layer). Standard scaled dot-product attention then runs:
Attention(Q, K, V) = softmax( Q K^T / sqrt(d_k) ) V
Trace it for one spatial position with query Q = (2, 0) against two toy text tokens, "sky" and "grass," with keys K_sky = (0, 3), K_grass = (3, 0), values V_sky = (0, 10), V_grass = (10, 0), and dimension d_k = 2 so the scale factor is sqrt(2) = 1.4142:
dot(Q, K_sky) = 2*0 + 0*3 = 0 scaled: 0 / 1.4142 = 0
dot(Q, K_grass) = 2*3 + 0*0 = 6 scaled: 6 / 1.4142 = 4.2426
softmax: e^0 = 1, e^4.2426 = 69.6, sum = 70.6
weight_sky = 1 / 70.6 = 0.0142
weight_grass = 69.6 / 70.6 = 0.9858
output = 0.0142 * (0,10) + 0.9858 * (10,0) = (9.86, 0.14)
The query at this spatial position happened to point in the same direction as K_grass (their dot product, 6, is far larger than Q's dot product with K_sky, 0), so softmax hands this position almost the entire "grass" value vector and almost none of the "sky" one. A different spatial position whose current features generate a query aligned with K_sky instead would receive the opposite mixture. This is the actual mechanism by which different regions of the same noisy latent get pulled toward different words in the same prompt, and it is recomputed, freshly, at every layer of the U-Net, on every one of the ~20 to 50 denoising steps, because the query changes every time the latent changes but the text's keys and values, computed once from the frozen CLIP embedding, never do.
Classifier-free guidance: turning up the prompt's volume
Cross-attention makes the text embedding visible to the network. Classifier-free guidance (Ho and Salimans, 2022) is the separate, sampling-time trick that controls how strongly the network is allowed to act on it. During training, the text condition is randomly dropped (replaced with an empty prompt) some fraction of the time, so the same U-Net learns to predict noise both with the prompt, eps_cond, and without it, eps_uncond. At sampling time, both predictions are computed at every step, and combined:
def classifier_free_guidance(eps_uncond, eps_cond, w):
return eps_uncond + w * (eps_cond - eps_uncond)
eps_uncond = 0.20 # noise predicted with no prompt
eps_cond = 0.35 # noise predicted with "a red sports car"
for w in (0, 1, 7.5):
print(w, classifier_free_guidance(eps_uncond, eps_cond, w))
# 0 0.2
# 1 0.35
# 7.5 1.325
At w = 0 the guided prediction is just eps_uncond, the prompt has no effect at all. At w = 1 it is exactly eps_cond, the model's raw conditional prediction, no exaggeration. Stable Diffusion's usual default of w = 7.5 pushes the guided prediction to 0.20 + 7.5 x 0.15 = 1.325, nearly four times the magnitude of the plain conditional prediction, by extrapolating past it in the direction the prompt moved it. Since the sampler's next latent is computed by subtracting a term proportional to this guided epsilon, a larger w produces a larger corrective step toward whatever the prompt pushed the prediction toward at that position, which is exactly why raising the guidance scale in a Stable Diffusion interface makes output more literally match the prompt, and why pushing it too high causes oversaturated colors and warped anatomy: the step size overshoots.
Common misconception: "the prompt is read once, at the start"
The most common wrong mental model of this pipeline treats the text prompt the way a random seed is treated: consulted once, at t = T, to decide roughly what the picture will be about, after which the model just "cleans up" noise on its own. That is not what happens, and the cross-attention trace above shows why it cannot be what happens. The latent tensor is a completely different set of numbers at every one of the ~20 to 50 steps; a signal injected only once, before step 1, would have no channel through which to influence step 30's decision about a specific spatial region, because nothing about step 30's input remembers where that one-time injection went. Instead, the exact same frozen K and V vectors, derived from the exact same CLIP text embedding, are re-offered to every cross-attention layer at every step, and the query the current latent generates at each step decides fresh, at that step, which words are relevant to which regions. The prompt is not a seed. It is a reference document the network re-reads from scratch at every single denoising step, which is also why editing a prompt mid-generation (as some interfaces allow) changes the image from that step onward rather than only at the start.
Active recall
Attempt each question before reading its answer.
1. With eps_uncond = 0.10, eps_cond = 0.40, and guidance scale w = 3, compute the classifier-free-guided noise prediction.
2. Stable Diffusion's VAE compresses a 512x512x3 image to a 64x64x4 latent. Show the exact numeric compression factor, and explain in one sentence why denoising in that latent space is cheaper per step than denoising in pixel space.
3. Why can the text embedding not simply be concatenated once with the initial noisy latent, instead of being re-supplied via cross-attention at every step?
4. A spatial position generates query Q = (2, 0). Text tokens "sky" and "grass" have keys K_sky = (0, 3), K_grass = (3, 0) and values V_sky = (0, 10), V_grass = (10, 0), with d_k = 2. Compute the cross-attention output at that position.
5. Name one concrete consequence, for how the final image is produced, of DALL-E 1 being autoregressive over discrete tokens while Stable Diffusion is a latent diffusion model.
6. For a CLIP training batch of N = 8, what is the expected image-to-text loss at random initialization, and how many times smaller is a trained loss of 0.02?
Answers
1. eps_hat = 0.10 + 3 x (0.40 - 0.10) = 0.10 + 0.90 = 1.00.
2. 786432 / 16384 = 48, an exact 48x reduction. Spatially that is 8x8 = 64 times fewer positions (512/64 = 8 per side), partly offset by 4 channels instead of 3 (64 x 3/4 = 48). Every convolution and attention computation in the U-Net scales with the number of positions it processes, so 48x fewer numbers per step makes each of the ~20 to 50 forward passes proportionally cheaper.
3. Because the latent is a different tensor at every step, and a single injection at t = T is only visible to the very first computation; by later steps nothing in the current latent retains a direct link back to that one-time signal, so the network would have no way to re-check its current output against the prompt. Cross-attention instead exposes the same frozen text keys and values at every layer of every step, so the model consults the exact, unmodified prompt no matter how far along denoising it is.
4. dot(Q,K_sky) = 0, dot(Q,K_grass) = 6; scaled by sqrt(2): 0 and 4.2426. Softmax: e^0=1, e^4.2426=69.6, sum=70.6, weights 0.0142 and 0.9858. Output = 0.0142x(0,10) + 0.9858x(10,0) = (9.86, 0.14), almost entirely the "grass" value, because Q aligned with K_grass.
5. DALL-E 1 must generate its 1024 image tokens strictly one at a time, each conditioned only on tokens already fixed before it, like GPT generating text; an early wrong token permanently biases everything generated after it and can never be revisited. Stable Diffusion updates every one of its latent's 4,096 spatial positions together at every step, so a region that is wrong after step 5 can still be corrected by step 20, because no position is committed to a final value until the last denoising step.
6. A uniform distribution over N = 8 gives probability 1/8 to the correct pair, so the loss is -ln(1/8) = ln(8) = 2.079. Divided by a trained loss of 0.02, that is 2.079 / 0.02 = 104, about 104 times larger before training than after.
Think About It
Think about this: How would you explain text-to-image: how dall-e and stable diffusion work to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where text-to-image: how dall-e and stable diffusion work is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting text-to-image: how dall-e and stable diffusion work to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind text-to-image: how dall-e and stable diffusion work, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.