A denoiser trained backwards
Point a phone camera at a dim room and the sensor captures too few photons per pixel to pin down a true colour — the result is grainy, speckled noise sitting on top of a barely visible scene. Modern computational photography (Google's Night Sight pipeline is the well-documented example) fixes this with a neural network trained on millions of pairs of clean and artificially-noised photos: show the network a noisy image, ask it to predict the noise, subtract that prediction, and the underlying scene reappears. The network never "imagines" a photo from scratch. It only ever does one narrow job — look at a slightly corrupted image and estimate exactly what corruption was added.
Diffusion models are built on a deliberately strange question: what happens if you keep asking that same denoiser to do its job, over and over, starting not from a photo with a little noise on it, but from an image that is pure noise — random static with no scene underneath at all? Each denoising step removes a little of the noise and reveals a little more structure. Run enough steps, and a coherent image emerges from static that never contained one to begin with. That is the entire mechanism behind DALL-E, Midjourney, and Stable Diffusion. There is no image "hidden" in the noise waiting to be uncovered; the network has learned, from millions of real photographs, what small nudges in pixel-space consistently push random noise toward looking like the kind of images it was trained on.
The generative modelling problem, stated precisely
The task a generative model solves is: given a training set of images assumed to be samples from some unknown distribution pdata(x), learn a way to draw new samples from that same distribution without ever having an explicit formula for it. A variational autoencoder learns this by compressing images into a latent code and decoding back; a GAN learns it by pitting a generator against a discriminator in a single forward pass. Diffusion models take a third route, one first proposed by Sohl-Dickstein, Weiss, Maheswaranathan and Ganguli in "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" (ICML, 2015) and made practical five years later by Ho, Jain and Abbeel in "Denoising Diffusion Probabilistic Models" (NeurIPS, 2020, the paper usually abbreviated DDPM): define a fixed, mathematically simple process that gradually destroys a real image into pure noise, then train a neural network to run that process backwards.
The forward process: a fixed recipe for destroying an image
The forward process adds a small amount of Gaussian noise at each of T timesteps, according to a fixed noise schedule beta_1, beta_2, ..., beta_T (each a small positive number, typically increasing with t). One forward step is defined as
q(x_t | x_t-1) = N(x_t ; sqrt(1 - beta_t) · x_t-1, beta_t · I)
In words: to get x_t, shrink the previous image slightly (multiply by sqrt(1 - beta_t)) and add fresh Gaussian noise scaled by sqrt(beta_t). Writing alpha_t = 1 - beta_t, one step is x_t = sqrt(alpha_t) · x_t-1 + sqrt(beta_t) · epsilon_t, with epsilon_t drawn fresh from a standard normal distribution at every step. The shrink-then-add-noise design is not arbitrary: it keeps the total variance of x_t exactly equal to 1 at every step, provided x_0 itself has unit variance. That is why running the process for enough steps drives x_t to a distribution indistinguishable from pure N(0, I) noise regardless of what the original image was — the schedule was built so that outcome is guaranteed, not just observed.
Because each step is Gaussian and the steps compose, there is a closed-form shortcut that skips straight from x_0 to any x_t without simulating every intermediate step. Defining the cumulative product alpha_bar_t = alpha_1 · alpha_2 · ... · alpha_t, one can show that
x_t = sqrt(alpha_bar_t) · x_0 + sqrt(1 - alpha_bar_t) · epsilon, epsilon ~ N(0, I)
This single equation is the workhorse of diffusion training: given any clean image x_0, a target timestep t, and one sampled noise vector, you can jump directly to x_t in one line, with no loop.
Worked example: four steps of forward diffusion, by hand
Real schedules use T on the order of 1000 steps with beta_1 = 0.0001 rising to beta_T = 0.02 (the exact linear schedule used in Ho et al., 2020) — too fine-grained to trace by hand. Here is a toy four-step schedule, chosen purely so every number can be checked on paper, with a single scalar "pixel" x_0 = 1.0 standing in for an image:
beta_1 = 0.1, beta_2 = 0.2, beta_3 = 0.3, beta_4 = 0.4, so alpha_1 = 0.9, alpha_2 = 0.8, alpha_3 = 0.7, alpha_4 = 0.6.
Fix four noise draws for reproducibility: epsilon_1 = 0.5, epsilon_2 = -0.3, epsilon_3 = 0.8, epsilon_4 = -0.6. Applying x_t = sqrt(alpha_t) · x_t-1 + sqrt(beta_t) · epsilon_t recursively:
x_1 = sqrt(0.9)(1.0) + sqrt(0.1)(0.5) = 0.9487 + 0.1581 = 1.1068
x_2 = sqrt(0.8)(1.1068) + sqrt(0.2)(-0.3) = 0.9900 - 0.1342 = 0.8558
x_3 = sqrt(0.7)(0.8558) + sqrt(0.3)(0.8) = 0.7160 + 0.4382 = 1.1542
x_4 = sqrt(0.6)(1.1542) + sqrt(0.4)(-0.6) = 0.8941 - 0.3795 = 0.5146
Now check this against the closed-form shortcut. alpha_bar_4 = 0.9 × 0.8 × 0.7 × 0.6 = 0.3024, so sqrt(alpha_bar_4) = 0.5499. But the closed form uses a single equivalent noise draw, not epsilon_4 alone — expanding the recursion out shows that x_4 equals sqrt(alpha_bar_4)·x_0 plus a weighted sum of all four original epsilon draws, with weights c_1 = sqrt(alpha_4·alpha_3·alpha_2·beta_1), c_2 = sqrt(alpha_4·alpha_3·beta_2), c_3 = sqrt(alpha_4·beta_3), c_4 = sqrt(beta_4). As a sanity check, these weights must satisfy c_1² + c_2² + c_3² + c_4² = 1 - alpha_bar_4 (variances of independent Gaussians add): 0.0336 + 0.084 + 0.18 + 0.4 = 0.6976, and 1 - 0.3024 = 0.6976 — confirmed. Plugging the same four epsilon values into the weighted sum: 0.1833(0.5) + 0.2898(-0.3) + 0.4243(0.8) + 0.6325(-0.6) = 0.0917 - 0.0869 + 0.3394 - 0.3795 = -0.0353. Adding the signal term: 0.5499 + (-0.0353) = 0.5146 — exactly matching the recursive computation. This is not a coincidence; it is the algebraic identity that makes the closed-form shortcut valid, and both routes to x_4 must agree to full floating-point precision. The following code reproduces both computations:
import numpy as np
betas = np.array([0.1, 0.2, 0.3, 0.4]) # beta_1 .. beta_4
alphas = 1.0 - betas # alpha_1 .. alpha_4
alpha_bar = np.cumprod(alphas) # cumulative products
x0 = 1.0
eps = np.array([0.5, -0.3, 0.8, -0.6]) # fixed noise draws
# recursive forward process
x = x0
trajectory = []
for t in range(4):
x = np.sqrt(alphas[t]) * x + np.sqrt(betas[t]) * eps[t]
trajectory.append(x)
print([round(v, 4) for v in trajectory])
# [1.1068, 0.8558, 1.1542, 0.5146]
# closed-form shortcut, checked against the recursive x_4
coeffs = [
np.sqrt(alphas[3] * alphas[2] * alphas[1] * betas[0]), # c1
np.sqrt(alphas[3] * alphas[2] * betas[1]), # c2
np.sqrt(alphas[3] * betas[2]), # c3
np.sqrt(betas[3]), # c4
]
x4_closed = np.sqrt(alpha_bar[3]) * x0 + sum(c * e for c, e in zip(coeffs, eps))
print(round(x4_closed, 4), round(trajectory[-1], 4))
# 0.5146 0.5146
The reverse process: what the network is actually trained to predict
Generating an image means running this process backwards: start from x_T ~ N(0, I) and repeatedly sample x_t-1 given x_t. The exact reverse conditional q(x_t-1 | x_t, x_0) has a known closed form, but it requires knowing x_0 — the very thing generation is trying to produce. So a neural network p_theta(x_t-1 | x_t) is trained to approximate this reverse distribution using only x_t and t, marginalising out the unknown x_0.
Ho et al. (2020) show that training simplifies dramatically if the network is asked to predict the noise epsilon that was added, rather than predicting the clean image x_0 directly. Since x_0 = (x_t - sqrt(1 - alpha_bar_t) · epsilon) / sqrt(alpha_bar_t), predicting epsilon and predicting x_0 are mathematically interchangeable — but epsilon has unit variance at every timestep, while x_0 prediction forces the network to "invent" nearly the entire image out of almost no signal at large t, producing wildly unstable gradients. The resulting training objective, called L_simple, is just a mean-squared error between true and predicted noise:
L_simple = E[ ||epsilon - epsilon_theta(x_t, t)||² ]
One training step: sample a real image, sample a random timestep t, sample noise, jump straight to x_t using the closed-form shortcut, and ask the network to recover the noise that was used.
import numpy as np
# unet(x, t): the noise-prediction network (assumed helper, not shown)
# sample_real_image(): draws a training image x_0 (assumed helper, not shown)
# optimizer.step(loss): applies the gradient update (assumed helper, not shown)
def training_step(unet, betas, optimizer):
alphas = 1.0 - betas
alpha_bar = np.cumprod(alphas)
T = len(betas)
x0 = sample_real_image()
t = np.random.randint(0, T)
eps = np.random.normal(size=x0.shape)
x_t = np.sqrt(alpha_bar[t]) * x0 + np.sqrt(1.0 - alpha_bar[t]) * eps
eps_pred = unet(x_t, t)
loss = np.mean((eps - eps_pred) ** 2) # L_simple
optimizer.step(loss)
return loss
Sampling: turning noise into an image
At inference time the network is run in the opposite direction. Starting from pure noise x_T, at every step it predicts the noise in the current x_t and uses that prediction to compute the mean of x_t-1, then adds a controlled amount of fresh noise back in (except on the very last step) so that sampling remains stochastic rather than collapsing to a single deterministic path. This is DDPM's Algorithm 2:
import numpy as np
# eps_theta(x, t): the trained noise-prediction network (assumed helper, not shown)
def ddpm_sample(betas, eps_theta, shape=(1,)):
T = len(betas)
alphas = 1.0 - betas
alpha_bar = np.cumprod(alphas)
x = np.random.normal(size=shape) # x_T ~ N(0, I)
for t in reversed(range(T)):
eps_pred = eps_theta(x, t)
coef = betas[t] / np.sqrt(1.0 - alpha_bar[t])
mean = (x - coef * eps_pred) / np.sqrt(alphas[t])
if t > 0:
z = np.random.normal(size=shape)
sigma_t = np.sqrt(betas[t])
x = mean + sigma_t * z
else:
x = mean # no noise added on the final step
return x # x_0: the generated sample
Running a full T = 1000 network evaluations per image is slow, which is why Song, Meng and Ermon's DDIM paper ("Denoising Diffusion Implicit Models," ICLR 2021) reformulates sampling as a non-Markovian process that skips most steps — 20 to 50 evaluations instead of 1000, at a small cost in sample diversity. Production text-to-image tools almost always use a DDIM-style sampler for exactly this reason: it is the difference between an image taking a second to generate versus a minute.
Architecture: why a U-Net, and how a text prompt gets in
epsilon_theta(x_t, t) is almost always implemented as a U-Net, an architecture Ronneberger, Fischer and Brox introduced for biomedical image segmentation (MICCAI, 2015) and which diffusion models adopted essentially unchanged in spirit. The encoder path repeatedly downsamples the input while increasing channel count, compressing spatial detail into increasingly abstract, low-resolution feature maps. A bottleneck at the lowest resolution applies self-attention — letting every spatial location influence every other one, which plain convolution cannot do at long range, and which matters for global coherence (the two eyes of a generated face staying symmetric, lighting staying consistent across the frame). The decoder path then upsamples back to full resolution, and at each stage it concatenates in the matching-resolution feature map from the encoder via a skip connection, exactly as in Ronneberger et al.'s original design — this is what lets the network recover fine spatial detail that pure downsampling would otherwise discard.
The single shared network needs to know which timestep it is operating at, since the right denoising correction is very different at t near T (mostly noise) versus t near 0 (mostly signal). This is solved with a sinusoidal timestep embedding — the same positional-encoding idea used in transformers — passed through a small MLP and injected into every block via a scale-and-shift (FiLM-style) modulation of that block's normalized activations.
Text conditioning, the mechanism that turns a prompt into a specific image, is added through cross-attention layers inserted into the U-Net's blocks: the image features supply the attention queries, while a frozen text encoder's output (CLIP or T5, depending on the model) supplies the keys and values. This is precisely how "a snow leopard at dawn" steers what noise the network predicts to remove at every step — described explicitly in Rombach, Blattmann, Lorenz, Esser and Ommer's "High-Resolution Image Synthesis with Latent Diffusion Models" (CVPR, 2022), the paper behind Stable Diffusion.
That same paper makes one further architectural decision worth naming precisely: it does not run the U-Net on raw pixels at all. A separately trained VAE first compresses a 512×512×3 image (786,432 numbers) down to a 64×64×4 latent tensor (16,384 numbers) — an exact 48-fold reduction — and every one of the T diffusion steps operates on that compressed latent, with the VAE decoder invoked only once, at the very end, to turn the final denoised latent back into pixels. This is why Stable Diffusion was trainable on modest hardware while earlier pixel-space diffusion systems needed far larger compute budgets; running attention and convolution over 16,384 values per image is a different cost class than running them over 786,432.
Diagram: forward/reverse diffusion and the denoising network
Common misconception: "diffusion models denoise pixels"
The single most common misunderstanding is that a model like Stable Diffusion runs its T denoising steps directly on the image's pixel grid — generating and refining a full-resolution picture at every one of the, say, 50 sampling steps. It does not. As described above, Rombach et al. (2022) train a VAE that compresses a 512×512×3 image into a 64×64×4 latent tensor first, and every diffusion step — forward during training, reverse during sampling — happens entirely inside that compressed 64×64×4 space. Pixels only exist at the very beginning (when the VAE encoder compresses the training image) and the very end (when the VAE decoder reconstructs the final latent back into a picture). This is not a minor implementation detail: it is the specific innovation that made large-scale text-to-image diffusion affordable to train, and it is why the technique in that paper's title is "latent diffusion," not "pixel diffusion." Not every diffusion model works this way — Google's Imagen (Saharia et al., 2022) does diffuse directly in pixel space, but only at a small base resolution (64×64), then uses a cascade of separate super-resolution diffusion models to upscale, which is a different way of avoiding the cost of full-resolution pixel-space diffusion rather than a rebuttal of the point. The general principle a student should take away is that "diffusion" describes an iterative noising-and-denoising procedure, and that procedure can be applied to any fixed-size vector representation of the data — pixels, a compressed latent, or, in other applications entirely, molecular coordinates or audio waveforms.
Guidance: steering generation toward the prompt
A U-Net trained the way described above will happily generate plausible images unconditionally, but text-to-image systems need the output to match a specific prompt, and matching it loosely is not enough for production quality. Ho and Salimans introduced classifier-free guidance ("Classifier-Free Diffusion Guidance," NeurIPS 2021 Deep Generative Models workshop) to solve this without needing a separate classifier network. During training, the text conditioning is randomly dropped some fraction of the time, so the same network learns both a conditional noise prediction epsilon_theta(x_t, t, prompt) and an unconditional one epsilon_theta(x_t, t, empty). At sampling time, both are computed at every step and combined:
epsilon_hat = epsilon_uncond + w · (epsilon_cond - epsilon_uncond)
where w is the guidance scale. Suppose at some step the network predicts epsilon_uncond = 0.20 and epsilon_cond = -0.10 for a given position, and w = 7.5 (a typical Stable Diffusion default): epsilon_hat = 0.20 + 7.5 × (-0.10 - 0.20) = 0.20 + 7.5 × (-0.30) = 0.20 - 2.25 = -2.05. Notice this deliberately overshoots the conditional prediction itself (-0.10) by a wide margin — guidance is an extrapolation past what the model actually predicts for the prompt, in the direction away from the unconditional prediction, which is precisely why it sharpens prompt adherence but also why pushing w too high (well past roughly 15) produces oversaturated colours and warped anatomy: the sampling trajectory is being dragged outside the region of latent space the network was ever trained to denoise accurately. Setting w = 1 exactly recovers plain conditional sampling with no extrapolation; w = 0 recovers pure unconditional generation, ignoring the prompt entirely. One production cost worth naming explicitly: computing both epsilon_uncond and epsilon_cond at every sampling step means two network evaluations per step instead of one (in practice the two are usually batched into a single forward pass for efficiency, but the FLOP cost is still double) — guidance is not free, and it is one of the reasons a guided sample takes measurably longer to generate than an unguided one at the same step count.
Active recall
Attempt each question before reading its answer.
1. In the DDPM forward process, why does the variance of x_t stay bounded — never exploding — even though noise is added at every single step, all the way out to t = T?
2. Using the four-step toy schedule from the worked example (beta_1 = 0.1, beta_2 = 0.2, beta_3 = 0.3, beta_4 = 0.4, x_0 = 1.0, epsilon_1..4 = 0.5, -0.3, 0.8, -0.6), suppose beta_2 is changed to 0.3 instead of 0.2. Recompute every quantity in the trajectory that changes as a result — not just x_2.
3. Why must epsilon_theta take the timestep t as an explicit input, rather than training a separate dedicated network for each of the T steps?
4. Stable Diffusion's VAE compresses a 512×512×3 image into a 64×64×4 latent. Derive the exact compression factor yourself, and explain in one sentence why this matters for training cost.
5. Under classifier-free guidance, what does a guidance scale of w = 1 correspond to, and why do production systems typically use w in the 5–15 range rather than w = 1?
Answers
1. Each forward step doesn't just add noise — it also shrinks the incoming signal by sqrt(alpha_t) = sqrt(1 - beta_t) before adding noise scaled by sqrt(beta_t). The schedule is constructed so total variance is conserved: Var(x_t) = alpha_t · Var(x_t-1) + beta_t, and if Var(x_t-1) = 1 this gives Var(x_t) = alpha_t + beta_t = 1 exactly. So variance never grows past 1; instead, as t grows, alpha_bar_t = product of all alpha_s shrinks toward 0, meaning the contribution of the original signal x_0 (scaled by sqrt(alpha_bar_t)) vanishes while the noise contribution (scaled by sqrt(1 - alpha_bar_t)) grows to fill the full unit variance — x_T converges to plain N(0, I) regardless of what x_0 was.
2. With alpha_2 changed to 0.7 (beta_2 = 0.3), x_1 = 1.1068 is unchanged, since it only depends on beta_1. But alpha_bar_2 = 0.9 × 0.7 = 0.63 (was 0.72), alpha_bar_3 = 0.63 × 0.7 = 0.441 (was 0.504), and alpha_bar_4 = 0.441 × 0.6 = 0.2646 (was 0.3024) — every cumulative product from step 2 onward shifts. Recomputing recursively: x_2 = sqrt(0.7)(1.1068) + sqrt(0.3)(-0.3) = 0.9260 - 0.1643 = 0.7617 (was 0.8558); x_3 = sqrt(0.7)(0.7617) + sqrt(0.3)(0.8) = 0.6373 + 0.4382 = 1.0755 (was 1.1542); x_4 = sqrt(0.6)(1.0755) + sqrt(0.4)(-0.6) = 0.8331 - 0.3795 = 0.4536 (was 0.5146). The closed-form coefficients for x_4 also shift wherever alpha_2 appears: c_1 = sqrt(0.6 × 0.7 × 0.7 × 0.1) = 0.1715 and c_2 = sqrt(0.6 × 0.7 × 0.3) = 0.3550 both change, while c_3 = sqrt(0.6 × 0.3) = 0.4243 and c_4 = sqrt(0.4) = 0.6325 are untouched, since neither depends on alpha_2 or beta_2. Re-summing confirms the same 0.4536: 0.5144(1.0) + 0.1715(0.5) + 0.3550(-0.3) + 0.4243(0.8) + 0.6325(-0.6) = 0.5144 + 0.0858 - 0.1065 + 0.3394 - 0.3795 = 0.4536. The full ripple touches alpha_bar_2 through alpha_bar_4, x_2 through x_4, and coefficients c_1 and c_2 — while x_1 and coefficients c_3, c_4 are the non-obvious survivors that stay exactly the same.
3. Training T = 1000 separate networks would multiply the parameter count and training cost by 1000, and — more importantly — it would throw away the fact that denoising at t = 500 and t = 501 are nearly identical problems that should share statistical strength. A single network conditioned on t via a sinusoidal embedding learns a smooth function of noise level, generalises across nearby timesteps, and, crucially, can be evaluated at timesteps spaced differently at inference than during training — exactly what DDIM exploits to sample in 50 steps using a network trained with T = 1000.
4. 512 × 512 × 3 = 786,432 values in the pixel image; 64 × 64 × 4 = 16,384 values in the latent. 786,432 ÷ 16,384 = 48, so the VAE achieves an exact 48-fold reduction in the number of values the diffusion U-Net has to process at every one of its steps. Since attention and convolution costs scale with the number of spatial positions being processed, this reduction is what makes training and running the diffusion model at high resolution computationally tractable on far less hardware than pixel-space diffusion would require.
5. w = 1 means epsilon_hat = epsilon_uncond + 1 × (epsilon_cond - epsilon_uncond) = epsilon_cond — plain conditional sampling, with no extrapolation at all. Production systems push w well above 1 because pure conditional sampling still under-represents the prompt: the network was trained to model the full distribution of images that go with a caption, which includes plenty of low-adherence, generic outputs. Extrapolating in the direction away from the unconditional prediction (w = 5 to 15) pulls the sample further toward the region of latent space specifically associated with the prompt, at the cost of two network evaluations per step and, past roughly w = 15, visibly oversaturated and distorted results as the trajectory is pushed outside what the model was actually trained to denoise well.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind diffusion models: how ai creates images, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.