Image Generation and Variational Autoencoders

A team building an automated diabetic-retinopathy screening tool for rural eye camps — the kind of program run in partnership with hospitals like Aravind Eye Hospital, where a technician photographs a patient's retina and an on-site model flags cases for referral — runs into a familiar wall. Their classifier for the rarest, most severe grade of the disease (proliferative retinopathy) has fewer than 400 labeled images to train on, against tens of thousands for the healthy class. The fix is synthetic data: train a generative model on the 400 real images, then sample new, plausible-looking retinal images of that grade to balance the training set. Two families of model can do this — a Variational Autoencoder (VAE) and a Generative Adversarial Network (GAN) — and a third, diffusion models, is the current state of the art for raw image quality. The team has to pick one, on a laptop with no GPU, for a deployment that has to keep working without a research engineer on call. That choice is what this chapter is about.

You already know how a VAE is trained: the encoder maps an image to a distribution over latent codes, the reparameterization trick lets gradients flow through a sampling step, and the ELBO balances reconstruction against a KL term that pulls the latent posterior toward a standard normal prior. This chapter assumes all of that. What it covers instead is what happens once training is done — how you actually get a new image out of the model, why that image tends to look soft and smoothed-over compared to a GAN or diffusion sample, and how to make the stability-versus-sharpness tradeoff explicit enough to defend a real engineering decision like the one above.

What actually happens when a trained VAE generates an image

The single most common misconception about VAE image generation is that it requires an input image to encode. It doesn't, and understanding why not is the key to understanding generation-time behavior. During training, the encoder network learns to approximate a posterior q(z|x) for each training image x, and the KL term in the loss explicitly penalizes that posterior for straying from the prior p(z) = N(0, I). By the end of training, the encoder has effectively packed the entire training distribution into a latent space that looks, in aggregate, like a standard normal distribution — that is the whole point of the KL term. Once that property holds, the encoder becomes unnecessary for generation: any vector drawn directly from N(0, I) already sits in a region the decoder learned to interpret, because the encoder spent all of training pushing real images into exactly that region.

So generation is decoder-only. Sample a latent vector z ~ N(0, I) — pure noise, no image involved — feed it into the trained decoder, and read off the output. The decoder's output is not itself the final pixel values in the strict probabilistic sense; it is the mean μ of a per-pixel distribution (typically Gaussian for continuous pixel intensities, or Bernoulli for binary/grayscale-as-probability images). In practice, almost everyone just uses this mean directly as the generated image rather than adding an extra sampling step on top of it, because the per-pixel noise that a true sample would add is visually indistinguishable from sensor noise and adds nothing useful. That practical shortcut — treating the decoder's mean output as "the image" — is itself one of the roots of the blur problem covered next.

The diagram below lays out this decoder-only pipeline against the two other generation mechanisms this chapter compares it to, so you have a map before working through the details of each.

Worked example: sizing and tracing a real decoder

To make "single forward pass" concrete, here is a decoder sized for 28×28 grayscale images with a 20-dimensional latent space — small enough to trace by hand, large enough to be a real architecture. It takes a 20-dimensional vector, expands it with a dense layer into a small feature map, then upsamples twice with transposed convolutions until it reaches full resolution.

import torch
import torch.nn as nn

class VAEDecoder(nn.Module):
    def __init__(self, latent_dim=20):
        super().__init__()
        self.fc = nn.Linear(latent_dim, 7 * 7 * 32)
        self.deconv1 = nn.ConvTranspose2d(32, 16, kernel_size=4, stride=2, padding=1)
        self.deconv2 = nn.ConvTranspose2d(16, 1, kernel_size=4, stride=2, padding=1)

    def forward(self, z):
        h = self.fc(z)                       # (batch, 1568)
        h = h.view(-1, 32, 7, 7)              # (batch, 32, 7, 7)
        h = torch.relu(self.deconv1(h))       # (batch, 16, 14, 14)
        mu = torch.sigmoid(self.deconv2(h))   # (batch, 1, 28, 28)
        return mu

decoder = VAEDecoder(latent_dim=20)
z = torch.randn(1, 20)            # sampled from the prior N(0, I) -- no image involved
generated_mean = decoder(z)        # this IS the generation step, start to finish
n_params = sum(p.numel() for p in decoder.parameters())
print(generated_mean.shape, n_params)

Trace the shapes by hand before trusting the comment. The dense layer maps 20 inputs to 7×7×32 = 1568 outputs, reshaped to a (32, 7, 7) feature map. Each transposed convolution uses the standard formula out = (in − 1) × stride − 2 × padding + kernel. The first: (7 − 1) × 2 − 2 × 1 + 4 = 12 − 2 + 4 = 14, giving (16, 14, 14). The second: (14 − 1) × 2 − 2 × 1 + 4 = 26 − 2 + 4 = 28, giving (1, 28, 28) — a full-size grayscale image, produced from noise in exactly two upsampling steps plus one dense layer. The parameter count is worth auditing too, since it's the number that determines how much compute a "single forward pass" actually costs: the dense layer contributes 20 × 1568 + 1568 = 32,928 parameters; the first deconvolution contributes 4 × 4 × 32 × 16 + 16 = 8,208; the second contributes 4 × 4 × 16 × 1 + 1 = 257. Summed: 32,928 + 8,208 + 257 = 41,393. The printed line reads torch.Size([1, 1, 28, 28]) 41393 — a complete image generated by one 41,393-parameter network in one pass, no iteration, no adversary, no image ever read as input.

Why VAE samples come out blurry: the Gaussian-likelihood culprit

The blur is not a training bug and not a sign of insufficient capacity. It follows directly from the loss function's mathematical structure, and you can derive exactly why with two numbers.

Treat each output pixel as the mean of a Gaussian with fixed variance, p(x | z) = N(x; μ(z), σ²). The negative log-likelihood of a true pixel value x under this model is proportional to (x − μ)² / (2σ²) — with σ fixed, minimizing this loss is mathematically identical to minimizing squared error, which is exactly why VAE decoders are so often trained with plain pixel-wise MSE. Now suppose a particular latent region is genuinely ambiguous: two training images that the encoder happens to map to nearby latent codes disagree at one pixel — one shows a bright hemorrhage spot (intensity 0.9), the other shows healthy tissue at that same pixel location (intensity 0.1). A single decoder mean μ has to answer for both, weighted equally, since the encoder gave the decoder no way to tell them apart at that point in latent space.

import numpy as np

x1, x2 = 0.9, 0.1  # two plausible true pixel values sharing one latent region
candidates = {"mean (blurred)": 0.5, "commit to bright": 0.9, "commit to dark": 0.1}
for name, mu in candidates.items():
    expected_nll = 0.5 * (x1 - mu) ** 2 + 0.5 * (x2 - mu) ** 2
    print(name, round(expected_nll, 4))

This prints mean (blurred) 0.16, commit to bright 0.32, and commit to dark 0.32. Setting the derivative of the expected loss to zero, (μ − x1) + (μ − x2) = 0, gives μ* = (x1 + x2) / 2 = 0.5 — a muddy gray, literally the average of "lesion present" and "lesion absent" — and that gray achieves a lower loss (0.16) than either sharp commitment (0.32 each). The optimizer is not failing to find the sharp answer; the sharp answer is mathematically worse under this loss whenever genuine ambiguity exists. A pixel-wise Gaussian likelihood cannot express "it's either this or that" — it can only express "it's probably somewhere around here" — and averaging is the loss-minimizing response to multimodal uncertainty. This is the real mechanism behind VAE blur, independent of network size, depth, or training duration: more capacity can shrink how much genuine ambiguity survives per latent code (finer-grained z means fewer, more similar images share a region), but it cannot make averaging stop being the optimal answer wherever ambiguity remains.

VAE vs GAN: stability bought at the price of sharpness

A GAN generator never optimizes a pixel-wise reconstruction loss at all, which is precisely why it doesn't inherit this blur mechanism. Its loss comes from a discriminator's verdict: min_G max_D E_{x~p_data}[log D(x)] + E_{z~p_z}[log(1 − D(G(z)))]. Goodfellow et al. (2014) showed that with an optimal discriminator, this reduces to the generator minimizing the Jensen–Shannon divergence between the real and generated distributions — and a blurred, averaged-out image is easy for a discriminator to catch, because it doesn't match the local pixel statistics of any real sample. There is no incentive to hedge between two plausible outputs; hedging gets penalized, not rewarded. (In practice, most implementations use Goodfellow's non-saturating variant of the generator loss, −log D(G(z)), to avoid the vanishing gradients that the original minimax objective suffers once the discriminator becomes confident — this version no longer has as clean a divergence-minimization interpretation, but empirically it still produces sharp, non-averaged samples.)

The cost shows up in what the generator is never forced to do: reproduce every training example. A VAE's per-example reconstruction term directly penalizes the model any time a real training image is assigned a poor decoder fit, so every mode in the training data pulls on the loss. A GAN generator, by contrast, never sees a real image as input during its own forward pass — it only receives a gradient signal filtered through whatever the discriminator currently cares about. If concentrating on a handful of highly convincing lesion patterns is enough to fool the discriminator at each step, nothing in the objective forces the generator to also cover the rarer patterns. This is mode collapse: confident, sharp, and systematically missing chunks of the true data distribution. For the retina-screening dataset — 400 images already thin on rare subtypes — a GAN that mode-collapses onto the two or three most common hemorrhage patterns would quietly make the classifier worse at exactly the cases it most needs to catch, while looking, to the human eye, like it generated great images. Training instability compounds the risk: the generator and discriminator are chasing a moving target in each other, and with 400 images there usually isn't enough data to keep that adversarial balance from oscillating or collapsing outright. A VAE's single, well-defined loss (reconstruction plus KL) has no adversary to destabilize it, which is the concrete, engineering reason to prefer a blurrier VAE over a sharper GAN when the downstream goal is broad, reliable coverage of a small, imbalanced dataset rather than photorealism.

VAE vs diffusion models: sample quality versus sampling speed

Diffusion models solve the blur problem from a different direction entirely, and currently produce the sharpest, most diverse samples of any of the three families — at a real compute cost that matters for a no-GPU clinic laptop. A Denoising Diffusion Probabilistic Model (DDPM), introduced by Ho, Jain, and Abbeel (2020), is trained to reverse a fixed process that gradually adds Gaussian noise to an image over many steps until it becomes indistinguishable from pure noise. A single U-Net is trained to predict, at any noise level t, the noise that was added — and generation runs this network repeatedly, subtracting a little predicted noise at each step, walking backward from pure noise x_T to a clean image x_0. Ho et al. used T = 1000 steps. Because each step is conditioned on a much narrower, less ambiguous denoising problem than "produce the whole image at once," the model is never forced into the VAE's single-shot averaging trade-off — but it pays for that with 1000 sequential network calls instead of one. Song, Meng, and Ermon's DDIM (2021) reformulates sampling as a non-Markovian process that can skip steps using the same trained network, commonly bringing that down to 20–100 steps with a modest quality cost.

Put concrete numbers on it, using the 41,393-parameter decoder's forward-pass cost as a stand-in for one network call on the clinic's CPU-only laptop — say roughly 5 milliseconds per call, a reasonable order of magnitude for a network this small on a modern laptop CPU. Generating 500 synthetic retina patches overnight: the VAE needs 500 × 5 ms = 2.5 seconds total. A full DDPM at T = 1000 needs 500 × 1000 × 5 ms = 2,500,000 ms ≈ 41.7 minutes. DDIM at 50 steps needs 500 × 50 × 5 ms = 125,000 ms ≈ 2.1 minutes. The VAE is roughly 1000× faster than full DDPM sampling and about 50× faster than DDIM on the same hardware — the gap that keeps VAEs relevant even though diffusion models now dominate image-quality leaderboards.

Closing the gap: hybrid architectures

The sharpness-versus-stability tradeoff is not a permanent ceiling; it's a consequence of using a plain pixel-wise likelihood as the reconstruction target, and swapping that target out fixes it without giving up the VAE's stable training. Larsen, Sønderby, Larochelle, and Winther (2016) proposed exactly this in "Autoencoding beyond pixels using a learned similarity metric" (the VAE-GAN): keep the encoder, the KL term, and the reparameterization trick, but replace the pixel-wise reconstruction loss with a feature-space distance measured inside a discriminator's hidden layers, combined with an adversarial loss on the discriminator's final output. Because a discriminator's internal features aren't built around per-pixel averaging, backpropagating through this "learned similarity metric" no longer rewards the decoder for hedging between plausible outputs — while the encoder and KL regularization retain much of the plain VAE's training stability, since there is still a well-defined per-example target rather than only an adversarial signal.

A different hybrid, and the one behind most production text-to-image systems today, goes the opposite direction: keep diffusion for sample quality, but use a VAE to make it fast enough to run at all. Rombach, Blattmann, Lorenz, Esser, and Ommer's Latent Diffusion Models (2022) — the architecture underlying Stable Diffusion — train a VAE-style autoencoder to compress images into a much smaller latent grid, then run the entire slow, iterative denoising process in that compressed latent space rather than on raw pixels, and use the VAE's decoder only once at the very end to expand the final latent back to a full-resolution image. The VAE's single-pass efficiency handles the expensive pixel-space compression and decompression; the diffusion process, running on a far smaller latent tensor, handles the sample quality. Neither the retina classifier nor a from-scratch VAE-GAN needs anything this elaborate, but it's worth knowing that the "VAE versus diffusion" framing in this chapter isn't a permanent either/or in the field — the two ideas increasingly compose.

Active recall

Attempt each question before reading its answer.

1. A trained VAE has a 20-dimensional latent prior N(0, I). To generate a brand-new image, which network(s) run, and on what input?
2. Using the two-pixel example (x1 = 0.9, x2 = 0.1, equal weight), suppose a third training example with pixel value x3 = 0.5 turns out to share the same latent region. What is the new optimal decoder mean μ*, and what is the resulting expected loss? Show the derivative step.
3. Why doesn't a GAN generator suffer from the same averaging mechanism as a VAE decoder — and what does it risk instead?
4. The rural clinic's laptop takes about 5 ms per network forward pass. Compare the wall-clock time to generate 500 images using the VAE decoder (1 pass each), a DDPM at T = 1000, and DDIM at 50 steps.
5. True or false: "A well-trained VAE's blur means the network is too small or undertrained — a bigger network fixes it." Justify your answer using the two-pixel derivation.
6. What does the VAE-GAN (Larsen et al., 2016) change relative to a plain VAE, and why does that change reduce blur without discarding the KL-regularized latent space?

Answers.

1. Only the decoder runs. Sample z ~ N(0, I₂₀) — pure noise, no image — and feed it forward through the decoder to get the pixel-mean output. The encoder is never invoked at generation time; it only mattered during training, where it shaped the latent space so that prior samples would land somewhere the decoder had learned to interpret.

2. With three equally weighted values, the loss becomes (1/3)[(x1 − μ)² + (x2 − μ)² + (x3 − μ)²]. Setting the derivative to zero, (μ − x1) + (μ − x2) + (μ − x3) = 0, gives μ* = (x1 + x2 + x3) / 3 = (0.9 + 0.1 + 0.5) / 3 = 0.5 — the sample mean is the general-purpose minimizer of squared error, for any number of points, which is why this result generalizes beyond the two-point case. The resulting loss is (1/3)[(0.4)² + (−0.4)² + 0²] = (1/3)(0.32) ≈ 0.1067, lower than the two-point case's 0.16, because adding a third example exactly at the eventual mean concentrates more of the distribution's mass near the center — a useful reminder that blur severity depends on how spread out the true ambiguity is, not just on whether ambiguity exists.

3. A GAN generator is never scored against a pixel-wise likelihood at all; its gradient comes entirely from a discriminator's real-or-fake judgment, and a blurred, averaged output is trivially easy for the discriminator to flag as fake because it doesn't match real local pixel statistics. So there's no reward for hedging between plausible outputs. What it risks instead is mode collapse: since the generator never has to reproduce any specific training image, it can satisfy the discriminator by committing fully to a narrow subset of realistic-looking outputs while leaving other parts of the true data distribution completely uncovered.

4. VAE: 500 × 5 ms = 2.5 seconds. DDPM (T = 1000): 500 × 1000 × 5 ms = 2,500,000 ms ≈ 41.7 minutes. DDIM (50 steps): 500 × 50 × 5 ms = 125,000 ms ≈ 2.1 minutes. The VAE is about 1000× faster than full DDPM and about 50× faster than DDIM on the same hardware.

5. False. The two-pixel derivation shows the optimal decoder mean under a pixel-wise Gaussian/MSE loss is provably the average of the ambiguous true values, regardless of how large or well-trained the network is — a bigger network can only shrink how much genuine multimodal ambiguity survives at each point in latent space (by giving nearby latent codes finer, more specific meanings), it cannot make the averaging response stop being loss-optimal wherever real ambiguity remains. Fixing blur requires changing the likelihood or loss itself — a learned similarity metric (VAE-GAN), an autoregressive or mixture pixel model, or discretizing pixel values — not simply scaling up capacity.

6. The VAE-GAN keeps the encoder, the KL term, and the reparameterization trick, but replaces the pixel-wise reconstruction loss with a distance measured in a discriminator's internal feature space, plus an adversarial term. Because those learned features aren't structured around per-pixel averaging, gradients flowing back through this similarity metric no longer push the decoder toward a blurred mean — while the encoder and KL regularization are untouched, so the model still has a stable, well-defined per-example training signal rather than relying purely on an adversarial game the way a plain GAN does.

Think About It

Think about this: How would you explain image generation and variational autoencoders to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind image generation and variational autoencoders, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.