Conditional GANs and Pix2Pix

During the monsoon, ISRO's RISAT and EOS radar-imaging satellites are the only satellites that can see the ground at all. Optical satellites like Cartosat photograph reflected sunlight, so a monsoon cloud deck blinds them completely — exactly when flood-mapping teams in the National Remote Sensing Centre need imagery most. RISAT instead carries a Synthetic Aperture Radar (SAR), which fires its own microwave pulses and reads the echoes, punching straight through cloud, smoke and darkness. The catch is that a SAR image is unreadable to a human analyst used to optical photos: metal roofs and wet pavement both glow bright, water looks uniformly dark whether it is a flooded field or a shadow, and the whole scene has a grainy, alien texture called speckle. Analysts want the geometry a SAR image guarantees — the shorelines, the roads, the flood boundary — rendered the way an optical photo would render it, in colours and textures a human can read at a glance. That is a translation problem: given one image, produce a second image of the same scene, same layout, different appearance. This is precisely the problem Pix2Pix, published by Isola, Zhu, Zhou and Efros in 2017, was built to solve, and the SAR-to-optical case is one of the domains researchers have actually applied it to. Understanding how Pix2Pix works means understanding two ideas in sequence: how to make a GAN listen to an input rather than just dreaming from noise, and how to make that listening precise enough to work pixel-for-pixel across an entire image.

From GANs to conditional GANs

Recall the vanilla GAN. A generator G takes a random noise vector z, sampled from some simple distribution like a standard normal, and maps it to an image: x = G(z). A discriminator D looks at an image and outputs the probability that it came from the real training set rather than from G. The two networks are trained adversarially:

min_G max_D V(D,G) = E_x~p_data[log D(x)] + E_z~p_z[log(1 - D(G(z)))]

D tries to maximise this quantity by correctly labelling real images as real (pushing D(x) toward 1) and generated images as fake (pushing D(G(z)) toward 0). G tries to minimise it, which means pushing D(G(z)) toward 1 — fooling the discriminator. At the Nash equilibrium of this game, G produces samples indistinguishable from the training distribution.

Notice what is missing: control. z is random noise with no semantic meaning attached to any particular dimension. You cannot hand a trained vanilla GAN a SAR image and ask for "the optical photo that corresponds to this." The generator only knows how to produce an image from the training distribution, not the image that matches a given input. A conditional GAN (cGAN), introduced by Mirza and Osindero in 2014, fixes this by handing both networks an extra piece of information y — the condition — alongside their usual input:

min_G max_D V(D,G) = E_x,y~p_data[log D(x,y)] + E_z~p_z,y~p_data[log(1 - D(G(z,y),y))]

Two changes matter here. First, G now maps (z,y) → x: it generates conditioned on y. Second — and this is the part students most often gloss over — D now also receives y, and judges the pair (x,y) together, not x alone. This second change is what actually enforces conditioning. If D only ever saw x by itself, G could satisfy it by producing any realistic-looking image regardless of whether it matches y — a perfectly photorealistic optical image of the wrong place would still fool a discriminator that never checks correspondence. By forcing D to score the pair, a mismatched-but-realistic (x,y) becomes something D can learn to reject, and that pressure is what pushes G toward outputs that are faithful to y, not merely realistic in isolation.

Pix2Pix: when the condition is an entire image

Pix2Pix is the special case where y is not a class label or a short embedding but a full image, and the desired output x is also a full image of the same spatial size, aligned pixel-for-pixel with y. This is called paired image-to-image translation: every training example is a matched pair — a SAR tile and the optical tile of the same ground patch, a building facade photo and its semantic label map, a hand-drawn shoe sketch and a photo of that shoe. The network's job reduces to learning a mapping G: y → x where the underlying geometry of y and x is identical and only the "rendering style" differs.

Because the condition already carries almost all the spatial information the output needs, Pix2Pix's generator is trained to be close to deterministic in practice: given the same SAR tile twice, you want essentially the same optical-style output, not wildly different random samples. The paper found that feeding an explicit noise vector z alongside y did not help — the generator learned to largely ignore it, since ignoring z and relying on y already minimises the loss. So Pix2Pix drops the explicit z input and instead injects a small amount of stochasticity by keeping dropout active in several generator layers not just during training but at test time as well. This is worth sitting with, because it directly contradicts the assumption most students carry over from vanilla GANs.

The generator: a U-Net, not a plain encoder-decoder

Since input and output are both images of the same size, an obvious generator design is an encoder-decoder: convolutional layers progressively downsample y into a compressed bottleneck representation, then transposed-convolutional layers progressively upsample it back to full resolution to produce x̂. This works, but it forces every bit of information needed to reconstruct the output — including low-level details like exact edge positions — to survive being squeezed through a narrow bottleneck. For image-to-image translation tasks specifically, input and output share a lot of low-level structure: a doorway's outline in a facade sketch is in the same place as the doorway's outline in the rendered photo; a coastline in a SAR tile is in the same place as the coastline in the optical tile. Forcing that shared structure through a bottleneck is wasteful and tends to blur fine boundaries.

Pix2Pix's generator is therefore a U-Net: an encoder-decoder with skip connections that copy the activation maps from each encoder layer directly across to the decoder layer at the matching resolution, concatenating them with the decoder's own upsampled features. If the encoder has n downsampling layers, layer i is connected to layer n − i. Low-level information — exact pixel boundaries, textures — takes a direct shortcut around the bottleneck, while the bottleneck itself is left to carry only the higher-level, more abstract information about what the scene actually contains. This is exactly why Pix2Pix outputs stay sharp and spatially faithful instead of looking like a decoded, blurred summary of the input.

The discriminator: PatchGAN

A standard GAN discriminator looks at the whole image and outputs one number: real or fake. Pix2Pix instead uses a PatchGAN, a fully convolutional discriminator that slides across the image and outputs a whole grid of real/fake scores, one per local patch, rather than a single scalar for the entire image. Each output value in that grid is not evaluating the whole picture — it only "sees" a limited receptive-field window of the input, so it is really asking: "does this local patch look like a real texture given its local context?" The final loss averages the classification result over every patch in the grid.

The reasoning behind this design is a division of labour between the two loss terms Pix2Pix combines (derived fully in the next section). Overall low-frequency correctness — is the picture the right colour, roughly the right layout, structurally correct at a coarse scale — turns out to be captured well by a simple pixelwise L1 loss between x̂ and the true x. What L1 loss does not enforce is high-frequency correctness: whether local textures look crisp and real rather than smoothed-out and plausible-on-average. That is exactly what a discriminator restricted to small local patches is good at judging, since a small patch has almost nothing else to check besides local texture statistics. So the PatchGAN is deliberately kept "small-sighted" — it is not trying to duplicate what L1 already does well.

Worked example 1: deriving the 70×70 receptive field

The Pix2Pix paper's default discriminator is called a "70×70 PatchGAN" — each output score corresponds to a 70×70-pixel window of the input image. This number is not arbitrary; it falls directly out of the architecture, and it is worth deriving rather than memorising. The default discriminator stacks five convolutional layers, each with a 4×4 kernel: three with stride 2, then two with stride 1 (the last of which produces the single-channel output map).

The receptive field of a stack of convolutional layers is computed layer by layer using

RF_l = RF_(l-1) + (k_l - 1) * S_(l-1)
S_l  = S_(l-1) * s_l

where k_l and s_l are the kernel size and stride of layer l, and S ("jump") tracks the cumulative stride of every layer processed so far. Starting from RF_0 = 1, S_0 = 1:

Layer 1  (k=4, s=2):  RF = 1  + (4-1)*1 = 4    S = 1*2 = 2
Layer 2  (k=4, s=2):  RF = 4  + (4-1)*2 = 10   S = 2*2 = 4
Layer 3  (k=4, s=2):  RF = 10 + (4-1)*4 = 22   S = 4*2 = 8
Layer 4  (k=4, s=1):  RF = 22 + (4-1)*8 = 46   S = 8*1 = 8
Layer 5  (k=4, s=1):  RF = 46 + (4-1)*8 = 70   S = 8*1 = 8

The receptive field lands on exactly 70, matching the published architecture's name. This is easy to check in code:

def receptive_field(layers):
    rf, jump = 1, 1
    for k, s in layers:
        rf = rf + (k - 1) * jump
        jump *= s
    return rf

layers = [(4, 2), (4, 2), (4, 2), (4, 1), (4, 1)]
print(receptive_field(layers))   # 70

If you stopped after only the first three stride-2 layers, the receptive field would be 22×22 — a much more local patch, less able to judge coherent texture but cheaper to compute and sometimes preferred for very large images. This is exactly the kind of computation you should be able to reproduce for any PatchGAN variant, not just quote.

The combined objective and worked example 2: a numeric loss

Pix2Pix's full training objective combines the conditional adversarial loss with the L1 reconstruction loss:

L(G,D) = L_cGAN(G,D) + λ * L_L1(G)

L_cGAN(G,D) = E_(x,y)[log D(x,y)] + E_y[log(1 - D(G(y),y))]
L_L1(G)     = E_(x,y)[ ||x - G(y)||_1 ]

L1 (mean absolute pixel difference) is used instead of L2 (mean squared difference) because L2 penalises large errors quadratically, which pushes the generator toward the statistical average of all plausible outputs whenever it is uncertain — the classic cause of blurring in image regression. L1 penalises linearly, tolerates a wider spread of plausible answers without averaging them into mush, and empirically produces noticeably sharper output. In the paper's default setting, λ = 100, meaning the L1 term is weighted a hundred times more heavily than the adversarial term. This looks lopsided until you check the typical magnitudes of each term, which the following toy example does explicitly.

Take a tiny 2×2 grayscale "image" so every number can be tracked by hand. Suppose the true target patch and the generator's current output are:

x (ground truth) = [[0.90, 0.10],
                     [0.20, 0.80]]

x̂ = G(y)         = [[0.75, 0.25],
                     [0.30, 0.65]]

L1 loss is the mean absolute difference over all four pixels:

|0.90-0.75| = 0.15
|0.10-0.25| = 0.15
|0.20-0.30| = 0.10
|0.80-0.65| = 0.15
mean = (0.15+0.15+0.10+0.15) / 4 = 0.55 / 4 = 0.1375

Now suppose the PatchGAN discriminator scores the fake pair (y, x̂) at four patches with predicted "probability real" values 0.40, 0.35, 0.45, 0.30 (recall the generator wants every one of these pushed toward 1). Using the standard non-saturating cross-entropy form L_cGAN(G) = -E[log D(G(y),y)], averaged over the four patches:

-ln(0.40) = 0.9163
-ln(0.35) = 1.0498
-ln(0.45) = 0.7985
-ln(0.30) = 1.2040
mean = (0.9163+1.0498+0.7985+1.2040) / 4 = 3.9686 / 4 = 0.9921

Combining with λ = 100:

L_total = L_cGAN + λ * L_L1 = 0.9921 + 100 * 0.1375 = 0.9921 + 13.75 = 14.7421

Verify with code:

import numpy as np

x      = np.array([[0.90, 0.10], [0.20, 0.80]])
x_hat  = np.array([[0.75, 0.25], [0.30, 0.65]])
l1     = np.mean(np.abs(x - x_hat))

d_fake = np.array([0.40, 0.35, 0.45, 0.30])   # D(x̂,y) per patch
adv    = -np.mean(np.log(d_fake))

lam    = 100
total  = adv + lam * l1
print(round(l1, 4), round(adv, 4), round(total, 4))
# 0.1375 0.9921 14.7421

Notice the arithmetic: the raw adversarial loss (≈0.99) and the raw L1 loss (≈0.14) sit in comparable ranges before weighting, but pixelwise L1 differences in a well-trained network shrink toward zero far faster than a discriminator's cross-entropy floor does. Without a large λ, the gradient signal pushing the generator toward pixel-level fidelity would be drowned out by the adversarial term, and the network would happily produce locally realistic-looking textures in structurally wrong places — a plausible-looking optical image of the wrong river shape, for a Pix2Pix trained on SAR-to-optical translation. Weighting L1 by 100 is what keeps the network anchored to the actual input geometry while the adversarial term does the narrower job of sharpening texture.

Architecture at a glance

A common misconception: "the randomness comes from z, like a normal GAN"

Students who have just studied vanilla GANs almost always assume Pix2Pix's diversity — the fact that it can produce slightly different outputs from the same input on different runs — comes from a random noise vector z, exactly as in DCGAN or a standard GAN. It does not. As described above, the Pix2Pix authors tried feeding z explicitly and found the generator learned to route around it almost entirely: since the input image y already determines nearly all of the correct output, the network has no incentive to pay attention to an independent noise input, and minimising the loss is easiest by ignoring z and treating the mapping as deterministic. The actual source of stochasticity in the released architecture is dropout, applied inside several of the generator's layers and — critically — left switched on at inference time rather than only during training, where dropout is normally used. Each forward pass through the trained generator effectively samples a different random dropout mask, giving slightly different outputs for the same input without needing an explicit z vector to carry that randomness. Getting this backwards leads to a common bug when students reimplement Pix2Pix from scratch: they wire up a z input exactly like a vanilla GAN, are confused when the network learns to ignore it, and never think to check whether dropout is even active at test time.

Active recall

Attempt every question before reading the answer beneath it.

1. In a vanilla GAN, D takes only x. What does D take in a conditional GAN, and why does omitting y from D's input break conditioning even if G still receives y?
2. Why does Pix2Pix use an L1 loss term instead of an L2 (mean squared error) term alongside the adversarial loss?
3. A discriminator uses three convolutional layers, each kernel 4, stride 2, with no further layers. Using the receptive-field recurrence from Worked Example 1, what patch size does each output score correspond to?
4. A generator output patch and its ground truth are: x = [[0.6,0.4],[0.5,0.9]], x̂ = [[0.5,0.5],[0.55,0.75]]. The discriminator scores the fake pair's four patches as [0.5, 0.5, 0.5, 0.5]. Using λ=100, compute L1 loss, adversarial loss (as −mean(log D)), and the total objective.
5. Why is a U-Net's skip connection more important for Pix2Pix's generator than it would be for a generator that maps random noise to an image with no structured input to preserve?
6. Pix2Pix requires paired training data — every input image must have a known, spatially aligned ground-truth output. Why is this a serious practical constraint for a task like SAR-to-optical translation, where you would need the exact same ground patch imaged by both a radar and an optical sensor at a similar time?

Answers

1. D takes the pair (x,y), not x alone. If D only ever scored x in isolation, it could never tell whether a realistic-looking x actually corresponds to the specific y it was supposed to be conditioned on — a generator could satisfy such a D by producing any plausible image regardless of input, since correspondence to y is never checked. Only by scoring the pair jointly does D create pressure for G to respect y.
2. L2 penalises errors quadratically, which rewards the generator for hedging toward the average of multiple plausible outputs whenever it is uncertain — producing blur. L1's linear penalty tolerates a wider range of correct-looking pixel values without collapsing them toward a blurred mean, so outputs stay sharper.
3. RF₁ = 1+(4−1)·1 = 4, S₁ = 2. RF₂ = 4+(4−1)·2 = 10, S₂ = 4. RF₃ = 10+(4−1)·4 = 22. Each output score corresponds to a 22×22 patch of the input.
4. L1: |0.6−0.5|+|0.4−0.5|+|0.5−0.55|+|0.9−0.75| = 0.10+0.10+0.05+0.15 = 0.40; mean = 0.40/4 = 0.10. Adversarial: −ln(0.5) = 0.6931 for each of four identical patches, so mean = 0.6931. Total = 0.6931 + 100×0.10 = 0.6931 + 10 = 10.6931.
5. Input and output in Pix2Pix share almost all of their low-level spatial structure (edges, boundaries, positions) even though their appearance differs — unlike a noise-to-image generator, where there is no structured input to preserve in the first place. Forcing that shared structure through a narrow bottleneck (as a plain encoder-decoder would) discards precise boundary information; skip connections let it bypass the bottleneck entirely, so the network only has to learn the appearance transformation, not re-derive geometry it was already given.
6. Producing a truly paired dataset needs the same ground patch captured by two different sensors, close enough in time that nothing on the ground changed (no new construction, no flooding since one image was taken), and then precisely co-registered pixel-for-pixel despite the two sensors having different geometry, resolution and viewing angles. This alignment work is expensive and often only approximate, which is exactly the gap that later unpaired approaches for image-to-image translation were built to avoid.

Think About It

Think about this: How would you explain conditional gans and pix2pix to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.