GANs: Generative Adversarial Networks

The forger and the cashier

Every year the Reserve Bank of India's currency verification units intercept lakhs of forged notes, and every year the forgeries get harder to catch. A forger who prints a note that gets rejected doesn't give up — they study exactly which security feature triggered the rejection (the watermark, the security thread, the micro-lettering) and fix it in the next batch. A cashier who is fooled by a good forgery doesn't shrug it off either — the next note they inspect gets scrutinized more carefully at exactly the feature that fooled them last time. Run this back-and-forth for long enough and two things happen simultaneously: the forgeries become good enough to pass casual inspection, and the cashier becomes very good at spotting the remaining tells. Neither side is optimizing in isolation. Each one's improvement is defined entirely in terms of the other's current weaknesses.

This is, almost exactly, the training procedure Ian Goodfellow and coauthors proposed in 2014 in "Generative Adversarial Networks" (NeurIPS 2014). Replace the forger with a neural network called the generator, replace the cashier with a neural network called the discriminator, and replace "genuine currency" with "real data" — images, audio, text, whatever the training set contains. The generator never sees a single real example. It only ever sees the discriminator's verdicts, and it improves purely by exploiting whatever the discriminator currently fails to catch. This chapter builds that game from first principles, derives exactly what each network is solving for, and works through why — unlike almost every other model you have trained so far — getting a GAN to converge at all is a genuinely hard, actively researched problem.

The minimax game, formally

A GAN couples two networks with opposing objectives:

• Generator G(z; θ_g) maps a noise vector z, sampled from a simple prior p_z (typically z ~ N(0, I) in, say, 100 dimensions), to a point in data space — a 64×64 image, for instance. G defines an implicit distribution p_g over that data space: the distribution you'd get by sampling infinitely many z and pushing each one through G.
• Discriminator D(x; θ_d) maps a point in data space to a scalar in (0, 1): its estimate of the probability that x came from the real dataset rather than from G.

Training alternates: show D a mix of real samples (label 1) and generated samples (label 0), update θ_d to classify them better; then freeze D, push generated samples through it, and update θ_g to make D's output on those samples closer to 1. Goodfellow et al. write this as a single value function both networks act on with opposite goals:

min_G max_D V(D, G) = E_{x ~ p_data}[log D(x)] + E_{z ~ p_z}[log(1 - D(G(z)))]

Read the two terms separately. The first is the expected log-probability D assigns to real data — D wants this large, so it wants D(x) ≈ 1 on real x. The second is the expected log-probability D assigns to generated data being fake — D wants this large too, so it wants D(G(z)) ≈ 0. G controls only the second term, and it wants the opposite: D(G(z)) ≈ 1, i.e. it wants to minimize log(1 − D(G(z))). Every other model you've trained in this course — a classifier, a regressor, an autoencoder — has one loss and one direction of descent for every parameter. A GAN has one loss and two opposite directions of travel on the same scoreboard. That single structural fact is the source of almost everything unusual about training GANs, and it is worth sitting with before going further.

What the optimal discriminator looks like

Fix G (and therefore p_g) and ask: for a given generator, what discriminator maximizes V? Because the expectations are over x and z independently, we can optimize D(x) pointwise for every x. Write y = D(x) and treat pdata(x) and pg(x) as fixed constants at that point (using pg to denote the density G induces via the change-of-variables from z). The contribution of a single x to V is:

f(y) = p_data(x)·log(y) + p_g(x)·log(1 - y)

Differentiate with respect to y and set to zero:

f'(y) = p_data(x)/y - p_g(x)/(1 - y) = 0
⇒ p_data(x)·(1 - y) = p_g(x)·y
⇒ p_data(x) = y·(p_data(x) + p_g(x))
⇒ y* = D*(x) = p_data(x) / (p_data(x) + p_g(x))

This is the single most important equation in the chapter, and it earns its own reading: the best possible discriminator at any point x is simply the local mix ratio of real to (real + fake) density there. Where the generator has piled up far more density than the real data ever had, D* is pushed toward 0. Where the generator has never placed any density at all, D*(x) = p_data(x)/(p_data(x) + 0) = 1 — total, correct confidence. And critically: if the generator ever matches the real distribution exactly, p_g = p_data everywhere, so D*(x) = p_data(x)/(2·p_data(x)) = 1/2 for every x. At the true optimum, the best possible discriminator cannot do better than a coin flip, because there is genuinely nothing left to distinguish. Substituting D* back into V and simplifying (Goodfellow et al.'s Theorem 1) gives the value at that optimum as C(G) = −log 4 + 2·JSD(p_data ‖ p_g), where JSD is the Jensen–Shannon divergence, which is always ≥ 0 and equals 0 only when p_data = p_g. So minimizing V over G is, in the limit of a perfectly-trained D, exactly minimizing the Jensen–Shannon divergence between the real and generated distributions, with a global minimum of −log 4 ≈ −1.3863.

Worked example: scoring a generator mid-training

Take a deliberately small discrete case so every number can be checked by hand. Suppose the "real data" is four equally likely categories — call them A, B, C, D (think: four denominations of a note the forger is trying to reproduce) — so p_data(A) = p_data(B) = p_data(C) = p_data(D) = 0.25. Suppose the generator, partway through training, has collapsed onto just two of them: p_g(A) = p_g(B) = 0.5, p_g(C) = p_g(D) = 0. This is mode collapse in miniature, and we'll return to it below.

Apply D*(x) = p_data(x)/(p_data(x) + p_g(x)) at each of the four points:

D*(A) = 0.25 / (0.25 + 0.50) = 0.3333
D*(B) = 0.25 / (0.25 + 0.50) = 0.3333
D*(C) = 0.25 / (0.25 + 0.00) = 1.0000
D*(D) = 0.25 / (0.25 + 0.00) = 1.0000

Read this before computing anything else: on the two categories the generator actually produces, the discriminator is only 33% confident they're real — not bad camouflage. But on the two categories the generator never touches, the discriminator is at absolute, unbeatable confidence: 1.0, because every single sample seen at C or D is real by construction. Now assemble C(G) = E_{x~p_data}[log D*(x)] + E_{z~p_g}[log(1 − D*(G(z)))]:

Term 1 = 0.25·log(0.3333) + 0.25·log(0.3333) + 0.25·log(1) + 0.25·log(1)
       = 0.5·(-1.0986) + 0 = -0.5493

Term 2 = 0.5·log(1 - 0.3333) + 0.5·log(1 - 0.3333)
       = log(0.6667) = -0.4055

C(G) = -0.5493 + (-0.4055) = -0.9548

Compare against the true optimum, −log 4 = −1.3863. The generator is falling short by 0.4315 nats. Converting that gap back through C(G) = −log 4 + 2·JSD tells you the Jensen–Shannon divergence between p_data and p_g right now is (−0.9548 − (−1.3863))/2 = 0.2158 nats — you can also compute this directly from the JSD formula using the mixture M = (p_data + p_g)/2 at each point, which gives the identical 0.2158, confirming the two routes agree. That gap is not evenly spread across the four categories — it is concentrated almost entirely in the fact that C and D receive zero generator density at all, which is exactly what "mode collapse" will mean formally in a moment.

Why GAN training is notoriously unstable

Every optimization problem you've met before this chapter — minimize cross-entropy, minimize squared error — has one network descending one loss surface. Gradient descent on a fixed loss with a fixed target is well-behaved: the loss can only go down (up to step-size issues), and a local minimum, once reached, stays reached. A GAN has no such guarantee, for a structural reason: G is descending V while D is simultaneously ascending the same V, and the surface G is descending changes shape every time D's parameters move. There is no single fixed loss landscape for G to walk downhill on — the landscape itself is being reshaped, adversarially, every discriminator step. Simultaneous gradient descent–ascent on a non-convex, non-concave two-player game has no general convergence guarantee the way convex optimization does; in practice, GAN training is known to oscillate — G's parameters can circle a region without settling, chasing whatever exploit D leaves open at each step, while D chases whatever G just did.

There is also a balance problem baked into the setup. If D is trained to near-perfection before G gets to update at all, D becomes extremely confident on essentially every sample G currently produces (as in the worked example above, where D*(C) and D*(D) hit exactly 1.0) — and, as derived in the next section, a very confident D provides almost no usable gradient to G. If instead D is undertrained, its verdicts are close to a coin flip everywhere and G receives a noisy, low-information signal about which direction actually helps. Practitioners have to tune the relative number of D-steps versus G-steps per iteration, and no fixed ratio works for every architecture or dataset — this is itself evidence that the two-player structure, not any single hyperparameter, is the root cause of the instability.

The vanishing-gradient trap, derived

The most concrete version of "D too strong ⇒ G stalls" can be derived exactly rather than asserted. Let a be the pre-sigmoid logit the discriminator computes for a generated sample, so s = D(G(z)) = σ(a), where σ is the logistic sigmoid. The minimax generator loss is L_minimax = log(1 − s). By the chain rule, the gradient that actually reaches the generator's parameters is proportional to dL/da = (dL/ds)·(ds/da). The sigmoid's own local derivative is ds/da = s(1 − s) — this is the same saturating-sigmoid term you already know flattens out near 0 and 1. For the minimax loss, dL/ds = −1/(1 − s), so:

dL_minimax/da = [-1/(1 - s)] · [s(1 - s)] = -s

Early in training, D correctly and confidently rejects G's samples, so s = D(G(z)) is close to 0 — exactly the D*(C) = D*(D) = 1.0 situation above, mirrored from the generator's side (a generated sample landing where D is near-certain it's fake means s ≈ 0, the flip side of D being near-certain real samples are real). Plug s ≈ 0 into dL_minimax/da = −s: the gradient reaching G is close to zero. Not because the sigmoid saturated in the usual sense, but because the −1/(1−s) term from the loss and the s(1−s) term from the sigmoid multiply together to cancel almost exactly at small s. G gets almost no signal precisely when it most needs correcting.

Goodfellow's fix, used in essentially every practical GAN implementation since 2014, is to keep D's training objective unchanged but swap G's loss for the non-saturating form L_nonsat = −log(s) (equivalently: G maximizes log D(G(z)) instead of minimizing log(1 − D(G(z)))). Redo the same chain rule: dL/ds = −1/s, so:

dL_nonsat/da = [-1/s] · [s(1 - s)] = -(1 - s)

At the same early-training point, s ≈ 0, this gives dL_nonsat/da ≈ −1 — a strong, healthy gradient, the opposite of what the minimax form produced at the identical point. Nothing about D changed; only the functional form of G's loss did, and that alone converts a vanishing gradient into a large one. It's worth checking the two forms don't just differ arbitrarily: at s = 0.5 (the Nash equilibrium point derived earlier, where D can no longer tell real from fake), dL_minimax/da = −0.5 and dL_nonsat/da = −(1 − 0.5) = −0.5 — identical. The two losses only disagree away from equilibrium, exactly where it matters: at s = 0.3333 (D's confidence on the mode-collapsed generator's own samples A and B from the worked example), minimax gives −0.3333 while non-saturating gives −0.6667 — exactly double the corrective push, at the point in training where a push is most needed.

Mode collapse

Mode collapse is what happens when G discovers that it does not need to match all of p_data to fool the current D — it only needs to find some region D currently misjudges, and pile probability mass there. The worked example above is mode collapse frozen at one instant: G produces only A and B, at 50% each, and never C or D at all, even though all four are equally represented in the real data. Formally, p_g places zero density on part of the support of p_data. The danger is that this can be a stable-looking failure from G's local point of view: as long as D hasn't caught up on A and B specifically, G's loss on those samples looks fine, and there is no term in G's loss that rewards diversity for its own sake — G is never shown C or D and gets no signal to reproduce them. The generator is optimizing "fool the current discriminator," not "match the full data distribution," and those two objectives only provably coincide at the unreachable Nash equilibrium, not at any point along the way.

In practice mode collapse frequently oscillates rather than sitting still: once D adapts enough to catch A and B confidently, G's best remaining exploit might be a different narrow region — say only C — and G can jump there wholesale, abandoning A and B, rather than smoothly covering all four. This is the practical, visible symptom engineers watch for: a generator whose sample diversity visibly narrows, or cycles between a few narrow "styles," well before its samples look convincingly real.

Architecture matters: what DCGAN changed

The original 2014 paper used fully-connected networks for both G and D and worked on small datasets. Getting GANs to work reliably on real images took an architectural contribution: Alec Radford, Luke Metz, and Soumith Chintala's "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks" (ICLR 2016, commonly cited as DCGAN), which established a set of architecture guidelines still used as the default starting point for convolutional GANs. Four choices from that paper matter enough to name explicitly:

• Strided convolutions instead of pooling. The discriminator downsamples using convolutions with stride > 1 rather than max- or average-pooling, and the generator upsamples using fractionally-strided (transposed) convolutions. Pooling throws away spatial information the network has no way to recover; letting the network learn its own downsampling/upsampling filters gives it that information back.
• Batch normalization in both networks (except the generator's output layer and the discriminator's input layer), which the authors found necessary to prevent the generator from collapsing all of its samples to a single point early in training — a direct, practical countermeasure against the mode-collapse dynamics described above.
• No fully-connected hidden layers in deeper models — the highest-level features connect straight to the convolutional/deconvolutional stack, improving training stability at higher resolutions.
• LeakyReLU in the discriminator instead of plain ReLU, versus ReLU in the generator (with a final tanh output). A standard ReLU zeroes out every negative activation, discarding gradient information for those units entirely; LeakyReLU keeps a small non-zero slope (the paper uses 0.2) for negative inputs, which the authors found helped the discriminator's gradients stay informative — directly relevant given how central the discriminator's gradient quality is to whether G learns anything at all.

None of these change the minimax game itself — V(D, G) is unchanged — but they change how reliably gradient descent–ascent actually finds a good solution to that game in practice, which is exactly the gap between the clean theory in the earlier sections and what it takes to train a working GAN on real images.

Common misconception

Students who first meet a discriminator's accuracy number (say, "D correctly classifies 96% of real vs. fake samples") tend to read a high number as the GAN succeeding — by analogy with every other classifier in the course, where higher accuracy is unambiguously better. For a GAN, this reasoning is backwards. Recall the derivation: at the actual global optimum, where p_g exactly equals p_data, D*(x) = 1/2 everywhere — the best possible discriminator can only guess, because generated and real samples are drawn from identical distributions and there is nothing left to tell apart. A discriminator sitting at 96% accuracy is telling you the opposite of success: it means p_g and p_data are still far enough apart that a well-trained classifier can separate them most of the time, and per the vanishing-gradient derivation above, a discriminator that confident is also the exact regime in which G's gradient signal is weakest. The correct read of discriminator accuracy on a GAN is: falling toward 50% over training is the sign of progress; staying pinned near 100% (D wins completely) or collapsing to near 0% (G has found one exploit fooling D on almost everything, a mode-collapse signature) are both failure signatures, just in opposite directions.

The adversarial training loop

Active recall

Attempt each question before reading its answer.

Questions

1. Why is a discriminator accuracy converging toward 50% a sign of a GAN succeeding, rather than failing?
2. In the worked mode-collapse example (p_g: A=0.5, B=0.5, C=0, D=0), suppose the generator retrains and its distribution shifts to p_g: A=0.3, B=0.3, C=0.4, D=0 — it now covers three of the four categories instead of two, but still never produces D at all. Recompute D*(x) at all four points, recompute C(G), and state whether the JSD between p_data and p_g increased or decreased, and by how much. Also state what happens to D*(D) specifically, and why.
3. Using dL/da = −s for the minimax generator loss and dL/da = −(1 − s) for the non-saturating loss, show that the two losses give identical gradients at s = 0.5, and explain in one sentence why that specific value of s is the meaningful one to check.
4. Why does DCGAN replace max-pooling with strided convolutions in the discriminator, and what does this choice have to do with the amount of information the discriminator can preserve about spatial position?
5. In the original mode-collapse example, using the non-saturating loss, compute the gradient magnitude |dL/da| the generator receives for samples currently landing at A (where D*(A) = 0.3333), and compare it to what the minimax loss would have given at the same point. Which loss pushes G harder to move away from A, and does this match the direction you'd want given that A is over-represented in p_g relative to p_data?
6. A classmate claims mode collapse and ordinary underfitting are the same phenomenon — "the model just hasn't learned enough yet." Give one property of mode collapse that a plain underfitting explanation does not account for.

Answers

1. Because the discriminator's Bayes-optimal value is D*(x) = p_data(x)/(p_data(x) + p_g(x)). This equals 1/2 for every x exactly when p_g(x) = p_data(x) everywhere — that is, only when the generator's distribution has become indistinguishable from the real one. A discriminator stuck near 50% is reporting that it has run out of distinguishing evidence, which is the definition of the generator having matched the data distribution, not a sign that D has stopped trying.
2. Applying D*(x) = p_data(x)/(p_data(x) + p_g(x)): D*(A) = D*(B) = 0.25/0.55 = 0.4545, D*(C) = 0.25/0.65 = 0.3846, D*(D) = 0.25/0.25 = 1.0 — unchanged from the original example. Term 1 = 0.25·[log(0.4545)+log(0.4545)+log(0.3846)+log(1)] = 0.25·(−0.7885 −0.7885 −0.9555 + 0) = −0.6331. Term 2 = 0.3·log(1−0.4545)+0.3·log(1−0.4545)+0.4·log(1−0.3846) = 0.3·(−0.6061)·2 + 0.4·(−0.4855) = −0.3637 − 0.1942 = −0.5579. C(G) = −0.6331 − 0.5579 = −1.1910, versus −0.9548 originally — closer to the optimum −1.3863, so the JSD decreased, from 0.2158 to (−1.1910+1.3863)/2 = 0.0977 nats, roughly a 55% reduction. The easy-to-miss ripple: D*(D) stays pinned at exactly 1.0 in both scenarios, unchanged by how the generator redistributes mass among A, B, and C — because that term only depends on p_g(D), which is still 0. Spreading coverage among the modes the generator already touches measurably shrinks the divergence, but it does nothing at all for the one mode still receiving zero density; the "improvement" the JSD reports can coexist with total, uncorrected non-coverage of part of the real distribution.
3. At s = 0.5: minimax gives dL/da = −s = −0.5; non-saturating gives dL/da = −(1−s) = −(1−0.5) = −0.5. Both equal −0.5. s = 0.5 is the value D* takes at the Nash equilibrium (p_g = p_data), so checking the gradients agree there confirms the two loss formulations only disagree away from equilibrium — exactly in the early-training, D-confident regime the non-saturating loss was designed to fix — and agree exactly at the point training is actually trying to reach.
4. Max-pooling keeps only the maximum activation in each window and discards everything about where within that window the feature occurred, as well as the exact activation values of the neighboring units — that spatial and magnitude information cannot be recovered later in the network. A strided convolution instead learns its own downsampling filter, so the network chooses what information to keep or discard as part of training rather than having a fixed, non-differentiable rule imposed on it; this preserves more of the signal the discriminator needs to judge fine spatial structure (edges, textures) that separates real images from generated ones, and — since it is a learned, differentiable operation like the rest of the network — it participates properly in backpropagation rather than acting as a gradient bottleneck.
5. Non-saturating: dL/da = −(1−s) = −(1−0.3333) = −0.6667. Minimax: dL/da = −s = −0.3333. The non-saturating loss gives exactly double the gradient magnitude at this point. This matches intuition: A is being produced at twice the rate the real data warrants (p_g(A)=0.5 vs p_data(A)=0.25), so a stronger corrective push away from over-producing A is exactly the direction training should favor, and the non-saturating loss supplies that stronger push precisely where the minimax loss would have been weakest.
6. Underfitting predicts uniformly poor performance across the whole data distribution because the model has not yet captured enough structure anywhere. Mode collapse instead produces samples that are locally excellent — the categories or regions the generator does cover can look highly realistic and fool the discriminator effectively — while entire other regions of the real distribution receive zero density. That combination (high local fidelity, simultaneous complete non-coverage elsewhere) is not something "hasn't learned enough yet" explains, since more training under the same adversarial dynamic can leave those uncovered regions untouched indefinitely, or shift which regions are covered rather than converging to cover all of them.

Think About It

Think about this: How would you explain gans: generative adversarial networks to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind gans: generative adversarial networks, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.