An analyst at a UPI-scale payments platform wants a single number for every transaction the instant it lands: how unusual is this? A ₹49,000 transfer at 2 a.m. to a new payee should score very differently from a ₹200 grocery payment at noon. The natural quantity for this is the probability density of the transaction under a model of "normal" behaviour, p(x), evaluated exactly, not estimated, not bounded above or below — because the decision to freeze an account is made off the actual number. This single requirement quietly rules out the two generative model families most students meet first. A GAN never computes p(x) at all; it only knows how to draw samples that look plausible. A variational autoencoder computes an evidence lower bound (ELBO) on log p(x), which is a guaranteed floor, not the true value — two transactions with identical ELBOs can have different true likelihoods if the bound is looser for one of them. Normalizing flows are the generative model family built specifically to close this gap: they hand you the exact log-density of any point, in closed form, by construction. That guarantee is the entire subject of this chapter, and it costs exactly one architectural constraint: every layer of the network must be an invertible function.
Three ways to model a distribution, and why only one gives an exact number
Every deep generative model answers the same underlying question — given data x, what is p(x), and how do I draw new samples from it — but they trade off exactness for flexibility differently. A GAN's generator G maps noise z to samples x = G(z); sampling is one forward pass, but there is no way to ask "what is p(x) for this exact x", because G is neither invertible nor volume-tracking. A VAE trains an encoder-decoder pair and only ever optimizes a lower bound on log p(x), because the true marginal likelihood requires an intractable integral over the latent variable. A normalizing flow instead insists that the generator itself, f, be a bijection: every z maps to exactly one x and back. That single requirement is what turns "compute the density" from an intractable integral into a closed-form algebraic formula, at the cost of the latent space z having to be exactly as many dimensions as the data x — a flow can reshape a distribution, but it can never compress it the way an autoencoder's bottleneck does.
The change-of-variables formula: the mathematical core of every flow
Start in one dimension for the intuition. Suppose z has a known density p_Z(z) — say the standard normal — and x = f(z) for some smooth, strictly increasing, invertible f. Probability mass is conserved under the transform: the mass in a small interval [z, z+dz] equals the mass in the corresponding interval [x, x+dx]. So p_Z(z) dz = p_X(x) dx, which rearranges to
p_X(x) = p_Z(z) * |dz/dx| = p_Z(f^{-1}(x)) * |d f^{-1}(x)/dx|In D dimensions the single derivative becomes the determinant of the Jacobian matrix of partial derivatives, and the formula generalizes to
p_X(x) = p_Z(f^{-1}(x)) * |det( d f^{-1}(x) / dx )|or, written the way it is actually used during training — going from the known base density outward to the data —
p_X(x) = p_Z(z) * |det( d f(z) / dz )|^{-1}, where z = f^{-1}(x)The determinant term is not decoration; it is the whole mechanism. It measures how much the transformation f locally stretches or squeezes a small volume of space around z. If f stretches space out (determinant magnitude greater than 1), the same probability mass spreads over a larger region, so the density there must drop, which is exactly why the determinant appears as a divisor. Working in log-space, which is what every real implementation does for numerical stability, the training objective for a single data point becomes
log p_X(x) = log p_Z(z) - log|det( d f(z) / dz )|This is an exact equality, not a bound. Everything else in this chapter is about making the two quantities on the right — a simple Gaussian log-density and a Jacobian log-determinant — cheap to compute for a function f expressive enough to be useful.
Why the Jacobian determinant is the entire engineering problem
For an arbitrary invertible neural network mapping R^D → R^D, the Jacobian is a dense D×D matrix, and computing a determinant of a dense matrix costs O(D^3) via LU decomposition. For a modest image with D = 3072 dimensions (32×32×3), that is on the order of 3072^3 ≈ 2.9×10^10 operations per data point, per layer, per training step — completely infeasible. Every real normalizing-flow architecture is a scheme for designing f so that its Jacobian is triangular, because the determinant of a triangular matrix is just the product of its diagonal entries: O(D) instead of O(D^3).
The affine coupling layer, introduced in the RealNVP architecture (Dinh, Sohl-Dickstein & Bengio, 2016) and reused in Glow (Kingma & Dhariwal, 2018), is the standard way to build such a triangular Jacobian. Split the D-dimensional input into two blocks, z_A and z_B. Leave the first block untouched and use it, unrestricted, to compute a scale and a shift applied to the second block:
x_A = z_A
x_B = z_B ⊙ exp(s(z_A)) + t(z_A)Here s(·) and t(·) can be arbitrary neural networks — they never need to be inverted, because z_A is copied straight through as x_A, and inversion only ever needs to run them forward:
z_A = x_A
z_B = (x_B - t(z_A)) ⊙ exp(-s(z_A))Because x_A depends only on z_A and not at all on z_B, the Jacobian of the whole map is block lower-triangular: the top-left block is the identity, the bottom-left block is whatever mess ∂x_B/∂z_A works out to be, and the bottom-right block is the diagonal matrix diag(exp(s(z_A))). A triangular block matrix with an identity block and a diagonal block has determinant equal to the product of the diagonal block's entries:
det(J) = exp( sum(s(z_A)) )No matrix inversion, no O(D^3) determinant — just a sum over the entries of s(z_A), which the network already computed on the forward pass.
Worked example: one coupling layer, every number traced
Take a 2-dimensional flow so every step is checkable by hand. Split z = (z_1, z_2) with z_A = z_1, z_B = z_2, and use the simplest possible scale/shift networks, linear functions of z_1:
s(z_1) = 0.5 * z_1
t(z_1) = z_1 + 1Sample z = (1, 2) from the base distribution, a standard 2D Gaussian N(0, I).
Step 1 — forward pass. s(1) = 0.5, t(1) = 2. Then x_1 = z_1 = 1, and x_2 = z_2 · exp(s(z_1)) + t(z_1) = 2 · exp(0.5) + 2. Since exp(0.5) = 1.648721, x_2 = 2 × 1.648721 + 2 = 5.297443. So x = (1, 5.297443).
Step 2 — Jacobian determinant. Because x_1 depends only on z_1 and x_2's dependence on z_2 is the single multiplicative term exp(s(z_1)), det(J) = exp(s(z_1)) = exp(0.5) = 1.648721, and log|det(J)| = 0.5 exactly — no computation beyond reading off s(z_1).
Step 3 — base log-density. The standard 2D Gaussian density is p_Z(z) = (1/2π) exp(-(z_1² + z_2²)/2). With z_1²+z_2² = 1+4 = 5: log p_Z(z) = -0.5×5 - log(2π) = -2.5 - 1.837877 = -4.337877, so p_Z(z) = exp(-4.337877) = 0.013064.
Step 4 — data log-density via change of variables. log p_X(x) = log p_Z(z) - log|det(J)| = -4.337877 - 0.5 = -4.837877, giving p_X(x) = exp(-4.837877) = 0.0079239. Checking the non-log form independently: p_X(x) = p_Z(z) / |det(J)| = 0.013064 / 1.648721 = 0.0079239. Both routes agree exactly, as they must.
Here is the same computation as executable code, including the inverse, which must recover the original z exactly since f is a bijection:
import math
def s(z1): return 0.5 * z1
def t(z1): return z1 + 1.0
def coupling_forward(z1, z2):
sv, tv = s(z1), t(z1)
x1 = z1
x2 = z2 * math.exp(sv) + tv
log_det = sv # det(J) = exp(s(z1))
return x1, x2, log_det
def coupling_inverse(x1, x2):
z1 = x1
z2 = (x2 - t(z1)) * math.exp(-s(z1))
return z1, z2
z1, z2 = 1.0, 2.0
x1, x2, log_det = coupling_forward(z1, z2)
print(x1, x2, log_det)
# 1.0 5.297442541400256 0.5
log_pz = -0.5 * (z1**2 + z2**2) - math.log(2 * math.pi)
log_px = log_pz - log_det
print(log_pz, log_px, math.exp(log_px))
# -4.337877066409345 -4.837877066409345 0.007923858032799689
print(coupling_inverse(x1, x2))
# (1.0, 2.0) -- exact recovery, confirming f is a bijectionThe printed values match the hand derivation exactly, and the inverse pass recovers (1.0, 2.0) to floating-point precision, which is the operational definition of invertibility: nothing was thrown away.
Stacking layers: composition, alternating masks, and the training objective
A single coupling layer is weak: it leaves z_A completely untouched, so half the dimensions never get transformed at all if you stop there. Real flows chain many coupling layers, f = f_K ∘ f_{K-1} ∘ ⋯ ∘ f_1, and alternate which half is "frozen" and which half is "transformed" at each layer — RealNVP does this with a checkerboard or channel mask that flips between layers. After enough alternations, every dimension has been transformed conditioned on every other dimension at some point in the stack, which is what gives the flow enough expressive power to warp a simple Gaussian into a genuinely complex, multimodal data distribution.
Composing invertible functions keeps the whole chain invertible, and log-determinants simply add, because det(J_1 J_2) = det(J_1) · det(J_2):
log p_X(x) = log p_Z(z_0) - Σ_{k=1}^{K} log|det( d f_k(z_{k-1}) / d z_{k-1} )|Training a flow is then direct maximum-likelihood estimation: for every data point, run the inverse pass x → z_0 through the whole stack, accumulate the sum of log-determinants along the way, plug z_0 into the known base density, and take the gradient of this exact log-likelihood with respect to every network parameter. There is no ELBO, no reparameterization trick, no adversarial min-max game — the loss the network optimizes is precisely the quantity you eventually want to report as the anomaly score.
Common misconception
Students who have already met VAEs tend to import that model's mental picture wholesale: "normalizing flows are just VAEs where the decoder happens to be reversible, so I still get an approximate likelihood." This is wrong in a way that matters. A VAE's decoder is not required to be invertible or dimension-preserving, so the marginal likelihood p(x) = ∫ p(x|z) p(z) dz has no closed form and training instead maximizes a provably-lower bound, the ELBO — the true log-likelihood is always at least as large as what you optimized, and by an unknown amount. A normalizing flow's log p_X(x) computed via the change-of-variables formula is not a bound on anything; it is the exact value, because the change-of-variables identity is an equality derived from conservation of probability mass, not an inequality derived from Jensen's inequality the way the ELBO is. The price for that exactness is the constraint the VAE does not have: the flow's latent dimension must equal the data dimension, and every layer must be an exact bijection. If a task genuinely needs a lower-dimensional latent code (compression, disentangled representation learning), a flow is the wrong tool regardless of how attractive exact likelihood sounds; if a task needs a trustworthy exact density number to threshold against, a VAE's bound is the wrong tool no matter how good its samples look.
Where this shows up outside the classroom
Glow (Kingma & Dhariwal, 2018) stacked exactly this coupling-layer machinery, with an added learned 1×1 invertible convolution to mix channels between coupling steps, to generate high-resolution face images and to perform exact latent-space interpolation and attribute manipulation — operations that need the model to invert real images back to z exactly, which only a flow can do losslessly. WaveGlow (NVIDIA, 2019) applied the same coupling-layer idea to raw audio waveforms to build a text-to-speech vocoder, the component that turns a predicted spectrogram into an audible waveform, and is the direct ancestor of flow-based vocoders used in many production speech systems, including Indian-language TTS pipelines that need to generate audio faster than real time. And the anomaly-scoring use case that opened this chapter is a live research and industry direction: flow-based density estimators trained on legitimate transaction feature vectors give a calibrated, exact p(x) that a rules engine can threshold directly, without the false confidence of a GAN's realism score or the systematic underestimate baked into a VAE's ELBO.
The diagram: how density and samples move in opposite directions through the same layers
Active recall
Attempt every question before reading its answer below.
Q1. A coupling layer sets x_A = z_A and x_B = z_B ⊙ exp(s(z_A)) + t(z_A). Explain, without computing anything, why the Jacobian of this map is triangular and why that makes the determinant cheap.
Q2. Using s(z_1) = 0.3 z_1 and t(z_1) = z_1 - 1, compute the forward output x and the log-determinant for z = (2, -1).
Q3. Why can a normalizing flow never reduce dimensionality the way a VAE's bottleneck does, even though both models have an encoder-like and decoder-like direction?
Q4. For a data point with D = 3072 dimensions, roughly how many operations does computing a dense Jacobian determinant cost via general methods, and why does this force the coupling-layer design instead of a free-form invertible MLP?
Q5. A classmate says "VAEs and normalizing flows both train by maximizing something called a likelihood, so their reported numbers should be directly comparable." What is wrong with that claim?
Q6. If every coupling layer in a stack used the same fixed split (say, dimensions 1–50 always frozen as z_A, dimensions 51–100 always transformed as z_B), what would break, and why do real architectures alternate the mask between layers?
Worked answers
A1. x_A is a direct copy of z_A and does not depend on z_B at all, so the block ∂x_A/∂z_B is the zero matrix. That makes the full Jacobian block lower-triangular: identity in the top-left, an arbitrary block in the bottom-left (∂x_B/∂z_A), zero in the top-right, and the diagonal matrix diag(exp(s(z_A))) in the bottom-right. The determinant of a triangular matrix is the product of its diagonal entries only, so the entire messy bottom-left block is irrelevant to the determinant — you never need to differentiate s or t with respect to z_A to get det(J), you just read off exp(sum(s(z_A))), an O(D) computation the network already produces on its forward pass.
A2. s(2) = 0.6, t(2) = 1. x_1 = z_1 = 2. x_2 = z_2·exp(s(z_1)) + t(z_1) = -1 × exp(0.6) + 1 = -1 × 1.822119 + 1 = -0.822119. So x = (2, -0.822119). The log-determinant is simply s(z_1) = 0.6, since det(J) = exp(s(z_1)) = 1.822119 here as well.
A3. Invertibility is a bijection: every point in the domain maps to exactly one point in the codomain and back, with no information discarded. A function R^D → R^m with m < D cannot be injective (multiple D-dimensional inputs must collide onto the same lower-dimensional output by a pigeonhole argument), so it cannot have an inverse, so the change-of-variables formula — which requires f^{-1} to exist — simply does not apply. A VAE's encoder is explicitly allowed to be non-injective and lossy (that is what makes it a "bottleneck" doing compression); a flow's f is required to preserve exactly all the information in z, which forces dim(z) = dim(x).
A4. A dense D×D Jacobian's determinant costs O(D^3) via LU decomposition; for D=3072 that is on the order of 3072^3 ≈ 2.9×10^10 operations, repeated for every data point and every layer of every training step — not tractable at any reasonable batch size. A free-form invertible MLP gives no structural guarantee that its Jacobian is anything but dense, so it inherits this cost. Coupling layers sidestep it entirely by construction: the copy-through half forces a triangular Jacobian regardless of how complex s and t are internally, dropping the cost to O(D).
A5. A VAE's training objective, the ELBO, is a provable lower bound on log p(x), derived from Jensen's inequality applied to the intractable marginalization over the latent variable; the gap between the ELBO and the true log-likelihood is generally unknown and varies across data points and across models. A normalizing flow's log p_X(x) from the change-of-variables formula is not a bound at all — it is exactly equal to the true density under the model, derived from conservation of probability mass, an identity rather than an inequality. Two numbers where one is a guaranteed-loose lower bound and the other is exact are not comparable at face value; a flow scoring lower than a VAE on "likelihood" could still represent a strictly better density model, because the VAE's number is artificially depressed by however loose its bound happens to be.
A6. With a fixed split, dimensions 1–50 (z_A) are copied straight through, x_A = z_A, in every single layer of the stack, no matter how many layers are chained. Those 50 dimensions would never be conditioned on information from dimensions 51–100, so the model could never represent any dependency running in that direction, and half the data's structure would be architecturally unreachable regardless of network depth. Alternating the mask (transform A conditioned on B in one layer, then transform B conditioned on the now-updated A in the next) lets information flow both ways across the stack, so that after enough layers every dimension has, at some point, been transformed as a function of every other dimension.
Think About It
Think about this: How would you explain normalizing flows: invertible transformations to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where normalizing flows: invertible transformations is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting normalizing flows: invertible transformations to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind normalizing flows: invertible transformations, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.