Energy-Based Models: Unnormalized Distributions

A fraud desk that never computes a probability

A bank running UPI fraud detection scores every transaction the instant it lands: odd hour, first-time payee, amount far above the account's usual range, unfamiliar device. The engine does not ask "what is the probability of this exact transaction among every transaction that could ever occur?" That question has no usable answer — the space of possible transactions is effectively unbounded, and no one has enumerated it, let alone normalized a probability distribution over it. What the engine actually computes is a single number per transaction: a score where ordinary behaviour comes out low and suspicious behaviour comes out high. Transactions get ranked by that score, a threshold cuts off the risky tail, and the system works perfectly well without ever having computed a true probability for anything.

That score is an energy function, and the fraud engine is — whether or not its builders used the term — an energy-based model. This chapter makes that idea precise: how to define a probability distribution through a scalar energy function, why the normalizing constant of that distribution is usually impossible to compute, and why, for most of the things you actually want to do with the model, that turns out not to matter.

Borrowing a law of physics for probability

You have already met this shape of idea in physics: a ball on a hillside settles into the valley, the lowest-potential-energy configuration, because low-energy states are the stable, likely ones and high-energy states are unstable, rare ones. Statistical mechanics turns "likely" into an exact number with the Boltzmann distribution: at thermal equilibrium, the probability that a physical system occupies microstate x is proportional to exp(-E(x)/kT), where E(x) is the state's energy, k is Boltzmann's constant, and T is temperature. Divide by whatever normalizing constant makes the total probability equal to 1, and you have a genuine distribution.

Energy-based models in machine learning lift this construction wholesale, dropping the physical constants (they get absorbed into how E is parameterized). Given any scalar function E(x; θ) — the "energy function," typically a neural network that eats a data point x and outputs one real number — define:

p(x; θ) = exp(-E(x; θ)) / Z(θ)
Z(θ) = Σ_x exp(-E(x; θ))          (discrete x)
Z(θ) = ∫ exp(-E(x; θ)) dx          (continuous x)

Z(θ) is called the partition function — a term borrowed directly from statistical mechanics, where it "partitions" probability mass across all states. exp(-E(x; θ)) by itself is called the unnormalized density: it is positive everywhere (because the exponential of any real number is positive) but does not sum or integrate to 1 on its own. Dividing by Z(θ) is what makes it a legitimate probability distribution.

The appeal of this construction is architectural freedom. A Gaussian must have a positive-definite covariance matrix for its formula to integrate to 1. A softmax-based categorical distribution must have its logits routed through a normalized exponential. Every classical parametric distribution carries this same burden: its functional form is constrained in advance so that normalization comes out to exactly 1 automatically. An energy-based model throws that constraint away. E(x; θ) can be any neural network at all — deep, convolutional, with skip connections, outputting any real number, positive or negative, unbounded. There is no constraint to enforce, because exp(-E(x)) is automatically positive for any real-valued E, and dividing by Z(θ) — whatever it turns out to be — automatically fixes the normalization. You get to design the energy landscape with total architectural freedom. The price is that Z(θ), unlike in a Gaussian, generally has no closed form and no cheap way to compute.

The wall: why Z resists computation

To find Z(θ) exactly for a discrete model, you must evaluate exp(-E(x; θ)) at every possible configuration of x and sum them. If x is a vector of n independent binary features (the fraud engine's flags: odd-hour yes/no, new-payee yes/no, and so on), there are 2ⁿ configurations to enumerate. That is fine at n = 3 — eight terms, done by hand in a minute. It is still technically fine at n = 20 — a modern feature vector for a fraud model might easily carry that many binary signals — but now 2²⁰ = 1,048,576 terms need to be summed for a single evaluation of Z, and Z would need to be recomputed every time θ changes during training. Push to n = 30 features and it is 2³⁰ = 1,073,741,824 terms — over a billion, per gradient step. For continuous x, such as a 784-dimensional vector of grayscale pixel values, Z(θ) is a 784-dimensional integral with no closed form for a general neural-network energy — not slow to compute, but not computable by any exact method at all.

This is the sense in which the title of this chapter matters: an energy-based model works natively with an unnormalized distribution. exp(-E(x; θ)) is available instantly, from one forward pass of the network. Z(θ), the thing needed to turn that into an actual probability, is the part that is generally out of reach.

Worked example: a 3-signal energy function, fully enumerated

Build a tiny, fully computable energy function for a transaction with three binary risk flags: x₁ = odd hour, x₂ = new payee, x₃ = large amount. Define:

E(x1, x2, x3) = 1.0·x1 + 1.0·x2 + 1.5·x3 + 1.5·x1·x2

The weight on each flag is its individual contribution to "unusualness," and the 1.5·x1·x2 cross term is an interaction penalty: an odd-hour transaction to a brand-new payee is riskier than the sum of its two parts, so the energy compounds when both fire together. Low energy should mean an ordinary transaction (high probability under the model); high energy should mean an unusual combination of flags (low probability).

Enumerate all 2³ = 8 configurations, compute the unnormalized density u(x) = exp(-E(x)) for each, and sum to get Z:

x1 x2 x3   E(x)   u(x)=exp(-E)    p(x)=u(x)/Z
 0  0  0   0.0    1.000000        0.462964
 0  0  1   1.5    0.223130        0.103301
 0  1  0   1.0    0.367879        0.170315
 0  1  1   2.5    0.082085        0.038002
 1  0  0   1.0    0.367879        0.170315
 1  0  1   2.5    0.082085        0.038002
 1  1  0   3.5    0.030197        0.013980
 1  1  1   5.0    0.006738        0.003119
                              Z = 2.159994

Every step of this table can be reproduced exactly in code, which is the only trustworthy way to check an enumeration like this — hand arithmetic across eight rows is exactly where a sign error hides:

import math

def E(x1, x2, x3, w1=1.0, w2=1.0, w3=1.5, w12=1.5):
    return w1*x1 + w2*x2 + w3*x3 + w12*x1*x2

states = [(a, b, c) for a in (0, 1) for b in (0, 1) for c in (0, 1)]
u = {s: math.exp(-E(*s)) for s in states}
Z = sum(u.values())
p = {s: u[s] / Z for s in states}

print(round(Z, 6))          # 2.159994
print(round(p[(0,0,0)], 6)) # 0.462964  -- most ordinary pattern, highest probability
print(round(p[(1,1,1)], 6)) # 0.003119  -- every flag fires, lowest probability
print(round(sum(p.values()), 6))  # 1.0

The all-clear transaction (0,0,0) gets 46.3% of the probability mass; the transaction that trips every flag at once gets 0.31% — roughly 148 times less likely. Notice, too, that computing Z here required visiting all eight states. That is exactly the operation that becomes a billion-term sum at n = 30. Nothing about the formula changes between n = 3 and n = 30 — only the cost of the sum.

Two escape routes that never touch Z

Route 1 — ratios. The fraud engine never needs p(x) in isolation; it needs to compare or rank transactions, and comparison is a ratio. Take the log-ratio of two configurations:

log p(a) - log p(b) = [-E(a) - log Z] - [-E(b) - log Z] = E(b) - E(a)

log Z appears in both terms and cancels exactly — it never needs to be evaluated. Check this against the worked table: p(1,1,1) / p(0,0,0) = exp(-5.0 + 0.0) = exp(-5.0) = 0.006738, matching 0.003119 / 0.462964 = 0.006737 from the normalized values, computed without ever forming Z. Any system that ranks candidates by relative likelihood — fraud scoring, ranking retrieved documents, comparing candidate completions — is Route 1 in disguise.

Route 2 — gradients with respect to x. Take the gradient of log p(x; θ) with respect to the input x itself, holding θ fixed:

∇x log p(x; θ) = ∇x [ -E(x; θ) - log Z(θ) ] = -∇x E(x; θ)

log Z(θ) depends only on θ, not on x, so its gradient with respect to x is the zero vector and it disappears entirely — not cancelled between two terms, simply absent. This quantity, ∇x log p(x; θ) = -∇x E(x; θ), is called the score, and it is computable directly from the energy network with one backward pass, for any x, without Z ever entering the calculation. The score is what powers sampling: Langevin dynamics generates a sample from p(x; θ) by repeatedly nudging a point downhill along the energy surface and adding a little noise to keep it from collapsing into the single deepest minimum —

x_{t+1} = x_t - (ε/2)·∇x E(x_t; θ) + sqrt(ε)·ξ_t,     ξ_t ~ N(0, I)

— and every step of that recursion depends only on ∇x E. This is the same mechanism, generalized to continuous, high-dimensional x, that underlies score-based diffusion models — an energy-based model with an unadjusted Langevin sampler is their direct ancestor.

Training an EBM: turning the intractable integral into an expectation

Training still has to move θ, and θ is exactly where Z(θ) lives — Route 2 does not save you here, because now the derivative is with respect to θ, not x, and log Z(θ) is not constant in θ. Differentiate directly:

log p(x; θ) = -E(x; θ) - log Z(θ)
∂/∂θ log p(x; θ) = -∂E(x; θ)/∂θ - ∂/∂θ log Z(θ)

Expand the second term using ∂/∂θ log Z = (1/Z)·∂Z/∂θ:

∂Z(θ)/∂θ = Σ_x ∂/∂θ exp(-E(x;θ)) = Σ_x exp(-E(x;θ))·(-∂E(x;θ)/∂θ)

∂/∂θ log Z(θ) = (1/Z)·Σ_x exp(-E(x;θ))·(-∂E/∂θ) = Σ_x p(x;θ)·(-∂E(x;θ)/∂θ) = -E_{x~p(θ)}[∂E(x;θ)/∂θ]

The intractable sum over all x has turned into an expectation under the model's own distribution. Substituting back:

∂/∂θ log p(x; θ) = -∂E(x; θ)/∂θ + E_{x'~p(θ)}[ ∂E(x'; θ)/∂θ ]

This is the entire training signal for an energy-based model, and it has a direct reading: to increase the log-likelihood of a real data point x, push θ in the direction that lowers the energy at x (the first term) while raising the energy at points sampled from the model itself (the second term, the expectation). The real data pulls the energy surface down where it stands; samples drawn from the model — the "negative samples" — get pushed back up wherever the model currently places too much probability mass. Since x' is drawn from p(θ) using exactly the Langevin sampler from Route 2, no exact Z is ever computed during training either — an intractable sum has been replaced by a Monte-Carlo estimate obtained by sampling. This is the core idea behind contrastive divergence, the standard training method for energy-based models.

The derivation is checkable on the worked example. Take θ = w3, the weight on the "large amount" flag, currently 1.5. Since E is linear in w3 with coefficient x3, ∂E/∂w3 = x3, so the formula predicts ∂ log Z/∂w3 = -E_p[x3]. Summing p(x) over the four states with x3 = 1 gives E_p[x3] = 0.182426, so the analytic derivative is -0.182426. A finite-difference check — perturbing w3 by 0.0001 and recomputing log Z from scratch — gives -0.182418, matching to four significant figures, exactly the agreement expected from a first-order numerical approximation:

import math

def E(x1, x2, x3, w3):
    return 1.0*x1 + 1.0*x2 + w3*x3 + 1.5*x1*x2

states = [(a, b, c) for a in (0, 1) for b in (0, 1) for c in (0, 1)]

def logZ(w3):
    return math.log(sum(math.exp(-E(*s, w3=w3)) for s in states))

# analytic: -E_p[x3] at w3 = 1.5
p = {s: math.exp(-E(*s, w3=1.5)) for s in states}
Z = sum(p.values())
Ep_x3 = sum((p[s]/Z) * s[2] for s in states)
print(round(-Ep_x3, 6))                     # -0.182426

# finite difference
d = 1e-4
print(round((logZ(1.5+d) - logZ(1.5)) / d, 6))  # -0.182418

The misconception: "you must compute Z to use the model"

Myth: since p(x; θ) = exp(-E(x; θ))/Z(θ) is the formula for the model's probability, every use of the model — scoring, sampling, training — must involve computing Z(θ) at some point, and since Z is intractable, energy-based models must be intractable to use.

Reality: Z(θ) is a single number depending only on θ — it does not depend on x at all. Every genuinely useful operation on an energy-based model either compares two values of x (Route 1: the ratio, where log Z cancels because it is the same additive constant on both sides) or differentiates with respect to x (Route 2: the score, where log Z vanishes because its derivative with respect to x is exactly zero). The one operation that genuinely depends on the value of Z — training, differentiating with respect to θ — turns out not to need the value of Z either, only an expectation under the model, which is estimated by sampling rather than computed exactly. The fraud engine that opened this chapter never once forms Z; it lives entirely on Route 1. What actually has to be tractable is not Z, but ∇x E — one backward pass through a neural network.

How the pieces connect

Active recall

Attempt every question before reading the answer beneath it.

1. A 2-bit energy function is E(x1, x2) = 2x1 + x2 − x1x2. Compute Z and p(1,1).
2. Explain, in one derivation, why p(a)/p(b) for an energy-based model never requires knowing Z(θ).
3. True or false: an energy function E(x; θ) must always output a non-negative number, otherwise p(x; θ) is not a valid distribution. Justify your answer.
4. For the model in Q1, compute ∂ log Z/∂w1 at w1 = 2, where w1 is the coefficient of x1.
5. A fraud model uses 30 independent binary flags. How many terms would exact enumeration of Z require?
6. Why is ∇x E(x; θ) called the "score," and why does it not depend on Z(θ) at all, even though p(x;θ) is defined using Z(θ)?

Answers

1. Enumerate all four states: (0,0): E=0, u=1.000000; (0,1): E=1, u=0.367879; (1,0): E=2, u=0.135335; (1,1): E=2·1+1−1·1=2, u=0.135335. Z = 1.000000+0.367879+0.135335+0.135335 = 1.638550. p(1,1) = 0.135335/1.638550 = 0.082595.

2. log p(a) − log p(b) = [−E(a) − log Z(θ)] − [−E(b) − log Z(θ)]. The two log Z(θ) terms are identical (they depend only on θ, not on which state is being evaluated), so they subtract to zero, leaving log p(a) − log p(b) = E(b) − E(a) — computable from the energy function alone, at any two points, without ever evaluating Z.

3. False. E(x; θ) can take any real value, positive or negative — the 2-bit example above even used a negative coefficient in one term. The only thing required for p(x;θ) to be a valid distribution is that exp(-E(x;θ)) sums or integrates to a finite, positive Z(θ); that is a condition on the total mass, not on the sign of any individual energy value. Students who assume "energy" must mean "non-negative" are importing an intuition from potential energy that the formula never actually requires.

4. ∂E/∂w1 = x1, so ∂ log Z/∂w1 = −E_p[x1] = −(p(1,0) + p(1,1)) = −(0.082595 + 0.082595) = −0.165191.

5. 2³⁰. Building it up: 2¹⁰ = 1024, so 2²⁰ = 1024² = 1,048,576, and 2³⁰ = 1,048,576 × 1024 = 1,073,741,824 — over a billion terms for a single evaluation of Z, from just 30 binary features.

6. ∇x log p(x;θ) = ∇x[−E(x;θ) − log Z(θ)] = −∇x E(x;θ) − ∇x log Z(θ). Since log Z(θ) is a function of θ alone and does not vary as x is perturbed, its gradient with respect to x is the zero vector, term by term — not approximately negligible, exactly zero. So the gradient of the log-density equals the negative gradient of the energy, both are called the "score" of the distribution, and both are obtainable from a single backward pass through the energy network regardless of whether Z(θ) is known.

Think About It

Think about this: How would you explain energy-based models: unnormalized distributions to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where energy-based models: unnormalized distributions is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting energy-based models: unnormalized distributions to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind energy-based models: unnormalized distributions, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.