Information Theory: Entropy and KL Divergence

Every time you set up a UPI PIN or an ATM PIN in India, you choose one code out of exactly 10,000 possibilities — 0000 through 9999. Banking apps like to describe this as security: "10,000 combinations." But 10,000 is an odd unit for security. Nobody says a padlock is "10,000 combinations secure." Security is usually measured in the number of attempts an attacker needs, or equivalently, in the number of bits of uncertainty they must overcome. That gap — between a raw count of possibilities and a precise, comparable measure of how much protection you actually have — is exactly what information theory was built to close.

Information theory answers a second, closely related question too, one that shows up every time an app tries to predict something about you: a bank's fraud model deciding whether a transaction looks genuine, a live cricket broadcast updating its win-probability meter after every ball, or a language model guessing the next word you are about to type. Every one of these systems works by producing a probability distribution — a set of possible outcomes, each with a confidence attached. None of them are ever exactly right. Information theory gives two precise tools for reasoning about that: entropy, which measures how much uncertainty is baked into a distribution in the first place, and KL divergence, which measures how far one distribution has drifted from another. Entropy comes from Claude Shannon's 1948 paper "A Mathematical Theory of Communication," which founded the field; KL divergence was added three years later by Solomon Kullback and Richard Leibler. Both now sit inside the loss function of nearly every classifier you will train this year, usually without you ever noticing.

Measuring Surprise: From Questions to Bits

Start with the simplest uncertain event possible: a fair coin toss. Before the flip there are 2 equally likely outcomes. The moment you learn the result, you have gained exactly enough information to tell those 2 outcomes apart. Shannon defined this amount of information as 1 bit — short for "binary digit." A single yes/no question ("did it land heads?") resolves it completely.

Scale up to a fair six-sided die: 6 equally likely outcomes. Two yes/no questions can distinguish at most 4 outcomes — not enough. Three questions distinguish up to 8 — more than enough, with some waste. The exact answer, allowing fractional bits, is log₂(6) ≈ 2.585 bits. In general, identifying which one of N equally likely outcomes occurred takes log₂(N) bits. This is exactly why base-2 logarithms appear in the first place: doubling the number of equally likely outcomes should cost exactly one more yes/no question, and log₂(2N) = log₂(N) + 1 guarantees precisely that.

Return to the PIN. If all 10,000 four-digit codes were equally likely, identifying which one you picked would take log₂(10,000) ≈ 13.29 bits — the maximum possible, since a uniform distribution over a fixed number of outcomes always carries the most uncertainty that distribution can hold. Real PIN choices are not uniform. People overwhelmingly gravitate toward memorable patterns — repeated digits, birth years, sequences like 1234 — instead of spreading evenly across the full range. Studies of large, leaked PIN datasets have repeatedly confirmed this skew: a small cluster of "obvious" choices accounts for a hugely disproportionate share of real PINs, while thousands of other four-digit codes are almost never picked. Since entropy is maximized only by a perfectly uniform distribution, that skew means the true entropy of how people actually choose PINs sits well below the 13.29-bit ceiling. That is exactly why an attacker who knows the real distribution needs far fewer than 10,000 guesses on average, and exactly why setup screens steer you away from "obvious" PINs.

Equally likely outcomes are the easy case. Real distributions are rarely uniform — a biased coin, a loaded die, a skewed PIN distribution. Shannon's key move was to define the self-information (sometimes called "surprisal") carried by a single outcome x with probability P(x) as I(x) = −log₂ P(x). This single formula captures something intuitive: rare events are surprising, and surprising events carry more information. If P(x) = 1, the outcome is certain and I(x) = −log₂(1) = 0. Learning that a sure thing happened tells you nothing new. As P(x) shrinks toward 0, I(x) grows without bound. This is why "man bites dog" is a headline and "dog bites man" is not: the newsworthiness of an event tracks its self-information almost exactly.

Entropy: The Average Surprise

Entropy is what you get by averaging self-information across an entire distribution, weighted by how often each outcome actually occurs:

H(X) = −Σ P(x) log₂ P(x)

This is the expected number of bits needed, on average, to communicate the outcome of X (the expected value of the self-information, E[I(X)]). For the fair coin, where P(heads) = P(tails) = 0.5:

H = −(0.5 × log₂ 0.5 + 0.5 × log₂ 0.5) = −(0.5 × −1 + 0.5 × −1) = 1 bit

That matches the "one yes/no question" intuition exactly. Now bias the coin heavily, say P(heads) = 0.9 and P(tails) = 0.1:

H = −(0.9 × log₂ 0.9 + 0.1 × log₂ 0.1) = −(0.9 × −0.1520 + 0.1 × −3.3219) ≈ 0.469 bits

Less than half a bit, because a heavily biased coin is far less surprising on average: most flips land heads exactly as expected, and only the occasional tail carries real information. Entropy is maximized by uniform distributions and shrinks toward zero as a distribution becomes more predictable, hitting exactly 0 when one outcome has probability 1 and every other outcome has probability 0 — a distribution with no uncertainty left to resolve.

Worked Example: Entropy of a Bowler's Deliveries

Consider a simplified analytics model for a T20 bowler, bucketing every delivery into one of four outcomes based on the bowler's historical record: a dot ball, a run (a single or double), a boundary (a four or six), or a wicket. Suppose the historical data gives this true distribution, P:

• P(dot) = 0.40
• P(run) = 0.40
• P(boundary) = 0.15
• P(wicket) = 0.05

These four probabilities sum to 1.00, as any valid distribution must. Applying H(P) = −Σ P(x) log₂ P(x) term by term:

• Dot ball: −0.40 × log₂(0.40) = −0.40 × (−1.3219) = 0.5288
• Run: −0.40 × log₂(0.40) = −0.40 × (−1.3219) = 0.5288
• Boundary: −0.15 × log₂(0.15) = −0.15 × (−2.7370) = 0.4105
• Wicket: −0.05 × log₂(0.05) = −0.05 × (−4.3219) = 0.2161

Summing the four terms gives H(P) = 0.5288 + 0.5288 + 0.4105 + 0.2161 ≈ 1.6842 bits. That is noticeably less than the 2-bit maximum a uniform distribution over 4 outcomes would give (log₂ 4 = 2), because this bowler's deliveries are not spread evenly across the four buckets: dot balls and runs dominate, which makes the outcome somewhat predictable.

When the Model Doesn't Know the Truth: Cross-Entropy

Entropy measures uncertainty when you already know the true distribution. A prediction model almost never has direct access to P; it only has a learned approximation, call it Q. Suppose an analytics model, trained on limited recent data, predicts this distribution for the same bowler's next delivery:

• Q(dot) = 0.35
• Q(run) = 0.45
• Q(boundary) = 0.10
• Q(wicket) = 0.10

Q is close to P but not identical: it underestimates dot balls and boundaries, and overestimates runs and wickets. Cross-entropy measures the average number of bits actually spent encoding outcomes drawn from the true distribution P, if the coding scheme were built assuming Q instead:

H(P, Q) = −Σ P(x) log₂ Q(x)

Working through the same four outcomes:

• Dot ball: −0.40 × log₂(0.35) = −0.40 × (−1.5146) = 0.6058
• Run: −0.40 × log₂(0.45) = −0.40 × (−1.1520) = 0.4608
• Boundary: −0.15 × log₂(0.10) = −0.15 × (−3.3219) = 0.4983
• Wicket: −0.05 × log₂(0.10) = −0.05 × (−3.3219) = 0.1661

Summing gives H(P, Q) = 0.6058 + 0.4608 + 0.4983 + 0.1661 ≈ 1.7310 bits. Notice this is larger than H(P) ≈ 1.6842 bits, and that gap is never an accident: cross-entropy is always at least as large as the true entropy, H(P, Q) ≥ H(P), with equality only when Q exactly matches P. Encoding outcomes using the wrong distribution never saves bits — it only ever costs extra ones.

KL Divergence: The Cost of Getting It Wrong

That gap between cross-entropy and true entropy has a name: the Kullback–Leibler divergence, or KL divergence, introduced by Solomon Kullback and Richard Leibler in 1951 as a general measure of how one probability distribution differs from another. It can be written two equivalent ways:

D_KL(P||Q) = H(P, Q) − H(P)

or, expanded directly:

D_KL(P||Q) = Σ P(x) log₂ (P(x) / Q(x))

Both give the same answer. Using the subtraction form with the numbers already computed: D_KL(P||Q) = 1.7310 − 1.6842 = 0.0468 bits. Confirming with the direct sum:

• Dot ball: 0.40 × log₂(0.40 / 0.35) = 0.40 × 0.1926 = 0.0771
• Run: 0.40 × log₂(0.40 / 0.45) = 0.40 × (−0.1699) = −0.0680
• Boundary: 0.15 × log₂(0.15 / 0.10) = 0.15 × 0.5850 = 0.0877
• Wicket: 0.05 × log₂(0.05 / 0.10) = 0.05 × (−1.0000) = −0.0500

Summing: 0.0771 − 0.0680 + 0.0877 − 0.0500 ≈ 0.0468 bits, matching the subtraction method exactly. The analytics model wastes about 0.0468 bits per delivery, on average, purely because its predicted distribution does not exactly match reality. It is always true that D_KL(P||Q) ≥ 0, a result known as Gibbs' inequality, and D_KL(P||Q) = 0 only when P and Q are identical everywhere. KL divergence can never be negative — a model cannot "save" bits by predicting incorrectly.

Why It Is Called a Divergence, Not a Distance

It is tempting to read D_KL(P||Q) as "the distance between P and Q," but that word is misleading. A genuine distance — like the distance between two cities — must be symmetric: the distance from A to B equals the distance from B to A. KL divergence is not. Swap the roles and compute D_KL(Q||P) = Σ Q(x) log₂ (Q(x) / P(x)) on the same two delivery distributions using the same method as before, and the result comes out to D_KL(Q||P) ≈ 0.0505 bits — a different number from D_KL(P||Q) ≈ 0.0468 bits. The two directions ask different questions: D_KL(P||Q) asks how many extra bits are wasted if reality is P but the plan was built for Q; D_KL(Q||P) asks the reverse. Because the answers differ, and because KL divergence also fails the triangle inequality that any genuine distance must obey, mathematicians deliberately call it a divergence rather than a distance or a metric.

Checking the Math With Code

The same computation, done in Python with NumPy, confirms every number above and generalizes to any distribution:

import numpy as np

# True distribution over delivery outcomes: dot, run, boundary, wicket
P = np.array([0.40, 0.40, 0.15, 0.05])

# Model's predicted distribution for the same outcomes
Q = np.array([0.35, 0.45, 0.10, 0.10])

def entropy(p):
    return -np.sum(p * np.log2(p))

def cross_entropy(p, q):
    return -np.sum(p * np.log2(q))

def kl_divergence(p, q):
    return np.sum(p * np.log2(p / q))

H_P  = entropy(P)
H_PQ = cross_entropy(P, Q)
D_KL = kl_divergence(P, Q)

print(f"H(P)        = {H_P:.4f} bits")
print(f"H(P, Q)     = {H_PQ:.4f} bits")
print(f"D_KL(P||Q)  = {D_KL:.4f} bits")
print(f"H(P,Q)-H(P) = {H_PQ - H_P:.4f} bits")

# Output:
# H(P)        = 1.6842 bits
# H(P, Q)     = 1.7310 bits
# D_KL(P||Q)  = 0.0468 bits
# H(P,Q)-H(P) = 0.0468 bits

# scipy.stats.entropy computes the same two quantities directly:
# one distribution argument returns Shannon entropy;
# two distribution arguments return KL divergence D_KL(first || second)
from scipy.stats import entropy as scipy_entropy

print(round(scipy_entropy(P, base=2), 4))     # 1.6842
print(round(scipy_entropy(P, Q, base=2), 4))  # 0.0468

Every line traces back to the hand calculation: H_P lands at 1.6842, H_PQ at 1.7310, and both routes to D_KL — the explicit kl_divergence function and the subtraction H_PQ - H_P — agree to four decimal places at 0.0468. The scipy.stats.entropy function, part of a library used throughout production data science pipelines, reproduces the same numbers: called with one distribution it returns Shannon entropy, and called with two it returns KL divergence directly, because the library's authors implemented exactly the formulas derived above.

From Cricket Deliveries to Neural Networks

This exact machinery sits inside almost every classifier you will build. When a neural network is trained to classify images, sentiment, or spam, the standard loss function is cross-entropy loss. The true label is usually one-hot encoded — a distribution P that places probability 1 on the correct class and 0 everywhere else — while the network's final softmax layer outputs a predicted distribution Q over all possible classes. A one-hot distribution has no uncertainty left in it, so its entropy is exactly H(P) = 0. Substitute that into D_KL(P||Q) = H(P, Q) − H(P) and the second term vanishes, leaving D_KL(P||Q) = H(P, Q) exactly. In other words, minimizing cross-entropy loss during training means minimizing the KL divergence between the true labels and the model's predictions. It is the same formula this chapter just built from scratch, computed at every training step, for every example, inside every training loop.

One practical note before opening PyTorch or TensorFlow: this chapter has used log₂ throughout, giving answers in bits, because that matches Shannon's original framing and keeps the "number of yes/no questions" intuition exact. Deep learning libraries almost always use the natural logarithm instead, giving answers in units called nats rather than bits (1 nat ≈ 1.4427 bits, since 1 / ln(2) ≈ 1.4427). The formulas for entropy, cross-entropy, and KL divergence are identical either way: only the unit changes, the same way a distance stays the same physical length whether it is reported in miles or kilometres.

KL divergence shows up well beyond classification loss. Variational autoencoders, a generative model architecture, include a KL divergence term directly in their training objective, pulling a learned latent distribution toward a simple reference distribution so the model's internal representations stay well organised. Language models are commonly scored using perplexity, a number derived directly from cross-entropy (perplexity = 2^H(P,Q) when entropy is measured in bits) that captures, roughly, how many equally likely words the model was choosing between at each step. Production machine learning systems — fraud detection, recommendation engines, credit scoring — routinely compute KL divergence between the distribution of live incoming data and the distribution the model was originally trained on, to catch distribution drift before it silently degrades predictions.

Closing the Loop: From PINs to Fraud Alerts

Return to the UPI PIN this chapter opened with. The raw count — 10,000 possible codes — was never the real security measure; entropy is. A uniformly random PIN carries 13.29 bits of uncertainty, the true ceiling. Real human choices carry less, because people are predictable in ways a uniform distribution assumes they are not. That is precisely why a bank's fraud-detection model cares about the gap between the distribution it assumed and the distribution reality actually follows.

That gap is KL divergence, and it runs quietly behind far more of daily digital life than PIN security alone. A fraud model built around a customer's typical UPI spending pattern — how often they pay auto-rickshaws versus e-commerce sites versus utility bills — encodes that pattern as a probability distribution, the same way this chapter encoded a bowler's deliveries. When a month's actual spending distribution drifts far enough from that baseline, measured in the same bits used throughout this chapter, the system flags it: not because any single transaction looked suspicious on its own, but because the whole distribution stopped matching what was expected. Entropy measures how much uncertainty a distribution holds. KL divergence measures how far two distributions have grown apart. Between them, they quantify something every intelligent system, from a bank's fraud engine to the neural network you train next, ultimately depends on: knowing, in exact bits, how wrong its current model of the world is.

Think About It

Think about this: How would you explain information theory: entropy and kl divergence to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where information theory: entropy and kl divergence is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting information theory: entropy and kl divergence to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind information theory: entropy and kl divergence, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.