Information Theory: Entropy, Cross-Entropy, and KL Divergence

The Number on the Cricket App

Open a cricket score app during an IPL match and somewhere near the scorecard sits a number that updates after almost every ball: a live "win probability," something like Team A 73%, Team B 27%. Nobody watching knows for certain what happens next: a wicket, a six, a rain break. Yet the app commits to a precise number anyway. Two questions follow naturally. Once the match actually ends, how do you measure whether that number was any good? And if two different apps show different probabilities for the same match, how do you say which one was more accurate, and by how much? The same two questions apply to a weather app predicting today's rain, or a spam filter guessing whether a message is junk: any system that outputs a probability can, in principle, be graded once the truth is known.

Both questions turn out to have exact mathematical answers, supplied by a branch of mathematics called information theory. It was founded in 1948 by the American mathematician and engineer Claude Shannon in a paper titled "A Mathematical Theory of Communication," written while he was at Bell Labs. Shannon was trying to solve an engineering problem: how many bits does a telephone line need to carry a message reliably? The tools he built ended up measuring something far more general than telephone signals, namely uncertainty itself. Those same tools, unchanged, are what a modern AI classifier uses to learn, what compression software uses to shrink a file, and what a well-built prediction app uses to grade its own guesses. This chapter builds three of those tools from first principles: entropy, cross-entropy, and KL divergence.

Measuring a Single Surprise

Start smaller than a whole match: a single unpredictable event. Suppose a friend hides a coin in one of 8 identical boxes, and you must find it by asking only yes/no questions. Ask smart questions, splitting the boxes in half each time ("is it in boxes 1 to 4?", then "is it in boxes 1 to 2?", then "is it box 1?"), and you always find the coin in exactly 3 questions, because 2 × 2 × 2 = 8. That number, 3, is log2(8): the power to which 2 must be raised to reach 8. In general, picking out one outcome from N equally likely outcomes takes log2(N) yes/no questions.

Now drop the assumption that the boxes are equally likely. If the friend is biased and hides the coin in box 1 nine times out of ten, guessing "box 1" first is a great strategy: you will often be right after a single question. A rare, unexpected outcome carries more "surprise" than a common one, and should cost more questions, on average, to pin down. Information theory turns that intuition into a formula. The self-information of an outcome x with probability P(x) is defined as:

I(x) = -log2(P(x)) = log2(1 / P(x))

Two design choices sit behind this formula. The 1 / P(x) term captures rarity directly: an outcome with probability 1/8 gets a rarity score of 8, while an outcome with probability 1/2 gets a rarity score of only 2. The logarithm makes information from independent events add up the way surprise intuitively should. If two unrelated unfair coins are flipped, the joint probability of a specific pair of outcomes is P(x) times P(y), and because log(a times b) equals log(a) plus log(b), the information content works out to I(x) + I(y), just as it should, since learning about one coin tells you nothing about the other. The base of the logarithm sets the unit: base 2 gives an answer in bits, matching the yes/no-question intuition directly. A certain outcome, where P(x) = 1, needs zero bits to communicate, since log2(1) = 0. There is no surprise, and hence no information, in being told something you already knew for certain.

Entropy: Surprise, Averaged

Self-information measures the surprise of one specific outcome. Entropy measures the average surprise across an entire probability distribution: how unpredictable the whole situation is before you know the outcome. For a random variable X with outcomes x occurring with probability P(x), entropy is the probability-weighted average of self-information:

H(X) = -sum over x of [ P(x) * log2(P(x)) ]

The sum runs over every possible outcome. By convention, a term where P(x) = 0 is treated as contributing 0: an event that never happens can never surprise you.

Entropy has a second, equally useful reading: it is the minimum average number of bits needed to encode outcomes drawn from this distribution, provided you are free to assign short codes to common outcomes and long codes to rare ones. No coding scheme can beat this lower bound, a connection this chapter returns to near the end.

Two quick cases build intuition before the full worked example. A fair coin has P(heads) = P(tails) = 0.5, so H = -(0.5 × log2(0.5) + 0.5 × log2(0.5)) = -(0.5 × (-1) + 0.5 × (-1)) = 1 bit. You genuinely need one full yes/no question, on average, to learn how it landed. Compare that to Mumbai in the monsoon: if the chance of rain on a given July day is, say, 90%, then H = -(0.9 × log2(0.9) + 0.1 × log2(0.1)) ≈ -(0.9 × (-0.152) + 0.1 × (-3.322)) ≈ 0.469 bits. Less than half a bit, because you already have a strong lean ("it will probably rain"), so the outcome carries far less new information than a coin flip. Entropy is highest exactly when there is no useful lean at all, which is why a fair coin, the most undecided possible two-outcome distribution, hits the maximum value for two outcomes: 1 bit.

With that baseline in place, here is the fully worked example. Suppose historical head-to-head data between two IPL teams gives this true outcome distribution, P, for their next match: Team A wins with probability 0.80, Team B wins with probability 0.15, and the match ends with no result, washed out by rain, with probability 0.05. Rain washouts are a real possibility during India's cricket season, which is why professional cricket uses statistical target-revision systems such as the Duckworth-Lewis-Stern method for rain-interrupted games. Computing H(P) term by term:

• Team A wins: -0.80 × log2(0.80) = -0.80 × (-0.32193) = 0.25754
• Team B wins: -0.15 × log2(0.15) = -0.15 × (-2.73697) = 0.41054
• No result: -0.05 × log2(0.05) = -0.05 × (-4.32193) = 0.21610

Adding the three contributions gives H(P) = 0.25754 + 0.41054 + 0.21610 = 0.8842 bits. Compare this to the maximum possible entropy for three outcomes, which occurs when all three are equally likely at 1/3 each: H_max = log2(3) ≈ 1.585 bits. The actual value, 0.884 bits, sits well below that ceiling, which makes sense: the true distribution leans heavily toward Team A winning, and that lean is what makes the match somewhat predictable before a ball is bowled.

Cross-Entropy: Grading a Model Against Reality

Entropy assumes the true distribution P is already known. A prediction app does not have that luxury. It estimates the distribution from limited data, producing its own guess, call it Q, which may or may not match reality. Suppose the app behind our live win-probability number is fairly basic: it only looks at each team's results over their last five matches, and from that thin signal it estimates Q as Team A wins with probability 0.60, Team B wins with probability 0.30, no result with probability 0.10. That is in the right neighbourhood of P, but visibly miscalibrated: it underrates how dominant Team A really is and overrates the chance of a washout.

Cross-entropy measures the average number of bits needed if predictions are built around Q while outcomes actually keep being drawn from the true distribution P:

H(P, Q) = -sum over x of [ P(x) * log2(Q(x)) ]

The formula is asymmetric in a specific way: probabilities are weighted by the true distribution P, because that is what actually happens on average, but the logarithm, the "cost in bits," is computed from the model's guess, Q. Working through the same three outcomes:

• Team A wins: -0.80 × log2(0.60) = -0.80 × (-0.73697) = 0.58957
• Team B wins: -0.15 × log2(0.30) = -0.15 × (-1.73697) = 0.26054
• No result: -0.05 × log2(0.10) = -0.05 × (-3.32193) = 0.16610

Summing gives H(P, Q) = 0.58957 + 0.26054 + 0.16610 = 1.0162 bits. That is larger than H(P) = 0.8842 bits, and it always will be for any Q that is not a perfect match for P: building predictions on the wrong distribution always costs extra bits, never fewer. That gap between the two numbers has its own name and its own formula, covered next.

Cross-entropy is the quantity that neural network classifiers minimize during training, under the name categorical cross-entropy loss. There, the "true distribution" for one training example is almost always a one-hot vector: all the probability mass sitting on the single correct label, because once you know what actually happened, no uncertainty is left about it. If Team A genuinely won this match, P becomes [1, 0, 0], and the general formula collapses neatly. Every term except the one for "A wins" is multiplied by zero and vanishes, leaving H(P, Q) = -log2(0.60) ≈ 0.737 bits. That is why, in the source code of a machine learning library, categorical cross-entropy loss is implemented as nothing more than -log(predicted probability of the correct class). The full summation formula reduces to a single term the moment the ground truth is certain, which it always is once you are looking at a labelled training example.

KL Divergence: Isolating the Cost of Being Wrong

Cross-entropy mixes together two different sources of uncertainty. One is the randomness already present in P, which even a perfect model could not remove, since cricket has an irreducible element of chance. The other is the extra, avoidable confusion caused by Q being miscalibrated. KL divergence, short for Kullback-Leibler divergence and also called relative entropy, isolates just that second part: the penalty for using the wrong distribution, over and above the entropy that was already unavoidable.

D_KL(P || Q) = H(P, Q) - H(P) = sum over x of [ P(x) * log2( P(x) / Q(x) ) ]

Using the numbers already computed: D_KL(P || Q) = 1.0162 - 0.8842 = 0.1320 bits. The ratio form of the formula gives the same answer as an independent check: 0.80 × log2(0.80/0.60) + 0.15 × log2(0.15/0.30) + 0.05 × log2(0.05/0.10) = 0.80 × log2(1.333) + 0.15 × log2(0.5) + 0.05 × log2(0.5) = 0.80 × 0.41504 + 0.15 × (-1) + 0.05 × (-1) = 0.33203 - 0.15 - 0.05 = 0.1320 bits. Both routes agree, as they must. The extra 0.132 bits is the price the basic five-match-form model pays for not knowing the fuller head-to-head history.

Three properties of KL divergence are worth knowing precisely, because intuition gets each one wrong at some point:

• D_KL(P || Q) is never negative, and it equals exactly 0 only when Q matches P at every outcome. There is no such thing as negative extra cost, and you cannot do better than knowing the truth.
• It is not symmetric: D_KL(P || Q) is generally not equal to D_KL(Q || P). Swapping the roles in the running example, weighting the average by Q instead of P, gives D_KL(Q || P) ≈ 0.151 bits, close to the 0.132 bits found the other way round, but not the same number. This is why KL divergence is called a divergence and not a distance: a true distance would have to be symmetric, and this quantity is not.
• Because of that asymmetry, order matters in both the formula and the sentence describing it. D_KL(P || Q) reads as "the extra bits paid for approximating true distribution P using model Q," where P is always the reference: the ground truth being measured against.

Tracing the Numbers in Code

Every hand-calculated number above can be reproduced with a few lines of Python, which is also the fastest way to check similar problems:

import numpy as np

# True distribution: [Team A wins, Team B wins, No result]
P = np.array([0.80, 0.15, 0.05])

# Basic model's predicted distribution, from limited recent-form data
Q = np.array([0.60, 0.30, 0.10])

def entropy(p):
    return -np.sum(p * np.log2(p))

def cross_entropy(p, q):
    return -np.sum(p * np.log2(q))

def kl_divergence(p, q):
    return np.sum(p * np.log2(p / q))

print(f"H(P)          = {entropy(P):.4f} bits")
print(f"H(P, Q)       = {cross_entropy(P, Q):.4f} bits")
print(f"D_KL(P, Q)    = {kl_divergence(P, Q):.4f} bits")
print(f"H(P,Q) - H(P) = {cross_entropy(P, Q) - entropy(P):.4f} bits")

H(P)          = 0.8842 bits
H(P, Q)       = 1.0162 bits
D_KL(P, Q)    = 0.1320 bits
H(P,Q) - H(P) = 0.1320 bits

The last two lines matching is the identity H(P, Q) = H(P) + D_KL(P || Q) checking itself out on real floating-point numbers. Now trace the one-hot special case that a real training loop actually uses once a match has been played and the true outcome is certain:

def cross_entropy_loss(y_true_onehot, y_pred_probs):
    return -np.sum(y_true_onehot * np.log2(y_pred_probs))

y_true = np.array([1, 0, 0])          # Team A actually won
y_pred = np.array([0.60, 0.30, 0.10]) # model's prediction, unchanged

loss = cross_entropy_loss(y_true, y_pred)
print(f"Loss on this single example: {loss:.4f} bits")
# Loss on this single example: 0.7370 bits

Only the first term of the sum survives; the other two are multiplied by zero and drop out, leaving -log2(0.60), which numpy confirms is 0.7370 bits. One practical note before you meet this in a real framework: PyTorch and TensorFlow implement cross-entropy loss with the natural logarithm, base e, rather than log base 2, because it simplifies the calculus used during training. The resulting unit is called a nat instead of a bit. Converting between them just means dividing or multiplying by ln(2) ≈ 0.6931. The concept, and the code structure, stay identical either way.

Where These Three Numbers Show Up

Entropy, cross-entropy, and KL divergence show up throughout computing, far beyond a three-outcome cricket example. A softmax-output neural network trained to recognise handwritten digits, flag a forwarded message as spam, or decide whether a UPI transaction looks fraudulent is, at every training step, computing the cross-entropy loss traced above and nudging its internal weights to make that loss smaller. Decision tree algorithms such as ID3 and C4.5, developed by the computer scientist Ross Quinlan, decide which feature to split on at each node by choosing whichever split produces the largest drop in entropy, a quantity they call information gain, built from the same H(X) formula covered here. Lossless compression tools, the ones behind a ZIP file or a shrunk image sent over WhatsApp, lean on Shannon's source coding theorem, which proves that the entropy of a data source is the theoretical lower bound on how few bits, on average, are needed to encode it without losing information. Well-designed compressors get close to that bound. KL divergence reappears in more advanced generative AI systems, where it keeps a learned probability distribution close to a reference one: the same core idea explored here, applied to far richer distributions than a three-outcome cricket match.

Back to the App

The live win-probability number on a cricket app is a distribution Q, updated ball by ball from whatever model sits behind the app. Once the match finishes, reality hands you a one-hot P: the team that won, won. Cross-entropy is the honest scorecard for that app's final prediction: a low number means the app assigned high probability to what actually happened, a high number means it was confidently wrong. Compare two rival apps on the same match, and the one with the lower cross-entropy made the better call. Subtract away the portion of that cross-entropy that no app could have avoided, the inherent, irreducible uncertainty of cricket itself, and what is left over, KL divergence, is a clean, honest number for how much better the app could have been. The two questions this chapter opened with, how good was that percentage and which app was more accurate, turn out to have precise, computable answers, in bits, using nothing more than probabilities and a logarithm. Next time that percentage flickers on the screen mid-over, remember it carries an exact, checkable price for being wrong, payable in bits.

Think About It

Think about this: How would you explain information theory: entropy, cross-entropy, and kl divergence to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where information theory: entropy, cross-entropy, and kl divergence is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting information theory: entropy, cross-entropy, and kl divergence to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind information theory: entropy, cross-entropy, and kl divergence, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.