Two notifications arrive on your phone on the same July afternoon. The first is from a weather app: "Heavy rain expected in Mumbai today." You barely register it. Mumbai in July sits deep in monsoon season, so of course it's going to rain. The second, from the same app an hour later, reads: "Heavy rain expected in Jaisalmer today." This one makes you stop and look twice. Jaisalmer sits in the Thar Desert, one of the driest regions in India, where a rainy day is genuinely unusual. Both messages use the same words, the same length, the same font. Yet one feels like background noise and the other feels like actual news.
What changed isn't the message. It's how expected the message was. That gap, between a fact you already assumed and a fact that catches you off guard, is the starting point of information theory: a branch of mathematics that turns genuine surprise, and therefore information, into a precise number instead of a vague feeling.
From a Feeling to a Formula
Give this "news value" a proper name: the self-information, or surprisal, of an event. If an event has probability p of happening, the goal is a function I(p) that converts that probability into a surprise score. Before writing the formula down, it helps to list what any sensible surprise score has to do.
A certain event should carry zero surprise. If Mumbai's July forecast always says "rain," and it always does rain, that message tells you nothing you didn't already know. So I(1) must equal 0.
Surprise should rise as probability falls. An event with a 1% chance of happening is more surprising than one with a 50% chance, which is more surprising than one with a 99% chance. So I(p) has to decrease as p increases.
The third requirement is the one that pins down the exact formula: surprise from independent events has to add up. Suppose the Jaisalmer rain happens on the same day that, completely separately, your favourite IPL team pulls off an upset win against heavy odds. Two unrelated surprising things happened. Because the events are independent, the probability of both happening together is the product of their individual probabilities: P(A and B) = P(A) × P(B). But the surprise you feel from the combination should be the sum of the two surprises, not their product. Hearing two unrelated pieces of shocking news back to back feels like double the shock, not shock multiplied by itself.
Is there a function that turns multiplication on the input into addition on the output? Yes: the logarithm. For any base, log(a × b) = log(a) + log(b). That single property is the entire reason information theory is built on logarithms rather than on some other decreasing function.
A quick numeric check confirms it works as intended. Take two independent events with probabilities p₁ = 0.5 and p₂ = 0.25. Their self-informations are I(0.5) = -log2(0.5) = 1 bit and I(0.25) = -log2(0.25) = 2 bits, which should add to 3 bits if the formula is behaving correctly. The combined event has probability p₁ × p₂ = 0.125, and computing its self-information directly gives I(0.125) = -log2(0.125) = 3 bits, exactly the sum the additivity rule demanded.
Putting the requirements together gives the formula for self-information:
I(p) = log2(1 / p) = -log2(p)log2 means "logarithm, base 2." It answers the question "2 raised to what power gives this number?" For instance, log2(8) = 3, because 2³ = 8, and log2(1) = 0, because 2⁰ = 1. Choosing base 2 specifically means the answer comes out in a unit called the bit, short for "binary digit," the same bit used to measure computer memory, network speed, and processor registers. That's not a coincidence. A single fair coin flip, with two equally likely outcomes, carries exactly one bit of surprise, matching the one binary digit needed to record which way it landed.
A few more checks confirm the formula behaves the way intuition demands. A fair coin has p = 0.5 for heads, so I(0.5) = -log2(0.5) = log2(2) = 1 bit, the standard definition of one bit of information. A certain event has p = 1, so I(1) = -log2(1) = 0: no surprise, no information, just as the first requirement demanded. An event that is one specific outcome out of 1024 equally likely possibilities (a specific 10-digit binary string, say) has p = 1/1024, so I(1/1024) = log2(1024) = 10 bits, precisely the 10 binary digits it takes to write that string down. This is also why computer memory has traditionally been measured in powers of two: 1 kilobyte is 1024 bytes, because 2¹⁰ = 1024.
Push the formula to its edge and it keeps behaving sensibly. As p approaches 0, I(p) grows without bound: an event you were certain could never happen, if it somehow does happen, carries an almost unlimited amount of information. This has a real consequence outside mathematics: a single confirmed case of something declared "impossible" can outweigh, in genuine news value, thousands of confirmations of something routine.
From One Surprise to Average Surprise: Entropy
Self-information measures the surprise of one outcome, after it has already happened. Often, though, the more useful statistical question is different: before anything happens, how unpredictable is the whole situation, on average? This is the kind of question the rest of statistics already answers for other quantities: a mean is the average value a random variable takes, and a variance is the average squared distance from that mean. Entropy, usually written H, comes from asking the same kind of "average" question about self-information instead: it is the average number of bits of surprise per observation, weighted by how often each outcome actually occurs.
H = Σ p_i × log2(1 / p_i)summed over every outcome i in the distribution. In the language of probability, this is simply the expected value of self-information, H(X) = E[I(X)]. Low entropy means the outcome is close to a foregone conclusion, like Mumbai rain in July. High entropy means every outcome is close to equally likely: the statistical definition of maximum unpredictability.
Two extremes make the idea concrete before working through a full example. A coin so heavily weighted that it lands heads 99% of the time has very low entropy: heads is almost certain, so most flips teach you almost nothing, and only the rare 1% tails delivers a large jolt of surprise. Working through the formula for this coin gives an entropy of about 0.08 bits per flip. A perfectly fair coin, by contrast, has the maximum possible entropy for a two-outcome system: exactly 1 bit per flip, because heads and tails are equally likely and neither is ever the safer bet. The same logic scales up with more outcomes: a fair six-sided die, where all six faces are equally likely, has entropy log2(6) ≈ 2.585 bits per roll, since more equally plausible outcomes mean more room for surprise. Entropy measures how close a distribution is to "anything could happen": the more lopsided the probabilities, the lower the entropy; the more evenly spread they are, the higher.
There's a hands-on way to feel what a bit of entropy actually costs: play a round of Twenty Questions, where a hidden outcome must be identified using only yes/no questions. A well-chosen question splits the remaining possibilities as evenly as possible, so asking it carries close to even odds of "yes" or "no": the fair-coin case worked out above, worth close to one bit per question. An outcome with an entropy of 1 bit should take about one such well-balanced question, on average, to pin down; an outcome with an entropy of 10 bits, like the 1-in-1024 example above, should take about ten. This is why entropy is sometimes described in plain language as the average number of well-aimed yes/no questions a situation demands before all uncertainty is gone.
Worked Example: The Entropy of a Cricket Delivery
To see the calculation in full, build a simplified model of what can happen on a single ball in a T20 innings. Real broadcasters and analytics platforms build far more detailed models than this, but a five-outcome version is enough to trace by hand. Suppose that for a particular batter facing a particular bowler, the next ball is modelled with these probabilities:
- Dot ball, no run scored: p = 0.40
- Single or two runs: p = 0.35
- Boundary, four runs: p = 0.15
- Six: p = 0.05
- Wicket: p = 0.05
These five probabilities sum to 1.00, as any complete probability distribution must, since exactly one of these five outcomes has to occur on every ball.
Step 1: find the self-information of each outcome. Apply I(p) = -log2(p) to each row.
- Dot ball: -log2(0.40) ≈ 1.322 bits
- Single or two: -log2(0.35) ≈ 1.515 bits
- Four: -log2(0.15) ≈ 2.737 bits
- Six: -log2(0.05) ≈ 4.322 bits
- Wicket: -log2(0.05) ≈ 4.322 bits
A pattern is already visible: the dot ball, the most likely event, carries the least surprise, while a six and a wicket (the two rarest events in this model) carry the most. A wicket earns a replay, a graphic, and a stadium roar; a dot ball earns none of it. The crowd's attention, like the self-information formula, tracks probability.
Step 2: weight each self-information value by its own probability. This gives each outcome's contribution to the average.
- Dot ball: 0.40 × 1.322 ≈ 0.529 bits
- Single or two: 0.35 × 1.515 ≈ 0.530 bits
- Four: 0.15 × 2.737 ≈ 0.411 bits
- Six: 0.05 × 4.322 ≈ 0.216 bits
- Wicket: 0.05 × 4.322 ≈ 0.216 bits
Step 3: add the five contributions to get entropy.
H = 0.529 + 0.530 + 0.411 + 0.216 + 0.216 ≈ 1.902 bitsOn average, each ball in this model carries about 1.9 bits of genuine uncertainty. Compare that to the maximum possible entropy for five outcomes, which would occur only if all five were equally likely at p = 0.20 each: log2(5) ≈ 2.322 bits. The actual entropy, 1.902 bits, sits noticeably below that ceiling, and the gap is itself meaningful: it quantifies how much the skew toward dot balls, singles, and twos makes a cricket delivery more predictable than a hypothetical world where all five outcomes were equally probable.
Checking the Arithmetic in Code
The same calculation translates directly into a short Python program, useful both as a check on the hand arithmetic and as a pattern that works for any probability distribution.
import math
def self_information(p):
"""Surprise of one outcome, in bits."""
return -math.log2(p)
def entropy(prob_dist):
"""Average surprise across a full distribution, in bits."""
return sum(p * self_information(p) for p in prob_dist if p > 0)
delivery = {
"dot_ball": 0.40,
"single_or_two": 0.35,
"four": 0.15,
"six": 0.05,
"wicket": 0.05,
}
for outcome, p in delivery.items():
print(f"{outcome}: surprise = {self_information(p):.3f} bits")
print(f"\nEntropy = {entropy(delivery.values()):.3f} bits")Running this prints:
dot_ball: surprise = 1.322 bits
single_or_two: surprise = 1.515 bits
four: surprise = 2.737 bits
six: surprise = 4.322 bits
wicket: surprise = 4.322 bits
Entropy = 1.902 bitswhich matches the hand calculation exactly. self_information implements I(p) = -log2(p) directly using Python's built-in math.log2. entropy is a one-line translation of its own definition: multiply each probability by its own self-information, and add the results. Because entropy accepts any iterable of probabilities, this same function computes the entropy of a coin flip, a die roll, or the letter frequencies of an entire book, as long as the numbers handed to it sum to 1.
Why Shannon Called It "Entropy"
This entire framework came from a single, remarkably self-contained paper, "A Mathematical Theory of Communication," published in two installments in the Bell System Technical Journal in July and October 1948, written by Claude Shannon, an American mathematician and electrical engineer working at Bell Labs. Shannon's path to that paper ran partly through his wartime work at Bell Labs on cryptography and secure communication during the Second World War. The same mathematics that measures how surprising a message is also measures how hard that message is to guess, which is what makes a code secure or breakable. It was not his first landmark idea: a decade earlier, Shannon's 1937 master's thesis at MIT had shown how Boolean algebra could design and simplify electronic switching circuits, work often credited as a founding document of digital circuit design. The 1948 paper is the one that gave the world the bit as a unit of information, but the switching-circuit thesis is why some historians credit the same person with helping start both digital hardware and information theory within a single decade.
Shannon needed a name for his new uncertainty measure, and, by a widely told account, the mathematician John von Neumann suggested "entropy," because Shannon's formula takes exactly the same mathematical form as the entropy formula from nineteenth-century statistical mechanics, developed by Ludwig Boltzmann to describe disorder in physical systems. The name stuck, which is why a concept about probability and surprise carries a word borrowed from thermodynamics. The word "bit" itself predates Shannon's paper by only a year: he credits his Bell Labs colleague John W. Tukey with coining it, as a contraction of "binary digit." Shannon's paper is what turned a convenient abbreviation into the basic unit of an entire science.
Where This Shows Up in Computing
Entropy sets a hard limit that real systems are built around. Shannon's source coding theorem proves that the entropy of a source is the minimum average number of bits per symbol that any lossless compression scheme can achieve; you cannot compress data below its own entropy without losing information. Applied to the cricket example above, this means no encoding scheme, however clever, could describe a long sequence of these ball-by-ball outcomes using fewer than 1.902 bits per ball on average. That number is a hard floor, not just this chapter's estimate.
Practical compressors chase this limit directly. Huffman coding, developed in 1952 by David Huffman, assigns short binary codes to frequent symbols and long codes to rare ones (precisely the strategy the entropy formula rewards) and still runs inside formats like ZIP, PNG, and MP3. The idea predates computing by a century: when Samuel Morse and Alfred Vail designed Morse code in the 1830s, they gave the most frequent English letters the shortest signals — E is a single dot, T is a single dash — while rare letters like Q and Z got long, four-symbol codes.
Inside machine learning, the same formula reappears as information gain. A decision tree learns by repeatedly asking yes/no questions about its data, and at each step it picks the question that reduces entropy the most: the one whose answer is, on average, most surprising, and therefore most informative, given everything the tree already knows. Language models and keyboard autocomplete work in the mirror-image direction: they assign a probability to every possible next word, and the suggestion offered first is simply the one with the lowest self-information, the least surprising continuation given everything typed so far.
The same formula also underlies something used every week: PIN security. Suppose a UPI PIN has six digits. There are 10⁶ = 1,000,000 equally likely combinations, so the self-information of guessing that exact PIN correctly on one attempt is log2(1,000,000) ≈ 19.93 bits. Security engineers describe password and PIN strength "in bits" for this reason: it is the self-information formula from earlier in this chapter, applied to a guess instead of a weather forecast.
Back to the Weather App
Return to the two notifications from the start of this chapter. "Heavy rain in Mumbai today," during monsoon season, might carry a self-information of a small fraction of a bit: the forecast probability is close to 1, so -log2(p) is close to 0. "Heavy rain in Jaisalmer today" might carry six, eight, or more bits, depending on how rare rain is there that time of year; the lower the true probability, the larger the number. The two messages are the same length in English, but nowhere near the same length in the unit that actually matters: bits of genuine information.
That is the practical takeaway to carry forward. Whenever something gets called "informative" (a dataset, a test result, a news alert, a single sensor reading), information theory offers a sharper question than "does this feel important?" It asks how probable the event was, and how many bits that makes it worth. Every later idea in this course that talks about a machine "learning" from data — a decision tree choosing the most useful question to ask, a compression algorithm squeezing a file smaller, a model built to predict the next word in a sentence or the next ball of an over — is, underneath the code, still doing exactly this: measuring surprise, one bit at a time. Later in this course, when a machine learning model is trained by minimising something called cross-entropy loss, that loss function is built directly from today's formula: on every training example, it scores the model by -log2 of the probability the model assigned to the correct answer. That is the exact self-information formula from the start of this chapter, now measuring how surprised a model was to be told it was wrong.
Think About It
Think about this: How would you explain information theory: measuring surprise to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where information theory: measuring surprise is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting information theory: measuring surprise to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind information theory: measuring surprise, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.