Active Learning: Selecting Informative Examples

NPCI's fraud-detection team at a major UPI-linked bank sees roughly 40 million transactions a day. A gradient-boosted model scores every single one for fraud probability in real time. But the model was trained on labeled data, and labels are the expensive part: a transaction only becomes "confirmed fraud" or "confirmed genuine" after a trained fraud analyst investigates it — checking device fingerprints, call records, merchant history, sometimes phoning the customer. A team of twenty analysts can close perhaps 2,000 cases a day. Out of 40 million transactions, 39,998,000 will never be looked at by a human unless someone decides which ones matter enough to send for review.

The naive approach is to hand the analysts a random sample of 2,000 transactions each day. This is what a standard supervised pipeline assumes happens in the background — someone, somewhere, produces labels, and the model consumes them. But a random sample is mostly wasted effort: the overwhelming majority of UPI transactions are unambiguously genuine (a ₹40 grocery payment at a known merchant, from a device the model has seen a hundred times before) or unambiguously fraudulent (a pattern the model has already learned cold). Handing an analyst a transaction the model is already 99.7% sure about teaches the model almost nothing new. The transactions worth an analyst's time are the ones the model is unsure about — probability near 0.5, sitting right on the decision boundary, where a human label would sharpen the model's understanding the most.

This is the problem active learning solves. Instead of a model passively consuming whatever labeled data it is given, the model itself decides which unlabeled examples are worth the cost of a label, and requests exactly those. The word "active" refers to the learner's role: it actively queries, rather than passively receiving.

Passive learning vs. active learning: the formal distinction

In ordinary supervised learning — the setting you've used throughout this course — you are handed a fixed training set D = {(x₁,y₁), ..., (xₙ,yₙ)} and you fit a model to minimize loss over it. The learner has no say in which examples got labeled; that decision was made upstream, usually by random or convenience sampling.

Active learning changes the protocol. The learner starts with a small labeled set L (sometimes just a handful of seed examples) and a large pool of unlabeled examples U. It also has access to an oracle — almost always a human expert — who can produce a true label for any example it asks about, at a cost. The learning loop is:

1. Train (or update) the current model Mₜ on L.
2. Use Mₜ to score every unlabeled example in U by a chosen informativeness criterion.
3. Select the example (or batch of examples) with the highest score: x* = argmax informativeness(u, Mₜ).
4. Query the oracle for the true label y* of x*.
5. Move (x*, y*) from U to L, retrain, and repeat.

Three query scenarios show up in the literature, and it's worth being precise about which one the UPI example is. Pool-based sampling assumes the entire unlabeled set is available up front and the learner can rank all of it before choosing — this fits UPI, where a day's transactions sit in a queryable pool. Stream-based selective sampling processes examples one at a time as they arrive and must decide immediately, query-or-discard, without ever seeing the rest of the stream — this fits a live payment gateway that cannot buffer transactions for batch scoring. Membership query synthesis is the most aggressive: the learner doesn't pick from existing data at all, it generates a synthetic input and asks the oracle to label that — feasible for a domain like handwriting recognition where you can synthesize a stroke pattern, essentially unusable for UPI because a human analyst cannot meaningfully "label" a fabricated transaction that never happened. The rest of this chapter builds on pool-based sampling, the dominant case in practice.

Measuring informativeness: uncertainty sampling

The simplest and most widely deployed family of query strategies is uncertainty sampling: score each unlabeled example by how unsure the current model is about it, and query the most uncertain ones first. For a binary classifier that outputs p = P(y=1 | x), three common uncertainty measures are:

Least confidence: U_LC(x) = 1 − max(p, 1−p). This is highest when p = 0.5 (value 0.5) and lowest when the model is certain in either direction.

Margin sampling: U_M(x) = 1 − |p − (1−p)| = 1 − |2p − 1|. Equivalently, for multi-class problems, the margin is p₁ − p₂, the gap between the top-two predicted class probabilities — small gap means the model is torn between two candidates, which is exactly what you want to resolve with a label.

Entropy sampling: treats the full predicted distribution as a probability distribution and measures its Shannon entropy, H(x) = −Σᵢ pᵢ log₂ pᵢ, summed over every class the model predicts probabilities for. For binary classification this is H(p) = −p log₂ p − (1−p) log₂(1−p), maximized at p = 0.5 where H = 1 bit, and falling to 0 as p approaches 0 or 1.

All three agree on binary problems — they all peak at p = 0.5 — but they diverge once there are three or more classes, because least confidence and margin sampling only look at the top one or two probabilities, while entropy accounts for the shape of the entire distribution. We'll see this divergence numerically below; it is a genuine, non-obvious subtlety, not a technicality.

The active learning cycle

Follow the solid arrows clockwise: the pool feeds the query strategy, which uses the current model to score every candidate and hands the single highest-scoring one to the oracle, whose label is appended to the labeled set, which retrains the model. The dashed purple arrow is the part a first read of the diagram tends to miss: the freshly retrained model Mₜ₊₁ loops back and re-scores the (now slightly smaller) pool for the next round. Every iteration uses a different, better-informed model to decide what to ask about next — the query strategy is not fixed, it evolves with the model. In production, "one query per round" is usually relaxed to a batch of the top-k scoring examples per day, since retraining after every single label is too slow to be practical at UPI scale.

Worked example 1 — ranking five transactions by uncertainty

Suppose today's model Mₜ has scored five unlabeled UPI transactions with fraud probability p. Compute entropy H(p) = −p log₂ p − (1−p) log₂(1−p) and margin |2p − 1| for each, and rank which one the analyst should see first.

Txn   p(fraud)   H(p) bits   |2p−1|    Least-conf. 1−max(p,1−p)
T1    0.50       1.0000      0.0000    0.5000
T2    0.92       0.4022      0.8400    0.0800
T3    0.05       0.2864      0.9000    0.0500
T4    0.68       0.9044      0.3600    0.3200
T5    0.35       0.9341      0.3000    0.3500

Trace T4 by hand to confirm the formula: p = 0.68, so 1−p = 0.32. log₂(0.68) = ln(0.68)/ln(2) = −0.3857/0.6931 ≈ −0.5565, giving −0.68 × (−0.5565) = 0.3784. log₂(0.32) = ln(0.32)/ln(2) = −1.1394/0.6931 ≈ −1.6439, giving −0.32 × (−1.6439) = 0.5260. Sum: 0.3784 + 0.5260 = 0.9044 bits — matches the table.

Ranked by entropy (most informative first): T1 (1.0000) > T5 (0.9341) > T4 (0.9044) > T2 (0.4022) > T3 (0.2864). Margin sampling and least-confidence give the identical ordering here — expected, since for a binary classifier all three uncertainty measures are strictly increasing functions of the distance from p = 0.5. Active learning sends T1 to the analyst first: the model is maximally torn (50/50) on that transaction, so whatever label comes back — fraud or genuine — sharpens the decision boundary the most. T3, at p = 0.05, is exactly the kind of transaction a random-sampling pipeline would waste analyst time on: the model is already 95% confident it's genuine, and a label confirming that teaches almost nothing.

Worked example 2 — where margin and entropy disagree

Now suppose the fraud model is multi-class, distinguishing three outcomes: Genuine, Phishing fraud, Mule-account fraud. Compare two transactions by their predicted distributions:

Txn A: [Genuine 0.40, Phishing 0.35, Mule 0.25]
Txn B: [Genuine 0.50, Phishing 0.48, Mule 0.02]

Margin sampling looks only at the gap between the top two classes. For A, that gap is 0.40 − 0.35 = 0.05. For B, it's 0.50 − 0.48 = 0.02. Since a smaller margin means the model is more torn between its top two guesses, margin sampling ranks B as more uncertain and queries it first.

Entropy sees the whole distribution, not just the top two entries. For A: H = −(0.40 log₂ 0.40 + 0.35 log₂ 0.35 + 0.25 log₂ 0.25) = 0.5288 + 0.5301 + 0.5000 = 1.5589 bits — close to the maximum possible for three classes, log₂3 ≈ 1.5850, because probability mass is spread fairly evenly across all three outcomes. For B: H = −(0.50 log₂ 0.50 + 0.48 log₂ 0.48 + 0.02 log₂ 0.02) = 0.5000 + 0.5083 + 0.1129 = 1.1211 bits, noticeably lower, because the third class carries almost no probability mass — the model has effectively already ruled out mule-account fraud for B and is only torn between two of the three options, not all three. Entropy ranks A as more uncertain and queries it first — the opposite conclusion from margin sampling.

Which is "right" depends on what a wrong answer costs. If misclassifying between any two of the three classes is equally costly, entropy's view — A has more genuinely open questions across the whole label space — is the more defensible one to act on. If you only ever care about getting the single most likely class right and don't care about the runner-up, margin sampling's narrower focus on B is closer to what you actually need. Neither strategy is universally superior; picking one is a modeling decision, not a formality, and this divergence is exactly why production active-learning systems document which criterion they use and why.

Correcting a common misconception

A very natural misreading of "active learning" is that it means the model labels its own data and teaches itself — no human required, since the model is "actively" doing something. This is wrong, and it's worth being precise about why, because it collides with a real neighboring technique.

Active learning always queries a real oracle — a human, or some other source of ground truth — for every label it adds to L. The "active" part is entirely about the selection of which example to ask about; the human still supplies the actual answer. What the student is probably thinking of is self-training (a form of semi-supervised learning): there, the model predicts labels for unlabeled data itself, keeps the predictions it's most confident about, and adds those model-generated pseudo-labels straight into its own training set — no human in the loop at all for those points. Self-training amplifies whatever the model already believes: it reinforces confident predictions, right or wrong, and can compound systematic errors over rounds. Active learning does the opposite by design — it deliberately seeks out the examples the model is least sure about and hands exactly those to a trustworthy outside source, precisely to correct the model's blind spots rather than reinforce them. The two techniques are sometimes combined (label the uncertain points with a human, pseudo-label the confident points automatically), but conflating them erases the entire reason active learning exists: to spend a scarce human labeling budget where it moves the model the most.

Beyond single-point uncertainty: committees, redundancy, and cost

Query-by-committee addresses a weakness of uncertainty sampling: it trusts a single model's probability estimates, which can be badly calibrated, especially early in training when L is small. Instead, train a committee of several models (different architectures, or the same architecture on different bootstrap samples of L) and query the unlabeled example where the committee members disagree most — measured, for instance, by vote entropy across the committee's predicted classes. High disagreement means the current data doesn't yet pin down the right answer, which is a different, often more robust, signal than one model's self-reported confidence.

Batch redundancy is a practical trap: if you naively pick the top-20 highest-entropy transactions in one day's pool, you can easily end up with 20 near-duplicates of the same fraud pattern (say, a particular SIM-swap scheme sweeping through one city that morning), because they all sit near the same point on the decision boundary. Twenty labels of near-identical transactions teach the model about as much as one label would. Production batch-mode active learning therefore combines the uncertainty score with a diversity or density term — for example, penalizing candidates that are too similar (in feature space) to others already selected in the same batch — so the batch spans the boundary rather than clustering on one segment of it.

Two more constraints shape real deployments. The cold-start problem: uncertainty sampling needs a model to compute uncertainty from, but the first round has no model yet, so the seed labeled set is usually chosen by random or stratified sampling before active learning takes over. And a stopping criterion is needed, since querying can continue indefinitely: common choices are a fixed labeling budget, or halting once the average uncertainty across the remaining pool drops below a threshold, signalling that most of what's left really is unambiguous.

Active recall

Attempt each question before reading its answer.

1. A model outputs p(fraud) = 0.5 for transaction X and p(fraud) = 0.02 for transaction Y. Under uncertainty sampling, which gets queried first, and why?
2. Why does membership query synthesis fail as a scenario for the UPI fraud problem, even though it works fine for handwriting recognition?
3. A three-class model outputs [0.34, 0.33, 0.33] for one example and [0.34, 0.33, 0.01, 0.01, ..., 0.01] (33 tiny classes summing to 0.33) for another in a 35-class problem. Which query strategy — margin or entropy — would treat these two very differently, and why?
4. A teammate says "we don't need human fraud analysts anymore, the active learning system labels its own data." What specific error is in that statement?
5. Why does batch active learning need an explicit diversity term, when a single-query active learning loop does not?
6. Given predicted probability p = 0.8 for a transaction, compute its entropy in bits and its margin score, and state whether it would be queried before or after the T4 transaction from Worked Example 1 (p = 0.68, H = 0.9044).

Answers

1. Transaction X is queried first. Both least-confidence, margin, and entropy sampling peak at p = 0.5 — the model is maximally uncertain there — while p = 0.02 means the model is already 98% confident the transaction is genuine, so a label would very likely just confirm what the model already believed, adding little information.

2. Membership query synthesis requires the oracle to label a synthetic input the learner generates. A synthesized handwritten stroke pattern is still a valid handwriting sample a human can read and label. A synthesized UPI transaction (a fabricated sender, receiver, amount, device fingerprint) never occurred, so a fraud analyst has no real-world context — no call records, no device history — to investigate; there is nothing genuine to determine.

3. Margin sampling treats them identically, since both have the same top-two gap (0.34 − 0.33 = 0.01 in both cases) — it only ever looks at the top two probabilities. Entropy treats them very differently: the second distribution spreads real probability mass (0.33 total) thinly across 33 additional classes the model considers non-trivially possible, so its entropy is far higher than the first distribution's, which concentrates all remaining mass on a single third class. Entropy captures that the model is genuinely unsure across many classes in the second case, not just torn between two.

4. Active learning still requires a real oracle (the analysts) to supply every label added to the training set; only the choice of which transaction to ask about is automated. What the teammate is describing — a model generating its own labels without human input — is self-training/pseudo-labeling, a different technique that risks reinforcing the model's existing errors rather than correcting them.

5. H(0.8) = −0.8 log₂0.8 − 0.2 log₂0.2 = 0.8×0.3219 + 0.2×2.3219 = 0.2575 + 0.4644 = 0.7219 bits. Margin: |2×0.8−1| = 0.6. Since H(0.8) = 0.7219 < H(0.68) = 0.9044, and the margin 0.6 > 0.36, the model is less torn about this transaction than about T4, so it is queried after T4.

Think About It

Think about this: How would you explain active learning: selecting informative examples to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind active learning: selecting informative examples, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.