Causal Inference in Machine Learning

Five ranking policies, one logged dataset, no budget for five A/B tests

A food-delivery app operating at Swiggy's scale ranks restaurants and dishes on the home screen for every user, every session. The ranking model currently in production, call it π0, is not deterministic: for engineering reasons (mostly to keep collecting exploration data) it mixes in randomness, so for a given user context x it shows action (restaurant) a with some known probability π0(a|x). The data science team has trained five candidate rerankers, π1 through π5, each a plausible improvement on paper. Running a live A/B test for all five would mean holding back real revenue and real delivery volume on four losing arms for however many weeks it takes to reach significance, while the fifth arm, if it happens to be bad, quietly costs orders. The team wants to know, before touching production traffic at all: for each candidate policy π_k, what would the expected reward (say, order-completion rate) have been, had that policy been running instead of π0, using only the exposure log they already have sitting in a data warehouse.

This is a causal question dressed up as an engineering question. The logged data was generated by π0's decisions. Asking what would have happened under π1 is asking about a policy that never ran, using data produced by a different one. It is the "what would have happened under a different intervention" question, applied not to a single treatment variable but to an entire decision-making function. This chapter builds the machinery ML systems actually use to answer it: importance-weighted off-policy evaluation, its variance pathology, self-normalization, and the doubly robust estimator that production teams reach for once plain importance weighting stops being trustworthy. It assumes you already have the formal causal-graph and do-operator vocabulary; here that machinery gets pointed at a specific, very common industrial problem: evaluating and choosing between candidate decision policies from logged bandit feedback, without deploying them first.

Why the deployed policy's own average reward is not the number you need

Suppose you simply compute the average reward across all logged rows: (1/n)Σr_i. What does that number estimate? Precisely, it is an unbiased estimate of E_a~π0[r(x,a)], the value of the policy that was actually running, V(π0). It says nothing directly about V(π1), the value of a policy that would have made different choices in the same contexts. This is not a subtle technicality; it is the bandit-scale instance of what Paul Holland (1986, building on Rubin's potential-outcomes framework) called the fundamental problem of causal inference: for each context x_i, you observe the reward under exactly one action, the one π0 actually chose. The reward you would have gotten under any other action, including whatever π1 would have picked, is a missing potential outcome. You cannot fill it in from the log, because the log simply does not contain it.

This immediately rules out the naive move of treating the exposure log like an ordinary labeled dataset and asking "how good is this new ranker's predicted click-through rate on held-out data?" A held-out AUC or log-loss number evaluates how well a score correlates with the click that was observed under π0's exposure pattern. It says nothing about the reward π1 would generate, because π1 would show different items to different users than π0 did, and the log has no record of what would have happened in those counterfactual exposures. The two questions, "is this score function well-calibrated to clicks that happened" and "how much reward would this decision policy produce if deployed," are related but not the same question, and conflating them is the single most common way an ML system that looks great in offline predictive metrics turns out to lose money live.

Importance sampling as a counterfactual query

The fix is a change of measure. Start from the definition of what you want:

V(π1) = E_x~D[ Σ_a π1(a|x) r(x,a) ]

Multiply and divide the inner term by π0(a|x), valid at every (x,a) pair where π0(a|x) > 0, which is exactly the positivity (overlap) condition carried over from the general causal framework, now applied to the bandit's action distribution:

V(π1) = E_x~D[ Σ_a π0(a|x) · (π1(a|x) / π0(a|x)) · r(x,a) ]

The inner sum, weighted by π0(a|x), is exactly an expectation over a~π0(·|x). So:

V(π1) = E_{x~D, a~π0(·|x)}[ w(x,a) · r(x,a) ], where w(x,a) = π1(a|x) / π0(a|x)

The right-hand side is an expectation over exactly the distribution the logged data was drawn from. That means the sample average of w_i·r_i across the log is an unbiased, consistent estimator of V(π1), even though π1 never touched production traffic. This is the inverse propensity score (IPS) estimator:

V̂_IPS(π1) = (1/n) Σ_i=1ⁿ w_i r_i, w_i = π1(a_i|x_i) / π0(a_i|x_i)

Mechanically, IPS re-weights each observed reward by how much more, or less, likely the candidate policy was to have made that same choice than the logging policy was. A logged click on an item π1 would have shown twice as often as π0 did counts double; a logged click on an item π1 almost never picks counts for almost nothing. This is the direct bandit analogue of inverse-propensity weighting: instead of reweighting to balance a treatment assignment against confounders, you reweight to simulate what a different action-selection policy would have produced, using only actions the logging policy actually explored. The requirement that π0(a|x) > 0 wherever π1(a|x) > 0 is the positivity assumption doing real, load-bearing work here: if the logging policy never had any chance of showing an action that the candidate policy wants to show in some context, no amount of reweighting recovers that missing region, because no data covers it. Léon Bottou and colleagues formalized this whole framework for production advertising systems at Microsoft in "Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising" (Bottou et al., Journal of Machine Learning Research 14, 2013), which is worth reading once you have the mechanics below.

Worked example: evaluating a new ranking policy from logged data

Six impressions were logged under π0. For each, the table records which action was actually shown, its logging probability π0(a|x), the candidate policy's probability π1(a|x) of choosing that same action, and the observed reward.

user | action shown | pi0(a|x) | pi1(a|x) | reward
u1   | A            | 0.50     | 0.20     | 1
u2   | B            | 0.30     | 0.60     | 0
u3   | A            | 0.50     | 0.20     | 0
u4   | C            | 0.20     | 0.10     | 1
u5   | B            | 0.30     | 0.60     | 1
u6   | C            | 0.02     | 0.10     | 1

User u6 came from a rarely-taken exploration slot: π0 only picked item C for that context 2% of the time. Compute the importance weights, then the naive average, IPS, and SNIPS estimates directly:

# Logged data from the current ranking policy pi0
actions  = ['A', 'B', 'A', 'C', 'B', 'C']
pi0      = [0.5, 0.3, 0.5, 0.2, 0.3, 0.02]   # P(action | context) under the logging policy
pi1      = [0.2, 0.6, 0.2, 0.1, 0.6, 0.10]   # P(action | context) under the candidate policy
rewards  = [1,   0,   0,   1,   1,   1]      # observed reward for the action actually shown

# Importance weights: how much more/less likely pi1 was to take the logged action than pi0
weights = [p1 / p0 for p1, p0 in zip(pi1, pi0)]

naive_estimate = sum(rewards) / len(rewards)
ips_estimate   = sum(w * r for w, r in zip(weights, rewards)) / len(rewards)
snips_estimate = sum(w * r for w, r in zip(weights, rewards)) / sum(weights)

print(f"Naive (on-policy) average reward: {naive_estimate:.4f}")
print(f"IPS estimate of V(pi1):           {ips_estimate:.4f}")
print(f"SNIPS estimate of V(pi1):         {snips_estimate:.4f}")

Trace it by hand. weights = [0.4, 2.0, 0.4, 0.5, 2.0, 5.0] (u6's weight is 0.10/0.02 = 5.0, by far the largest, because it divides by the smallest propensity). w·r per row = [0.4, 0.0, 0.0, 0.5, 2.0, 5.0], summing to 7.9. naive_estimate = 4/6 = 0.6667. ips_estimate = 7.9/6 = 1.3167. snips_estimate = 7.9 / (0.4+2.0+0.4+0.5+2.0+5.0) = 7.9/10.3 = 0.7670. Running the code prints exactly:

Naive (on-policy) average reward: 0.6667
IPS estimate of V(pi1):           1.3167
SNIPS estimate of V(pi1):         0.7670

The naive number, 0.6667, is a perfectly valid estimate of something, just not of V(π1); it estimates V(π0), the reward rate of the policy already in production. IPS, 1.3167, is impossible on its face: rewards are 0/1, so no policy's expected reward can exceed 1. IPS is unbiased in expectation over repeated logging, but on this one finite sample it is dominated by u6's single enormous weight of 5.0, which happened to land on a reward of 1. That single rare, barely-explored data point is doing almost two-thirds of the numerator's work. SNIPS, which normalizes by the sum of the weights instead of by n, is guaranteed to stay inside [0, 1]: it is a weighted average of the r_i (each in {0,1}) with non-negative weights that sum to 1 once divided through, so it is a convex combination of numbers in [0,1] and cannot leave that interval. 0.7670 is the trustworthy read from this log.

When importance weights explode: variance, positivity, and self-normalization

The mechanism behind IPS's misbehavior here is variance, not bias. A single logged impression with a tiny π0(a|x) generates a huge weight, and a huge weight multiplying a single Bernoulli reward has huge variance: get unlucky (or lucky, depending on your target metric) on that one row and the whole estimate swings wildly. This is exactly the pattern Bottou et al. (2013) document from live advertising systems, and it is why production off-policy pipelines almost never use raw IPS unmodified. Two standard fixes exist. The first, weight clipping, caps w_i at some threshold M before use, trading a small, controllable bias for a large reduction in variance; clipping w6 at, say, 2.0 instead of 5.0 would have pulled the IPS estimate back toward something sane at the cost of systematically underweighting genuinely rare, high-value actions. The second, self-normalization (the SNIPS estimator above), was formalized by Adith Swaminathan and Thorsten Joachims in "The Self-Normalized Estimator for Counterfactual Learning" (NeurIPS 2015): dividing by Σw_i instead of n is a small-sample-consistent trick that removes the pathology directly, at the cost of making SNIPS a ratio of two random quantities rather than a clean sample mean, which complicates its variance analysis but almost always helps in practice.

Neither fix rescues you from a structural failure: positivity. If a candidate policy wants to show an action that the logging policy had zero probability of ever showing in some context, π0(a|x) = 0 there, w(x,a) is (division by zero), and no amount of reweighting recovers a value for that slice, because the log genuinely contains no information about it. This is why production bandit systems (John Langford and collaborators built much of the open-source tooling around this, notably Vowpal Wabbit's contextual bandit module) insist on keeping every action's logging probability bounded away from zero wherever future candidate policies might plausibly want to explore it. It is also why Lihong Li, Wei Chu, John Langford, and Xuanhui Wang's "replay" method for offline bandit evaluation ("Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms," WSDM 2011) required a fully randomized logging policy and simply discarded any logged event where the evaluated policy's choice did not match the logged action: replay is IPS's degenerate special case for a uniformly-random π0, and it is unbiased but wastes almost all the data, which is precisely the inefficiency IPS and its variants were built to fix.

Doubly robust evaluation: combining a reward model with importance weights

IPS and SNIPS use only the propensities, never modeling the reward itself. An alternative, the direct method, does the opposite: fit a reward model r̂(x,a) to the logged data (e.g., logistic regression predicting click probability from context and action features), then estimate V(π1) = E_x[Σ_a π1(a|x) r̂(x,a)] purely from the model's predictions, with no reweighting at all. This has low variance (no importance weights to blow up) but is only as good as the reward model, and is biased whenever that model is wrong in a way correlated with how π1 differs from π0.

Miroslav Dudík, John Langford, and Lihong Li combined both in "Doubly Robust Policy Evaluation and Learning" (ICML 2011). The doubly robust (DR) estimator starts from the direct method's model-based guess, then adds an importance-weighted correction for how wrong that guess was on the action actually observed:

V̂_DR(π1) = (1/n) Σ_i [ dm_i + w_i·(r_i - r̂(x_i,a_i)) ], where dm_i = Σ_a π1(a|x_i) r̂(x_i,a)

The name comes from its guarantee: V̂_DR stays unbiased if either the propensities π0 are correct or the reward model r̂ is correct, even when the other piece is wrong. Continue the worked example, assuming a reward model r̂ has already been fit on the logged data (a helper not shown here; these are hypothetical model outputs used to illustrate the mechanism):

# Doubly robust: assume a reward model r_hat(x, a) was already fit on the logged
# data (e.g. logistic regression) -- assumed helper output, not derived here.
r_hat_taken = [0.7, 0.2, 0.6, 0.8, 0.4, 0.9]   # r_hat(x_i, a_i): model's guess for the action shown
dm_estimate = [0.5, 0.5, 0.5, 0.3, 0.5, 0.3]   # sum_a pi1(a|x_i) * r_hat(x_i, a)

dr_terms = [
    dm + w * (r - rh)
    for dm, w, r, rh in zip(dm_estimate, weights, rewards, r_hat_taken)
]
dr_estimate = sum(dr_terms) / len(dr_terms)

print(f"Doubly robust estimate of V(pi1): {dr_estimate:.4f}")

Trace u6, the row that broke plain IPS: w6 = 5.0, r6 = 1, r̂(x6,a6) = 0.9, so the residual r6 - r̂6 is only 0.1, and the term is dm6 + w6·0.1 = 0.3 + 0.5 = 0.8, versus IPS's raw contribution of w6·r6 = 5.0. The huge weight is still there, but it now multiplies a small residual instead of the full reward, because the reward model already predicted this row correctly. Working through all six rows the same way (dm + w·(r-r̂) for each) gives terms [0.62, 0.10, 0.26, 0.40, 1.70, 0.80], summing to 3.88, so dr_estimate = 3.88/6 = 0.6467, printed by the code above as 0.6467. This is not a general property (DR does not always land near the naive estimate), but the mechanism it demonstrates is general: whenever the reward model is roughly right on a rare, high-weight row, DR's variance contribution from that row shrinks; the estimator remains unbiased as long as either the model or the propensities are correct, even where the other is not.

A common misconception

Misconception: "My new ranker has higher held-out AUC (or lower log-loss) than the old one on the logged click data, so it will perform better if I deploy it." Correction: AUC on logged data measures how well a score separates clicks from non-clicks among the exposures π0 actually generated. It says nothing about V(π1), because π1 would generate a different exposure pattern entirely, and the log contains no ground truth for the counterfactual exposures π1 would have produced. Two rerankers can have statistically indistinguishable offline AUC while differing sharply in V(π1), if one of them happens to reweight probability mass toward actions π0 rarely explored (where the log carries little counterfactual signal at all) while the other stays close to π0's own action distribution (where the log is informative). Predictive accuracy on factual outcomes and causal value under a new policy are different quantities that happen to correlate when the candidate policy barely differs from the logging policy, and decouple sharply as they diverge. The only way to know V(π1) from offline data is to actually run the importance-weighted estimators above, not a predictive-accuracy metric.

Where this shows up in production

Off-policy evaluation is the standard first gate before any ranking, recommendation, or ad-serving policy reaches a live experiment at scale: it lets a team screen five or fifty candidate policies against one exposure log, discard the ones with poor or wildly uncertain V̂, and reserve the expensive, risk-bearing online A/B test for the one or two survivors. It also explains a design constraint that surprises engineers new to these systems: production recommenders deliberately keep some randomness in their serving policy (an explicit exploration bucket, or logging softmax temperature) even when a fully greedy policy would look better on every offline predictive metric, because a completely deterministic π0 assigns zero probability to everything except its own top choice, which makes every future candidate policy's positivity requirement fail everywhere except where π0 already agreed with it. Paying a small amount of ranking quality now for guaranteed overlap later is the price of being able to counterfactually evaluate anything at all next quarter.

Active recall

Attempt each question before reading its answer.

1. Why can't you estimate V(π1) by simply averaging the rewards in a log collected under π0?
2. Show algebraically what the IPS estimator reduces to when π1 = π0 exactly (evaluating the deployed policy against itself).
3. In the worked example, suppose u6's logging propensity was actually π0 = 0.10 instead of 0.02 (a less aggressive exploration bucket), with π1 = 0.10 unchanged and everything else identical. Recompute the weights, IPS, SNIPS, and DR estimates, and state whether IPS still breaches the valid [0,1] range.
4. A team wants to evaluate a policy π1 that always shows item D in some context, but the logging policy π0 never showed item D in that context at all (π0(D|x)=0 there). Can IPS, SNIPS, or DR produce a valid estimate for that slice? Why or why not?
5. Two candidate rankers have identical AUC on a held-out logged test set. Does this guarantee they will produce the same V(π1) if deployed?
6. Prove that SNIPS is always between 0 and 1 for binary rewards and non-negative weights, while plain IPS carries no such guarantee.

Answers.

1. The average reward of the log is an unbiased estimator of V(π0), the value of the policy that generated the data, not V(π1). Each context in the log has an observed reward for only the one action π0 actually chose; the reward under any other action, including whatever π1 would pick, is a missing potential outcome that the raw average cannot recover.

2. If π1 = π0, then w_i = π1(a_i|x_i)/π0(a_i|x_i) = 1 for every logged row, since the numerator and denominator are the same function evaluated at the same point. IPS then becomes (1/n)Σ1·r_i = (1/n)Σr_i, exactly the naive average. This is the correct sanity check: IPS must collapse to the plain on-policy estimate when there is no policy change to correct for.

3. New weight for u6: w6 = 0.10/0.10 = 1.0 (down from 5.0); all other weights are unchanged at [0.4, 2.0, 0.4, 0.5, 2.0]. Sum of w·r: previously 7.9, now the u6 term drops from 5.0 to 1.0, giving 0.4+0+0+0.5+2.0+1.0 = 3.9. Sum of weights: previously 10.3, now 0.4+2.0+0.4+0.5+2.0+1.0 = 6.3. New IPS = 3.9/6 = 0.65 — no longer breaches the [0,1] range; the earlier breach was entirely an artifact of the rare, extreme weight, and removing that extremity removes the symptom too. New SNIPS = 3.9/6.3 = 0.6190. For DR, only u6's term changes: dm6 + w6·(r6 - r̂6) = 0.3 + 1.0·(1-0.9) = 0.3 + 0.1 = 0.4 (down from 0.80); the new DR sum is 0.62+0.10+0.26+0.40+1.70+0.40 = 3.48, giving DR = 3.48/6 = 0.58. All three estimators shift, and the diagnostic signal itself (IPS exceeding the impossible range) disappears along with the extreme weight that caused it.

4. No. If π0(D|x) = 0 exactly, the weight π1(D|x)/π0(D|x) is (division by zero) for that context, and none of IPS, SNIPS, or DR's importance-weighted correction term can be computed there; DR's direct-method component alone could still produce a number, but it would carry no correction from real outcome data, so it inherits all of the reward model's potential bias with no way to check it against logged evidence. This is the positivity/overlap violation: no reweighting scheme can manufacture information about an action region the logging policy never explored. The only real fix is to run π0 with nonzero probability on item D going forward (or in a dedicated exploration bucket) before trying to evaluate policies that favor it.

5. No. AUC measures predictive separation on the factual exposures π0 generated; it does not account for how the ranker's own decisions would change which exposures occur, which is exactly the quantity the missing-potential-outcomes problem hides. Two rankers with identical offline AUC can have very different V(π1) if they redistribute probability mass differently across actions the logging policy explored unevenly; only an off-policy estimator (IPS, SNIPS, or DR) run against the logged propensities answers the deployment-value question.

6. SNIPS = Σw_ir_i / Σw_i = Σ(w_i/Σw_j)·r_i. The coefficients w_i/Σw_j are non-negative (weights are ratios of probabilities, hence ≥0) and sum to exactly 1 by construction, so SNIPS is a convex combination of the r_i. Any convex combination of numbers in [0,1] is itself in [0,1], so SNIPS ∈ [0,1] always. IPS = (1/n)Σw_ir_i instead divides by the fixed constant n rather than by Σw_i, so its "coefficients" w_i/n need not sum to 1 (they sum to Σw_i/n, which can be far above or below 1), which is exactly why IPS is not a convex combination and can land outside [0,1], as it did in the worked example.

Think About It

Think about this: How would you explain causal inference in machine learning to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind causal inference in machine learning, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.