AI Interpretability: Understanding Model Decisions Through SHAP, LIME, and Mechanistic Analysis

In September 2022 the Reserve Bank of India's Guidelines on Digital Lending made a specific demand of every NBFC and fintech running an automated underwriting model: you must assess the borrower's creditworthiness "in an auditable way." A gradient-boosted tree with 200 features and 800 trees does not produce an audit trail. It hands you a number, 0.31, below the 0.5 approval threshold, and the loan officer has no idea whether that 0.31 came from a thin credit history, an income mismatch, or a bug in how the pipeline encoded the applicant's PIN code. This is not a hypothetical compliance headache. It is the exact gap that the two techniques in this chapter's title, SHAP and LIME, were built to close, and it is also precisely where they stop being enough, which is why the chapter's third technique, mechanistic interpretability, exists at all. All three answer some version of "why did the model say what it said," but they answer it from different distances: SHAP and LIME query the model from the outside, as a black box, while mechanistic analysis opens the weights and reads the mechanism directly.

Three different questions hiding inside one word

"Explain this prediction" is ambiguous in a way that matters for engineering choices. It can mean: (a) how much did each input feature push this one prediction away from a baseline, (b) what would a simple, human-readable model say in the immediate neighbourhood of this one input, or (c) what computation, in terms of the actual neurons, attention heads, and weight matrices, produced this output. SHAP answers (a). LIME answers (b). Mechanistic interpretability answers (c). All three are post-hoc in the SHAP/LIME case, meaning they explain a model that was never designed to be interpretable, applied after training. Mechanistic interpretability is post-hoc too, in the sense that it is not baked into training, but it is not a local approximation of the function; it is closer to reverse-engineering a compiled binary. Keeping these three questions distinct is the single most useful discipline in this chapter, because the most common way students misuse these tools is to treat a SHAP value, a LIME coefficient, and a "this neuron detects sarcasm" claim as three flavours of the same fact. They are not. The worked examples below will make the difference load-bearing rather than definitional.

SHAP: fair credit-splitting borrowed from cooperative game theory

SHAP (SHapley Additive exPlanations), introduced by Scott Lundberg and Su-In Lee at NeurIPS 2017, is built entirely on a 1953 result from game theory: Lloyd Shapley's method for splitting a payoff among players in a cooperative game so that the split satisfies four fairness axioms (efficiency, symmetry, dummy, additivity). Lundberg and Lee's insight was to treat a model's input features as the "players" and the model's prediction as the "payoff," then ask: if features can join a prediction one at a time, in any order, how much does each feature's presence change the output on average, across every possible order?

Formally, for a prediction on instance x with feature set N, and a value function v(S) giving what the model would output if only the features in coalition S ⊆ N were "known" (the rest replaced by a baseline), the Shapley value for feature i is

φ_i = Σ_{S ⊆ N\{i}}  [ |S|! (n-|S|-1)! / n! ]  ×  [ v(S ∪ {i}) - v(S) ]

Read this as: for every possible subset of the other features, measure the marginal effect of adding feature i to that subset, then average those marginal effects with weights that make every ordering of the n features equally likely. The one property that makes this worth the computational cost is efficiency: the φ values always sum exactly to v(N) - v(∅), the actual gap between the real prediction and the baseline prediction. No approximation, no leftover unexplained residual.

Worked example: underwriting a credit line

Take a toy version of the RBI scenario. A model scores a credit-line application out of 100 using two binary signals: x1, whether the applicant has a stable, verifiable salary credit, and x2, whether they have six or more months of UPI transaction history. Suppose the (black-box, but let's peek for the sake of the worked example) scoring function is

f(x1, x2) = 20 + 30·x1 + 25·x2 + 15·x1·x2

The 15·x1·x2 term is an interaction: having both signals together is worth more than the two effects added separately, which is exactly the kind of behaviour a linear scorecard cannot represent and a tree ensemble learns easily. Our applicant has both: x1=1, x2=1, giving f(1,1)=90. The baseline, an applicant with neither signal, is f(0,0)=20. The question SHAP answers: of the 70-point gap between 90 and 20, how much belongs to the income signal and how much to the UPI history?

Compute the value function for every coalition, replacing an absent feature with the baseline value 0:

v(∅)     = f(0,0) = 20
v({1})   = f(1,0) = 50
v({2})   = f(0,1) = 45
v({1,2}) = f(1,1) = 90

With two features there are only two orderings to average over for each feature, so the sum in the formula reduces to a simple average of the marginal contribution when a feature joins first and when it joins second:

φ1 = ½[v({1}) - v(∅)] + ½[v({1,2}) - v({2})]
   = ½(50-20) + ½(90-45) = 15 + 22.5 = 37.5

φ2 = ½[v({2}) - v(∅)] + ½[v({1,2}) - v({1})]
   = ½(45-20) + ½(90-50) = 12.5 + 20 = 32.5

Check efficiency: 37.5 + 32.5 = 70 = v({1,2}) - v(∅) = 90 - 20. Exact, every time, by construction. Notice what the 15-point interaction term did to the split: neither φ is 30 or 25 (the "solo" coefficients), because each feature gets credited for half of the interaction bonus it helps unlock, whichever order you imagine the features "arriving" in. This brute-force computation, generalized to n features by enumerating all 2^n coalitions, is exactly what the following code performs and verifies:

from itertools import combinations
from math import factorial

def f(x1, x2):
    return 20 + 30*x1 + 25*x2 + 15*x1*x2

def v(subset, x, baseline):
    x1 = x[0] if 0 in subset else baseline[0]
    x2 = x[1] if 1 in subset else baseline[1]
    return f(x1, x2)

def shapley(feature_idx, n_features, x, baseline):
    others = [i for i in range(n_features) if i != feature_idx]
    total = 0.0
    for r in range(len(others) + 1):
        for S in combinations(others, r):
            weight = factorial(r) * factorial(n_features - r - 1) / factorial(n_features)
            total += weight * (v(S + (feature_idx,), x, baseline) - v(S, x, baseline))
    return total

x = (1, 1)
baseline = (0, 0)
phi1 = shapley(0, 2, x, baseline)
phi2 = shapley(1, 2, x, baseline)
print(phi1, phi2, phi1 + phi2)
# 37.5 32.5 70.0

This brute-force approach is exact but scales as 2^n, which becomes unusable past roughly 20 features. Production explainability libraries use two shortcuts: KernelSHAP, which samples coalitions and fits a weighted regression to approximate the same φ values, and TreeSHAP (Lundberg, Erion, and Lee, 2018–2020), which exploits the recursive structure of decision trees to compute the exact Shapley values in low-order polynomial time. A real NBFC underwriting model with 150 engineered features runs TreeSHAP, not brute force.

LIME: a linear ruler held up to one small patch of a curved surface

LIME (Local Interpretable Model-agnostic Explanations), introduced by Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin at KDD 2016, takes a different route to the same goal. Instead of a game-theoretic split of credit, LIME asks: near this one instance, can I fit a simple, interpretable model, typically a sparse linear regression, that mimics the black box closely enough to trust its coefficients as a local explanation? The recipe: generate perturbed samples z around the instance x, query the real model f for each perturbed sample's output, weight each sample by a proximity kernel π_x(z) that favours points close to x, and fit an interpretable model g that minimizes a weighted loss plus a complexity penalty:

ξ(x) = argmin_g  L(f, g, π_x) + Ω(g)

Applied to the same credit model and the same instance (1,1), using the exhaustive perturbation set of all four binary combinations (a real deployment would sample many more, but our feature space only has four points total) and an exponential proximity kernel on Hamming distance, π(z) = exp(-D(x,z)² / σ²) with σ=1:

z=(0,0): D=2, weight = e^-4 ≈ 0.0183, f(z)=20
z=(1,0): D=1, weight = e^-1 ≈ 0.3679, f(z)=50
z=(0,1): D=1, weight = e^-1 ≈ 0.3679, f(z)=45
z=(1,1): D=0, weight = e^0  = 1.0000, f(z)=90

Fitting a weighted linear model g(z) = β0 + β1·z1 + β2·z2 to these four points by weighted least squares:

import numpy as np

Z = np.array([[1, 0, 0],
              [1, 1, 0],
              [1, 0, 1],
              [1, 1, 1]])          # columns: intercept, x1, x2
y = np.array([20, 50, 45, 90])      # f(z) queried from the black box

hamming = np.array([2, 1, 1, 0])    # distance of each z from the instance (1,1)
weights = np.exp(-(hamming**2) / 1.0)

W = np.diag(weights)
beta = np.linalg.solve(Z.T @ W @ Z, Z.T @ W @ y)
print(beta)                 # [ 6.58186135 44.08618843 39.08618843]
print(beta[1] + beta[2])    # 83.17237686375952

The fitted surrogate is g(z) ≈ 6.58 + 44.09·z1 + 39.09·z2. Check it against nearby points: g(1,1)=89.75 (true 90, close, because that point had the largest weight), g(1,0)=50.67 (true 50, close), but g(0,0)=6.58 (true 20, far off, because that corner was weighted almost to zero and the local model was never asked to fit it well). That last mismatch is not a bug; it is LIME doing exactly what it is designed to do: buy local fidelity near x at the cost of global fidelity everywhere else.

The misconception: these numbers are not a property of the model

Here is the mistake nearly every student makes on first contact with SHAP: treating φ1 and φ2 as intrinsic, model-level "feature importances," the way a coin's mass is a property of the coin regardless of who weighs it. They are not. A Shapley value is only defined relative to a chosen baseline (the v(∅) reference point, often called the background distribution), and changing that baseline changes the answer, sometimes drastically enough to flip which feature looks more important. Re-run the exact same trained model, the exact same instance (1,1), but change the baseline from "an applicant with neither signal" (0,0) to "the average applicant in the training set," say (0.7, 0.3), meaning 70% of applicants have stable income and 30% have UPI history:

v(∅)     = f(0.7,0.3) = 20 + 21 + 7.5 + 3.15  = 51.65
v({1})   = f(1,0.3)   = 20 + 30 + 7.5 + 4.5   = 62.00
v({2})   = f(0.7,1)   = 20 + 21 + 25 + 10.5   = 76.50
v({1,2}) = f(1,1)     = 90.00

φ1 = ½(62.00-51.65) + ½(90-76.50) = 5.175 + 6.750 = 11.925
φ2 = ½(76.50-51.65) + ½(90-62.00) = 12.425 + 14.00 = 26.425

Sum check: 11.925 + 26.425 = 38.35 = 90 - 51.65, still exact, still efficient. But look at the ranking: under the zero baseline, income dominated (37.5 vs 32.5). Under the population-mean baseline, UPI history dominates by more than two-to-one (26.425 vs 11.925). Same model, same trained weights, same applicant, same prediction of 90, and the "most important feature" flips. This is not an error in either computation; both are correct Shapley values, for two different, equally legitimate questions ("how much does this applicant differ from having nothing?" versus "how much does this applicant differ from an average applicant?"). Two data scientists reporting different SHAP values for the same row, on the same model, are not necessarily disagreeing about the model. They are very likely using different background datasets, and neither number is "the" feature importance until the baseline is stated alongside it. This is the detail that regulatory and audit contexts, like RBI's auditable-underwriting requirement, cannot skip: an explanation without a stated reference point is not falsifiable.

LIME has the analogous trap in a different place: its explanation depends on the kernel width σ and the perturbation sampling distribution, not just the baseline. A narrower kernel weights only extremely close neighbours and can produce a locally accurate but wildly unstable surrogate (small perturbations of x itself can flip the sign of a coefficient); a wider kernel smooths over exactly the nonlinear or interaction behaviour, like our 15·x1·x2 term, that made the model worth using over a plain linear scorecard in the first place. Notice that even our best-fit LIME surrogate has β1+β2 = 83.17, overshooting the true, efficiency-guaranteed total effect of 70 by 13.17 points, precisely because a linear model has no way to represent an interaction term and instead smears part of it into both coefficients.

Beyond attribution: opening the box

SHAP and LIME both stop at the model's input-output boundary; they never look at a weight matrix. Mechanistic interpretability, the newest and least mature of the three approaches, tries to reverse-engineer the actual algorithm a trained network implements, in the same spirit as reading disassembled machine code. The methodology traces back to Chris Olah and collaborators' "Zoom In" agenda (Olah et al., 2020, Distill), which proposed that networks contain identifiable "circuits," small sub-graphs of neurons and weights implementing a specific, nameable computation, and that these circuits could in principle be found and verified the way a chip designer verifies a logic block. For transformers specifically, Elhage et al. (2021, Anthropic, "A Mathematical Framework for Transformer Circuits") decomposed attention layers into interpretable linear algebra (the QK circuit deciding where to attend, the OV circuit deciding what gets moved), and Olsson et al. (2022, Anthropic) used that framework to identify "induction heads," attention heads that implement a simple prefix-matching-and-copy algorithm and that emerge sharply during training in a way strongly correlated with a model's in-context learning ability.

The key experimental move that separates mechanistic interpretability from SHAP/LIME-style attribution is causal intervention, not correlation. The dominant technique is activation patching (used at scale in Meng, Bau, Andonian, and Belinkov's 2022 ROME paper on locating and editing factual associations in GPT-style models): run the model on a "corrupted" input where some fact is scrambled, then splice in ("patch") the clean run's activation at one specific layer and position, and measure how much of the correct output is restored. If patching a single attention head at a single token position recovers most of the correct answer while patching any other head barely moves it, that is direct causal evidence that computation is localized there, not merely a correlation between that head's activation and the output. This is a fundamentally stronger claim than "removing this feature changed the SHAP value," because it operates on the mechanism, not on a fitted local surrogate of the input-output map. The frontier of this line of work, sparse autoencoders that decompose a layer's superposed, polysemantic neuron activations into a much larger set of monosemantic "features" (Bricken et al., 2023, Anthropic, "Towards Monosemanticity"), exists precisely because individual neurons in a dense network usually do not correspond to single human-interpretable concepts; the concepts live in directions in activation space that sparse coding can recover, one more reason "just look at which neuron fired" is not a substitute for causal testing.

How the three approaches relate

Active recall

Attempt each question before reading its answer.

1. Why must Shapley values for a prediction always sum exactly to f(x) - v(∅), while LIME's local-model coefficients have no such guarantee?
2. A two-feature value function gives v(∅)=10, v({1})=40, v({2})=25, v({1,2})=60. Compute φ1 and φ2 by hand and check the efficiency property.
3. Two engineers compute SHAP values for the same row, from the same trained model, and get different numbers. Neither made an arithmetic error. What is the most likely cause?
4. Take the credit model from the worked example but raise the interaction coefficient from 15 to 40: f(x1,x2) = 20 + 30x1 + 25x2 + 40x1x2, keeping the baseline at (0,0) and the LIME kernel weights unchanged. Recompute all four coalition values, the SHAP φ1/φ2, and the LIME weighted-regression coefficients. What happens to the gap between the SHAP total and the LIME coefficient total, and why?
5. An activation-patching experiment finds that patching one specific attention head restores 81 percentage points of accuracy on an induction task, while patching any other single head changes accuracy by less than 2 points. Is this sufficient to claim "this head implements the induction algorithm"? What additional evidence would strengthen the claim?
6. TreeSHAP computes exact Shapley values for gradient-boosted trees in polynomial time instead of the brute-force 2^n-coalition enumeration used in this chapter's code. Why does this matter for an NBFC underwriting model with 150 features?

Answers

1. Efficiency is one of the four axioms Shapley (1953) proved uniquely characterize this credit-splitting scheme; the weighted average of marginal contributions across every possible feature ordering is constructed so the terms telescope to exactly v(N)-v(∅). LIME fits a linear model by minimizing a weighted loss with no such constraint built in; its coefficients only need to fit the weighted sample points well, not sum to any particular target, which is exactly why our example's coefficients summed to 83.17 instead of the true 70.

2. φ1 = ½(40-10) + ½(60-25) = 15 + 17.5 = 32.5. φ2 = ½(25-10) + ½(60-40) = 7.5 + 10 = 17.5. Sum = 50 = 60-10, matching v({1,2})-v(∅). Efficiency holds.

3. Different background/baseline data. As shown in the misconception section, changing the baseline from an all-zero reference to the population-mean applicant changed φ1 from 37.5 to 11.925 and φ2 from 32.5 to 26.425 on the identical model and identical prediction, even flipping which feature ranked higher. A SHAP value is only meaningful alongside a stated baseline.

4. New coalition values: v(∅)=20, v({1})=50, v({2})=45, v({1,2})=20+30+25+40=115. SHAP: φ1=½(50-20)+½(115-45)=15+35=50, φ2=½(45-20)+½(115-50)=12.5+32.5=45, sum =95=115-20 (still exact). LIME, refitting the same weighted regression with the new y values (20,50,45,115) and unchanged weights (0.0183, 0.368, 0.368, 1.0), gives β≈(-15.78, 67.56, 62.56), coefficient sum ≈130.13. The SHAP-LIME gap widens from 83.17-70=13.17 to 130.13-95=35.13, roughly 2.7×, tracking the roughly 2.7× increase in the interaction coefficient (40/15). The intercept also flips negative, a further symptom: a linear surrogate has no way to represent curvature, so as the true interaction strength grows, the local plane tilts further to compensate near the instance and becomes correspondingly worse away from it.

5. Necessary but not sufficient. Patching demonstrates the head is causally necessary for recovering the behaviour on this task, which already rules out a merely correlational reading. But "implements the induction algorithm" is a specific mechanistic claim (prefix-matching plus copying via the QK/OV circuits, per Elhage et al., 2021 and Olsson et al., 2022), and confirming it needs more: inspecting whether the head's QK circuit actually attends back to prior occurrences of the current token, and whether its OV circuit copies the token that followed that prior occurrence, ideally on synthetic repeated-token sequences engineered to isolate exactly that mechanism and rule out confounds like positional heuristics.

6. Brute-force Shapley evaluates 2^n coalitions per explained row; at 150 features that is roughly 10^45 subsets, wildly infeasible even for one prediction, let alone the millions scored daily. TreeSHAP (Lundberg, Erion, and Lee) exploits the fact that a decision tree only branches on one feature per node, letting it compute the exact same Shapley values by tracking coalitions along tree paths in time polynomial in the number of trees, leaves, and depth, which is exactly what makes SHAP explanations feasible to attach to every real-time underwriting decision rather than a hand-picked sample.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where ai interpretability: understanding model decisions through shap, lime, and mechanistic analysis is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting ai interpretability: understanding model decisions through shap, lime, and mechanistic analysis to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind ai interpretability: understanding model decisions through shap, lime, and mechanistic analysis, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.