Interpretability Methods: Understanding Model Internals

Where does a fact live inside a language model?

Suppose your team fine-tunes an open-weights language model on an Indian civics and general-knowledge corpus, and it starts confidently answering "The capital of Rajasthan is Jaipur." Later, after a second round of fine-tuning on a batch of news articles, the same model starts answering "The capital of Rajasthan is Udaipur" — plausible-sounding, but wrong. An input-attribution method like SHAP or LIME can tell you which input tokens — "Rajasthan," "capital" — the wrong output depended on most. That is useful, but it does not answer the question an engineer actually needs answered before touching the model: where inside the network's many layers did the correct fact get computed, and where did it get overwritten or lost? Attribution tells you what the model looked at. It says nothing about which of its internal computations — which layer, which sub-block, which token position — actually carried the answer forward to the output.

This chapter is about a different family of methods that answer that second question directly: causal tracing and, more generally, activation patching. Instead of asking which input features correlate with an output, these methods intervene surgically on the network's own intermediate computations and measure the causal effect of that intervention. The technique was formalized by Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov in "Locating and Editing Factual Associations in GPT" (NeurIPS 2022), the paper that introduced ROME — Rank-One Model Editing. It is one of the most consequential interpretability results of the last several years, because it did not just explain a model's behavior after the fact — it located the exact computation responsible for a specific fact precisely enough that the authors could then edit that fact by changing a single weight matrix.

The core idea: causal mediation, not correlation

Causal tracing borrows a tool from causal inference, causal mediation analysis, and applies it to a transformer's own hidden states. The procedure runs three forward passes over the same prompt.

The clean run. The prompt "The capital of Rajasthan is" is passed through the network normally. Every hidden state at every layer and every token position is cached. The model outputs the correct continuation, "Jaipur," with high probability.

The corrupted run. The same prompt is passed through again, but before the first layer, Gaussian noise is added to the embedding of the subject tokens, "Rajasthan." This deliberately destroys the model's ability to identify which entity is being asked about, without touching anything else about the sentence structure. The corrupted run's output probability for "Jaipur" collapses, because the network genuinely no longer has clean information about the subject.

The corrupted-and-patched run. The corrupted run is repeated, except that at one single layer-and-token-position location, the hidden state is overwritten with the value cached from the clean run at that exact location. Everything else in the forward pass stays corrupted. The effect of this one substitution on the final output probability is then measured.

This third step is repeated separately for every layer and every token position, one restoration at a time, producing a grid of scores, a causal trace, showing how much each individual location in the network, if restored to its clean value, single-handedly recovers the correct answer. A location whose restoration recovers most of the probability was carrying most of the causally relevant information. A location whose restoration does nothing was, causally, irrelevant to the fact, even if an attribution method might assign it a nonzero importance score for some indirect, correlational reason.

Formally, if P(correct | run) is the model's probability on the correct token under a given run, the Average Indirect Effect (AIE) of restoring location L is:

AIE(L) = P(correct | corrupted, patch L) − P(correct | corrupted)

averaged over many prompts expressing different facts of the same relational type. A large AIE at a specific layer and token position is direct causal evidence that the computation performed there is load-bearing for the fact.

A worked trace: locating "capital of Rajasthan → Jaipur"

Real transformers make this tractable because of one structural property: the residual stream is additive. Every attention sub-block and every MLP sub-block does not overwrite the running hidden state, it adds a correction vector onto it. The hidden state entering the unembedding matrix at the final token is the sum of the token embedding plus every layer's output vector. This means a small, fully transparent toy model can capture the causal-tracing arithmetic exactly, without needing a full-scale network to illustrate the method.

Restrict attention to two candidate output tokens, "Jaipur" (correct) and "Mumbai" (a plausible but wrong Indian-city distractor), and suppose each of three toy layers contributes an additive amount to each of these two logits. In the clean run, the subject "Rajasthan" is processed correctly, so the middle layer injects a strong, fact-specific boost toward "Jaipur":

Summing columns gives total logits (4.0, 0.8). Converting to a probability with the two-way softmax P = e^a / (e^a + e^b): e^4.0 ≈ 54.598, e^0.8 ≈ 2.226, giving P(Jaipur | clean) ≈ 0.9608. The model is confident and correct.

Now noise the subject embedding before layer 1. Layer 1 itself is unaffected, since it has not yet used subject identity, but layers 2 and 3, which depend on knowing the subject is "Rajasthan," can no longer inject a fact-specific boost:

Totals: (1.2, 1.25). e^1.2 ≈ 3.320, e^1.25 ≈ 3.490, giving P(Jaipur | corrupted) ≈ 0.4875, almost coin-flip, in fact very slightly favoring the wrong city. The corruption has genuinely erased the fact.

Now run the patching experiment: take the corrupted run, but restore only layer 2's contribution to its clean value (2.0, 0.2), keeping layers 1 and 3 corrupted. Totals become (0.5+2.0+0.4, 0.5+0.2+0.35) = (2.9, 1.05). e^2.9 ≈ 18.174, e^1.05 ≈ 2.858, giving P(Jaipur | patch L2) ≈ 0.8641. Restoring layer 2 alone recovers most of the confidence. Patching layer 3 instead, with layer 2 left corrupted, gives totals (2.3, 1.0) and P(Jaipur | patch L3) ≈ 0.7858, a real but smaller recovery. Patching layer 1 changes nothing, because layer 1's clean and corrupted contributions are identical (0.5, 0.5); there was never any subject-specific information there to restore.

The resulting AIE values are AIE(L1) = 0, AIE(L2) = 0.8641 − 0.4875 = 0.3766, and AIE(L3) = 0.7858 − 0.4875 = 0.2983. Layer 2 has the largest causal effect: it is the layer where the fact "Rajasthan to Jaipur" is actually being retrieved and injected, and layer 3 is mostly forwarding that information onward. This is exactly the pattern of reasoning Meng et al. used, at scale, across hundreds of factual prompts and every layer-and-token pair, not three toy layers, to localize factual recall in GPT-2-XL and GPT-J.

This entire toy calculation is short enough to run and check independently:

import math

def softmax2(logit_a, logit_b):
    ea, eb = math.exp(logit_a), math.exp(logit_b)
    return ea / (ea + eb), eb / (ea + eb)

def total_logits(layer_contributions):
    logit_a = sum(pair[0] for pair in layer_contributions)
    logit_b = sum(pair[1] for pair in layer_contributions)
    return logit_a, logit_b

# per-layer additive contribution to (logit_Jaipur, logit_Mumbai)
clean = [(0.5, 0.5), (2.0, 0.2), (1.5, 0.1)]
corrupted = [(0.5, 0.5), (0.3, 0.4), (0.4, 0.35)]

p_clean = softmax2(*total_logits(clean))
p_corrupted = softmax2(*total_logits(corrupted))

print(f"P(Jaipur | clean)     = {p_clean[0]:.4f}")
print(f"P(Jaipur | corrupted) = {p_corrupted[0]:.4f}")

for i, layer_name in enumerate(["Layer 1", "Layer 2", "Layer 3"]):
    patched = corrupted.copy()
    patched[i] = clean[i]                 # restore the clean value at layer i only
    p_patched = softmax2(*total_logits(patched))
    aie = p_patched[0] - p_corrupted[0]
    print(f"Patch {layer_name}: P(Jaipur) = {p_patched[0]:.4f}, AIE = {aie:.4f}")

Running this prints exactly:

P(Jaipur | clean)     = 0.9608
P(Jaipur | corrupted) = 0.4875
Patch Layer 1: P(Jaipur) = 0.4875, AIE = 0.0000
Patch Layer 2: P(Jaipur) = 0.8641, AIE = 0.3766
Patch Layer 3: P(Jaipur) = 0.7858, AIE = 0.2983

What causal tracing found inside real GPT models

Meng et al. ran exactly this procedure, noise-corrupt the subject, restore one hidden state at a time, across GPT-2-XL (48 layers, 1.5B parameters) and GPT-J (6B parameters), sweeping layer-and-token pairs for over a thousand factual prompts such as "The Space Needle is located in the city of ___." Two sharp, separated peaks of causal effect emerged, not one diffuse blob.

An early site: restoring the MLP output at the last subject token in a narrow band of middle layers, roughly layer 15 to layer 18 of GPT-2-XL's 48, produced almost all of the recovery, echoing exactly what layer 2 did in the toy trace above. A late site: restoring attention outputs at the final token position in later layers also mattered, but for a different reason, it is where the retrieved fact gets moved from the subject's position to the position where the model must actually produce an answer.

The early-site result is the more surprising and more useful one. It suggests that a mid-layer MLP block is functioning like a key-value lookup: the subject token acts as a key, and the MLP's weight matrices store an association that gets written into the residual stream as a value. That interpretation is not just a metaphor, it is precisely what motivated ROME's edit procedure. Having localized the fact to one MLP's weights at one layer, Meng et al. computed a targeted rank-one update to that weight matrix that changes "Rajasthan → Jaipur" to "Rajasthan → some other city" while leaving the model's behavior on unrelated prompts essentially untouched, a form of surgical, causally-justified model editing that no attribution method, which only explains and never edits, could have supported. A follow-up technique, path patching (Wang, Variengien, Conmy, Shlegeris, and Steinhardt, "Interpretability in the Wild," ICLR 2023, best known for reverse-engineering GPT-2's indirect-object-identification circuit), refines this further: instead of patching an entire layer's output node, it patches only the specific edge, the direct connection from one component into one downstream component, holding every other path fixed, letting researchers isolate not just which layer matters but which specific communication channel between two components carries the effect.

Diagram: tracing the fact through three forward passes

The figure below lays out the three runs from the worked example side by side, shows exactly where the patch is injected, and shows how much probability it recovers.

Common misconception: patching is not the same as ablation

Students who have already met the idea of turning off a component to see if it matters, zeroing a neuron, an attention head, or an entire layer, often assume activation patching is just a fancier name for the same thing. It is not, and the difference matters for what conclusions you are allowed to draw. Ablation, whether zero-ablation or mean-ablation, asks a blunt necessity question: if this component's output is destroyed or replaced with its average value, does performance drop? That is an out-of-distribution intervention: a hidden state of all zeros, or a constant average, is a value the network never actually produces during normal operation, so behavior under ablation can reflect artifacts of being pushed off-distribution rather than the component's real causal role. Patching, as used in causal tracing, always substitutes a real activation value that the network genuinely computed on some other legitimate input, the clean run's own hidden state. Because the substituted value is itself something the network can and does produce, the network downstream of the patch is still operating on realistic inputs; only the specific piece of information carried by that one hidden state has changed. This is why causal tracing distinguishes two directions with different names: denoising patches (restoring a clean value into a corrupted run, as done above, isolating what is sufficient to recover correct behavior) and noising patches (injecting a corrupted or counterfactual value into an otherwise clean run, isolating what is necessary to break it). Ablation collapses this into one blunt on-or-off test and cannot distinguish sufficiency from necessity; patching, run in both directions, can.

Active recall

Attempt each question before reading its answer.

Q1. A colleague says: "We already ran SHAP on this prompt and it told us 'Rajasthan' was the most important input token, so causal tracing would just tell us the same thing, why bother?" What is wrong with this reasoning?

Q2. Suppose the noise added during corruption were weaker, so that layer 2's corrupted contribution became (0.9, 0.3) instead of (0.3, 0.4), while layers 1 and 3's corrupted contributions stay the same as before. Recompute P(Jaipur | corrupted) and the AIE for patching each of the three layers. Does the layer with the largest AIE change?

Q3. True or false, with justification: "Patching a hidden state is mathematically equivalent to zeroing it out, since both remove the original information at that location."

Q4. Path patching restricts a patch to one specific edge between two components rather than an entire node's output. Concretely, what question can path patching answer that whole-layer activation patching cannot?

Q5. Given a causal trace showing effect concentrated almost entirely at the last subject token in mid-layer MLPs, and near zero everywhere else, what does this imply about how you should attempt to edit the fact, and why would editing a late-layer attention head instead be the wrong target?

Q6. In the original worked example, AIE(Layer 1) came out to exactly 0.0000, not just small. Why exactly zero, and would you expect exactly zero in a real transformer too?

Answers

A1. SHAP and LIME are input-attribution methods: they perturb or coalition-weight the input and observe how the output changes, treating everything between input and output as an opaque function. That can correctly show "the word Rajasthan mattered a lot," but it cannot say which of the network's layers, or which sub-block within a layer, performed the computation that mattered, because it never looks inside the network at all. Causal tracing intervenes on internal hidden states directly, which is the only way to answer where a computation happens, and that localization is what makes targeted weight edits like ROME possible. The two methods answer different questions and are complementary, not redundant.

A2. New corrupted totals: (0.5+0.9+0.4, 0.5+0.3+0.35) = (1.8, 1.15). e^1.8 ≈ 6.050, e^1.15 ≈ 3.158, giving P(Jaipur | corrupted) ≈ 0.6570, a much higher baseline than before, since the corruption is weaker. Patching layer 1 still does nothing (AIE = 0, since layer 1's clean and corrupted values are unchanged and identical). Patching layer 2 gives totals (2.9, 1.05) → P ≈ 0.8641 (unchanged from before, since layer 2's clean-value restoration is the same substitution as before), so AIE(L2) = 0.8641 − 0.6570 = 0.2071. Patching layer 3 gives totals (0.5+0.9+1.5, 0.5+0.3+0.1) = (2.9, 0.9) → e^2.9 ≈ 18.174, e^0.9 ≈ 2.460, P ≈ 0.8808, so AIE(L3) = 0.8808 − 0.6570 = 0.2238. Notice the ripple: changing only layer 2's corrupted value shifted the shared corrupted baseline, which changed every layer's AIE, not just layer 2's, and it flipped the ranking: layer 3 now has the larger AIE (0.2238 > 0.2071), where layer 2 was larger before. This is a real methodological hazard in causal tracing: the strength and shape of corruption used is itself a design choice, and a weaker or differently-shaped corruption can change which layer looks most important, so conclusions should always be checked against more than one corruption scheme.

A3. False. Zeroing sets the hidden state to a value, all zeros, that the network essentially never produces during normal operation, pushing the rest of the forward pass off-distribution in a way that can create misleading effects unrelated to the component's real function. Patching substitutes a value the network actually computed on a real input, the clean run, so the rest of the network downstream continues to operate on realistic activations; only the specific fact-relevant content at that one location has been swapped.

A4. Whole-layer activation patching can show that "layer 2's total output matters," but a layer's output is itself the sum of several attention heads and an MLP block, and that output is read by several different downstream components. Path patching can isolate, for example, whether head 5 in layer 2 matters specifically through its direct connection into the MLP in layer 3, or whether it matters through some other downstream path, answering questions about which specific wires in the circuit carry the effect, not just which layer contains it.

A5. The trace says the fact is stored and retrieved by an MLP block at a specific mid-layer, keyed on the subject token, so the weight matrix to edit is that MLP's down-projection at that layer, using a rank-one update of the kind ROME performs, treating the MLP as a key-value memory and overwriting the value associated with the "Rajasthan" key. A late-layer attention head showed up in the trace too, but its causal role was moving an already-computed fact from the subject position to the output position, it does not store the fact, it relays it. Editing it would at best disrupt how facts in general get routed to the output, a far blunter and more damaging intervention than the targeted MLP edit the trace actually points to.

A6. Because in this toy construction, corruption is applied only to the subject-token embedding before layer 1 runs, and layer 1's contribution was defined identically in both the clean and corrupted lists, (0.5, 0.5) in both, meaning layer 1 in this toy never actually used subject-specific information in the first place. Patching a value onto itself is a no-op by construction, so AIE = 0 exactly, with no rounding involved. In a real transformer this would not be exactly zero, early layers do encode some subject information immediately from the embedding and earliest attention layers, but Meng et al.'s empirical result is that this effect really is close to zero at the very earliest layers and only becomes large in the identified mid-layer band, which is why a real causal trace shows a genuine, sharp peak rather than a flat line.

Think About It

Think about this: How would you explain interpretability methods: understanding model internals to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where interpretability methods: understanding model internals is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting interpretability methods: understanding model internals to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind interpretability methods: understanding model internals, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.