A fraud-detection team at a UPI payment processor builds a small neural network to flag suspicious transactions in real time. Engineering constraints are tight: the model runs on a low-latency inference path, so the hidden layer is squeezed down to just two neurons before the final fraud score is computed. The team has identified six binary risk signals worth tracking per transaction: a foreign IP address, an odd transaction hour, a first-time payee, a domestic micro-transaction pattern typical of testing stolen credentials, rapid repeat attempts, and a detected VPN. Six meaningful signals, two neurons to carry them. When an analyst later tries to explain why the model flagged a specific transaction by inspecting the two hidden neuron activations directly, the numbers make no sense — neuron 1 reads 1.5, neuron 2 reads 0.87, and neither value corresponds cleanly to any single one of the six signals. The analyst has run into the central fact this chapter is about: when a network has fewer dimensions than the concepts it needs to represent, it does not simply drop concepts — it superimposes them, and the individual neuron stops being a meaningful unit of explanation.
This is a different problem from the one attribution methods like SHAP or LIME solve. Those methods ask "which input features pushed this specific output up or down" while treating the trained network as a fixed black box. What the fraud analyst needs here is a different question entirely: "what does this internal neuron actually represent, and can I trust reading it in isolation?" That question can only be answered by opening the network up and studying its internal geometry — the province of mechanistic interpretability. This chapter builds one specific, fully worked mechanism — the superposition hypothesis — from first principles, shows exactly how a real interpretability tool (the sparse autoencoder) recovers meaning from it, and then contrasts it with a second, independently important mechanistic result — the induction head circuit — to show how the field reasons about behavior that lives in *interactions* between components rather than in any single one.
The superposition hypothesis
Start from a design question the fraud team's network is silently answering: given six sparse features (each risk signal is present in only a small fraction of transactions, and multiple signals rarely fire on the same transaction) and only two hidden dimensions, what is the loss-minimizing way to encode them? Linear algebra says you cannot fit six linearly independent directions into a two-dimensional space — at most two vectors can be mutually orthogonal in ℝ². If the network insisted on giving each feature its own private, non-interfering direction, it could represent only two features and would have to ignore the other four entirely.
Gradient descent does not do that. Instead, as shown in Nelson Elhage et al., "Toy Models of Superposition" (Anthropic, 2022), a network trained to reconstruct sparse features through a narrow bottleneck learns to pack more feature directions into the space than its dimensionality allows, tolerating a controlled amount of cross-talk between them. The key word is sparse: if two features are rarely or never active on the same input, the interference between their directions costs the network nothing in expectation, because the offending cross-term only appears in the loss on inputs where both fire simultaneously — and if that never happens in training, gradient descent is free to place those two directions anywhere, including directly opposite each other, without paying a penalty. Features that sometimes do co-occur get pushed toward smaller mutual interference (closer to orthogonal), because that interference bites more often. The result is a specific, non-arbitrary geometric arrangement: for six roughly-equally-important, roughly-equally-sparse features compressed into two dimensions, the network's cheapest solution is to arrange the six feature directions as a regular hexagon around the origin, each 60° from its neighbors and 180° from its opposite.
A toy model you can compute by hand
Build the fraud team's bottleneck explicitly. Let x ∈ ℝ⁶ be the sparse input vector of the six risk signals (F1 = foreign IP, F2 = odd hour, F3 = new payee, F4 = domestic micro-transaction, F5 = rapid repeat, F6 = VPN detected), each coordinate either 0 or 1. Let W be a 2×6 encoder matrix whose six columns are unit vectors placed at 0°, 60°, 120°, 180°, 240°, and 300° — the hexagon the superposition hypothesis predicts. The hidden activation is h = Wx, and the network reconstructs an approximation of x as x̂ = ReLU(Wᵀh + b), using the *same* matrix transposed as a tied decoder (a standard simplification in these toy models) with a learned bias b that suppresses small interference terms before they leak into the output.
Because adjacent hexagon vectors are 60° apart, cos(60°) = 0.5; vectors two steps apart are 120° apart, cos(120°) = −0.5; and opposite vectors are 180° apart, cos(180°) = −1. These cosines are exactly the dot products between feature columns, since all columns are unit vectors — and Wᵀh, expanded per output coordinate, is just h projected onto each of those six directions in turn.
import numpy as np
# Six feature directions at 0, 60, 120, 180, 240, 300 degrees
# in a 2-dimensional hidden space (the hexagon packing).
angles = np.radians([0, 60, 120, 180, 240, 300])
W = np.vstack([np.cos(angles), np.sin(angles)]) # shape (2, 6)
b = -0.5 # learned bias that clips small interference terms to zero
def toy_model(x):
h = W @ x # encode: 6 sparse features -> 2 numbers
pre = W.T @ h + b # decode back toward 6 dimensions
return h, np.maximum(pre, 0) # ReLU
# round only for display: raw floats carry ~1e-16 rounding noise
# from cos/sin that would otherwise clutter the printed arrays
# Case 1: only F1 (foreign IP) fires
x1 = np.array([1, 0, 0, 0, 0, 0])
h1, xhat1 = toy_model(x1)
print(np.round(h1, 3)) # [1. 0.]
print(np.round(xhat1, 3)) # [0.5 0. 0. 0. 0. 0. ]
# Case 2: F1 and F2 fire together (60 degrees apart, adjacent)
x2 = np.array([1, 1, 0, 0, 0, 0])
h2, xhat2 = toy_model(x2)
print(np.round(h2, 3)) # [1.5 0.866]
print(np.round(xhat2, 3)) # [1. 1. 0. 0. 0. 0.]
# Case 3: F1 and F4 fire together (180 degrees apart, antipodal)
x3 = np.array([1, 0, 0, 1, 0, 0])
h3, xhat3 = toy_model(x3)
print(np.round(h3, 3)) # [0. 0.]
print(np.round(xhat3, 3)) # [0. 0. 0. 0. 0. 0.]
Trace Case 1 by hand to see where the printed numbers come from. Column 1 of W is (cos 0°, sin 0°) = (1, 0), so h1 = W·x1 is just that column: (1, 0). To decode, take the dot product of h1 with every column: column 1 gives 1·1 + 0·0 = 1; column 2 (60° away) gives cos 60° = 0.5; column 3 (120° away) gives cos 120° = −0.5; column 4 (180° away, F4) gives cos 180° = −1; column 5 gives −0.5; column 6 gives 0.5. Before the bias, the reconstruction is [1, 0.5, −0.5, −1, −0.5, 0.5]. Subtracting the bias 0.5 gives [0.5, 0, −1, −1.5, −1, 0], and ReLU clips every negative entry to zero: [0.5, 0, 0, 0, 0, 0]. The model correctly identifies that F1, and only F1, is active — the magnitude is attenuated to 0.5 rather than 1 (a real trained model would learn a slightly larger W norm to correct this scaling), but the *support* of the reconstruction, which single flag is on, is recovered exactly. The bias is doing real work: without it, the interference terms of ±0.5 would leak into F2 and F6's reconstructed values as false positives.
Case 2 shows the mechanism succeeding under load. F1 and F2 are 60° apart, so h2 = (1.5, 0.866). Projecting back: F1 and F2 each get a pre-bias value of 1.5 (verify: column 2 · h2 = 0.5×1.5 + 0.866×0.866 = 0.75 + 0.75 = 1.5), while F3 and F6 land exactly at 0, and F4 and F5 land at −1.5. After subtracting the bias and applying ReLU, the reconstruction is exactly [1, 1, 0, 0, 0, 0] — both active flags recovered cleanly, with zero false positives elsewhere. Superposition is working as designed: two features share a two-dimensional space and the network still reads out both correctly, because their shared 60° angle produces interference small enough for the bias and ReLU to absorb.
Case 3 shows the mechanism's designed-in failure mode. F1 and F4 are placed at exactly 180° — antipodal — because in the training data these two signals never co-occurred, so gradient descent paid no cost for maximizing their interference. But maximizing interference between two vectors means their sum, when both are active, is (1,0) + (−1,0) = (0,0): the hidden activation collapses to the origin. Every downstream reconstruction, positive or negative, is derived from a zero vector, so after bias and ReLU the entire output is [0, 0, 0, 0, 0, 0]. Not "F1 and F4, degraded" — nothing at all. The compression scheme is not a random compromise; it is a deliberate bet, made silently by gradient descent, that F1 and F4 will never need to be distinguished from "nothing happening" at the same time. If a new fraud pattern later combines a foreign IP with a domestic-looking micro-transaction — exactly the combination the model was never shown — the fraud model does not raise a weaker alarm. It raises no alarm, with full confidence, because from the model's point of view the input is indistinguishable from complete quiet.
Sparse autoencoders: undoing the compression
The fraud analyst's original complaint — "neuron 1 reads 1.5, and that number means nothing on its own" — is now explainable: neuron 1's raw activation is a superposition of contributions from multiple features projected onto one arbitrary coordinate axis of the hidden space, not a dedicated readout of any single concept. Reading meaning out of raw neurons or raw residual-stream coordinates in a real transformer runs into the identical problem, just at a scale where nobody can eyeball a hexagon: production language models have residual streams with thousands of dimensions carrying, by current evidence, many millions of interpretable concepts in superposition.
The tool the field converged on to reverse this is the sparse autoencoder (SAE), introduced for language-model interpretability in Trenton Bricken et al., "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" (Anthropic, 2023) and scaled to a production model in Adly Templeton et al., "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" (Anthropic, 2024). An SAE takes a layer's activation vector a and learns f = ReLU(W_enc·a + b_enc), where f is *much higher-dimensional* than a — an overcomplete dictionary, exactly mirroring the toy model but run in reverse and at far larger scale. It is trained to reconstruct a as â = W_dec·f + b_dec while minimizing an L1 penalty on f that pushes most entries of f to exactly zero for any given input. This is precisely the inverse problem the hexagon toy model poses: given a compressed, superposed representation, recover a larger set of *sparse*, ideally monosemantic directions that individually correspond to single human-interpretable concepts. The 2024 Claude 3 Sonnet study trained SAEs at several dictionary sizes and reported the largest, at roughly 34 million learned features, discovering individual directions that fired selectively and specifically — famously including one that activated on mentions and images of the Golden Gate Bridge regardless of language, and which, when artificially amplified, caused the model to describe itself as the bridge. The dictionary-learning approach works precisely because it exploits the same sparsity assumption the toy model relies on: real concepts are individually rare across any given input, so an overcomplete, L1-regularized decomposition can, in principle, separate a hexagon's worth of tangled directions back into six clean labeled axes — or a transformer's worth into millions.
Circuits: when the unit of meaning is an interaction, not a feature
Superposition and SAEs address *what a direction represents*. A second, complementary strand of mechanistic interpretability addresses *how information flows between components to produce a behavior* — a circuit. The clearest documented example is the induction head, identified in Catherine Olsson et al., "In-context Learning and Induction Heads" (Anthropic, 2022). Consider a transformer processing a sequence containing "... UPI txn ID 9821 flagged ... UPI txn ID 9821" — the model's job, on reaching the second "9821", is to predict what followed it the first time. Olsson et al. found this is implemented not by one attention head but by two heads composing across layers: a previous-token head in an earlier layer, whose job is only to write "the token at the previous position was X" into each position's residual stream; and an induction head in a later layer, whose query-key (QK) circuit searches backward for a position whose previous-token information matches the *current* token (prefix matching), and whose output-value (OV) circuit then copies whatever token followed that earlier match into the current position's output logits (the copying step). Neither head alone performs the behavior. Ablating only the later induction head and observing that in-context copying accuracy collapses tells you the head is *necessary*, but not *how* — you would still be missing that its query depends entirely on information the earlier previous-token head had to write first, a dependency only visible by tracing the two heads' composition together, typically via causal interventions such as activation patching, where an activation from one run is swapped into another to measure exactly which component's contribution changes the output. This is why real circuit analysis reports pairs and chains of components with their composition explicitly diagrammed, rather than ranking individual heads by importance in isolation — the same discipline the toy model teaches for features: the unit of interpretation is whatever the mechanism actually is, not whatever unit is easiest to point to.
Correcting a common misconception
Students meeting interpretability for the first time tend to import an intuition from early computer-vision folklore: the "grandmother neuron" idea that somewhere in the network there is one neuron that lights up if and only if the input is your grandmother — or, in the fraud case, one neuron that lights up if and only if the transaction has a foreign IP. The toy model shows exactly why this fails as a general assumption. When a network is forced to compress more sparse features than it has dimensions, gradient descent's cheapest solution is to spread each feature across *every* hidden coordinate at some angle, not to dedicate one coordinate per feature. Case 2 above makes this concrete: neuron h1's value of 1.5 is not "how much F2 fired" — it is the sum of F1's and F2's projections onto one arbitrary axis of a hexagon the network happened to orient a particular way during training, and would land on a completely different number if the same information were encoded in a bottleneck rotated by 30°. A neuron's raw activation is only ever interpretable relative to the specific basis the network settled into; interpretability work therefore searches for the *right basis* — via sparse dictionary learning, or via causal tracing of circuits — rather than assuming the network's own coordinate axes were ever aligned to human concepts in the first place. They were never optimized to be.
Superposition and circuit geometry
Active recall
Attempt each question before reading its answer.
Q1. In the toy model, why does placing F1 and F4 at 180° apart make sense given that they never co-occurred in training, and what does the network gain elsewhere in the hexagon by doing so?
Q2. Suppose the fraud team retrains the same 2-neuron model after a new scam pattern emerges in which F1 (foreign IP) and F4 (domestic micro-transaction) start co-occurring in 8% of fraud cases, while every other pairwise co-occurrence rate is unchanged. Trace the full effect: (a) what does the *old*, already-deployed model output on the new combined pattern, (b) what does retraining do to the hexagon's geometry, and (c) what happens to the other four features as a result?
Q3. A sparse autoencoder trained on a transformer's residual stream (a width Anthropic has not publicly disclosed, but on the order of thousands of dimensions, typical of production transformer hidden sizes) discovers on the order of 34 million distinct feature directions. Explain how this is possible without violating the fact that that thousands-wide space cannot hold 34 million linearly independent vectors, using the toy model as your explanation.
Q4. Why is an induction head described as a two-component circuit rather than a single interpretable unit, and what would ablating only the later head fail to reveal?
Q5. A classmate argues: "SHAP already tells me how much each feature contributed to a prediction, so I don't need sparse autoencoders or circuit analysis." What specifically can the toy model's F1–F4 blind spot reveal that SHAP, applied to the same trained model, cannot?
Answers
A1. Interference between two feature directions only costs the network anything on inputs where both features are simultaneously active; if F1 and F4 never co-occur in the training distribution, the cos(180°) = −1 penalty between them is never actually incurred, so gradient descent is free to place them at maximum interference for zero cost. That freedom is spent elsewhere: every *adjacent* pair in the hexagon, which does occasionally co-occur (F1 with F2, "foreign IP" with "odd hour," is a realistic joint pattern), is kept at only 60° apart, cos(60°) = 0.5 — a much smaller, ReLU-and-bias-absorbable interference. The hexagon is a budget allocation: near-zero separation is spent on pairs that are safe to conflate, tighter separation is reserved for pairs that actually need to be told apart.
A2. (a) Nothing changes about the deployed weights just because the world changed — feeding x=(1,0,0,1,0,0) through the old W still gives h=(0,0) and x̂=(0,0,0,0,0,0) exactly as derived in Case 3. The model reports zero risk on the new fraud pattern with full confidence; this is worse than a weak or uncertain signal, because there is no partial signal to notice at all. (b) If retrained on data where this combination now occurs 8% of the time, the loss picks up a real, non-negligible penalty for the F1–F4 cancellation, so gradient descent pulls F4's direction away from exactly 180° from F1 toward some smaller angle, trading a reduction in F1–F4 interference against interference somewhere else. (c) Because the hidden space is still fixed at two dimensions, F4 cannot move without disturbing its neighbors: F3 and F5 (formerly 120° and 240° from F1, i.e., adjacent to F4 in the hexagon) get squeezed to preserve their own separation from F4's new position, which in turn nudges F2 and F6. The angular budget around the full circle is conserved, so the average interference across *all six* features rises slightly — fixing one blind spot under a fixed-width bottleneck imposes a small tax on every other feature's fidelity, not just the pair being fixed.
A3. The 34 million SAE directions are not claimed to be linearly independent — exactly as in the hexagon, where six directions coexist in two dimensions by relying on sparsity rather than orthogonality. As long as almost every pair of the 34 million concepts is almost never simultaneously active on the same input, most pairwise interference terms are never actually incurred in practice, so an overcomplete, non-orthogonal set of directions can be packed into that same undisclosed, thousands-wide space the same way six directions were packed into two — just with vastly more directions relying on vastly higher-dimensional near-orthogonality and much sparser co-activation to make the compression viable.
A4. The induction behavior — copy whatever token followed the last occurrence of the current token — requires two facts to be available together: which token immediately preceded the current position, and which earlier position in the sequence had a matching predecessor. The previous-token head supplies the first fact by writing it into the residual stream one layer early; the induction head's query-key circuit consumes that written fact to perform prefix matching, and its output-value circuit copies the result. Ablating only the induction head shows that removing it breaks the behavior, but a researcher stopping there could wrongly conclude the entire mechanism lives inside that one head. What that ablation cannot reveal is that removing the *earlier* previous-token head — leaving the induction head itself completely untouched — breaks the identical behavior, because the induction head's query never had the information it needed in the first place. The behavior is a property of the composition, not of either head alone.
A5. SHAP explains the fixed, trained model's output for one specific input by attributing the output to input features — it never opens up or decomposes the hidden representation, so it has no way to represent a pattern that has not yet appeared in any input it was asked to explain. On a transaction combining F1 and F4, SHAP applied *before* that combination ever occurs in the data has nothing to attribute, because no such transaction exists to attribute anything to — the blind spot is a property of the model's internal geometry, present and discoverable the moment training finishes, regardless of whether any real transaction has triggered it yet. Only a mechanistic method that inspects the weights directly — recovering the hexagon geometry, or in a real model, an SAE's feature directions and their pairwise similarities — can find that F1 and F4 sit at cos θ = −1 and flag the latent failure mode before it costs the fraud team a real, undetected loss.
Think About It
Think about this: How would you explain interpretability: mechanistic understanding of neural networks to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind interpretability: mechanistic understanding of neural networks, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.