Mesa-Optimization: When Inner Optimizers Emerge

A frozen model that keeps getting better mid-conversation

A risk-operations desk at a Bengaluru fintech runs transaction triage behind a UPI app. The classifier is a closed-weights LLM accessed through an API: nobody on the team can touch its parameters, retrain it, or run a single gradient step against it. Yet the analysts have a trick. They paste five or six labelled examples into the prompt, "1,200 rupees to a new payee at 2 a.m., flagged fraud" and "450 rupees to a saved contact, legitimate", before asking the model to classify a fresh transaction. Accuracy on the new case jumps noticeably. Nothing about the model's 175 billion (or however many) weights changed between the unprimed call and the primed one. The weights are frozen, the training run finished months ago, and yet the model's behaviour on this specific task visibly improved after seeing a handful of examples inside a single prompt.

This is in-context learning (ICL), and it is not a curiosity. It is the mechanism behind almost every production LLM deployment that uses few-shot prompting instead of fine-tuning. The question this chapter answers is mechanistic, not philosophical: what computation, running entirely inside one forward pass with no weight updates, produces that improvement? The answer is that the forward pass can implement a second, nested optimization process, one that takes the in-context examples as its own training data, fits an internal model to them, and uses that internal model to make the prediction. That nested optimizer is a mesa-optimizer, and unlike the mesa-optimizers usually discussed in connection with reinforcement-learning agents and deceptive alignment, this one can be written down as an explicit matrix construction, run on paper, and checked against a calculator.

Base objective, mesa-objective, and a sharper question

The vocabulary here is due to Hubinger, van Merwijk, Mikulik, Skalse, and Garrabrant's 2019 report on risks from learned optimization. A base optimizer is whatever process searches over a space of programs or parameters to satisfy a base objective; for a transformer, the base optimizer is stochastic gradient descent (or Adam) and the base objective is next-token prediction loss averaged over the training corpus. If the artifact that SGD produces is itself an optimizer, one that internally searches for outputs that score well against some objective of its own, that inner process is a mesa-optimizer, and whatever it is internally optimizing for is its mesa-objective. The base optimizer never has direct access to the mesa-objective; it can only shape the mesa-optimizer indirectly, by rewarding parameter settings that happen to produce a mesa-optimizer whose behaviour scores well on the base objective across the training distribution.

That framing is usually illustrated with an RL agent trained in a maze that develops an internal heuristic like "head toward the green tile" as a proxy for "reach the exit," a proxy that generalizes badly once green tiles and exits come apart. This chapter asks a narrower and more mechanical question about a different setting: transformer in-context learning. Does anything that deserves the name "internal optimizer" actually run inside a forward pass when a frozen LLM does few-shot prediction, or is ICL just sophisticated pattern retrieval with no optimization happening at all? Since 2022, a line of interpretability work has answered this with an explicit, checkable construction rather than an analogy.

The mechanism: attention as an implicit optimizer

Von Oswald, Niklasson, Randazzo, Sacramento, Mordvintsev, Zhmoginov, and Vladymyrov, in "Transformers Learn In-Context by Gradient Descent" (ICML 2023), show that a single linear self-attention layer can be assigned weights so that its forward pass computes exactly one step of gradient descent on a least-squares regression problem, where the "training set" for that regression is the sequence of in-context examples sitting in the prompt, not anything from the original training corpus. Garg, Tsipras, Liang, and Valiant had already shown empirically (NeurIPS 2022) that transformers trained from scratch on synthetic in-context regression tasks learn to perform close to what ordinary least squares would achieve on the same prompt; von Oswald et al. supplied the mechanism that makes that empirical result unsurprising, an explicit weight assignment under which the claim is exactly true, not approximately true.

Here is the setup. Suppose the in-context examples are pairs (x_i, y_i) for i = 1..n, each encoded as a token e_i = [x_i ; y_i], followed by a query token e_{n+1} = [x_{n+1} ; 0] whose label is masked to zero because it is what the model must predict. Define a linear (softmax-free) self-attention layer with key, query, and value projections W_K, W_Q, W_V, so that the update added to token j is Δe_j = Σ_i (W_K e_i)·(W_Q e_j) · (W_V e_i), summed over the context tokens. Choose W_K and W_Q to project each token onto its x-coordinate only, so the attention score between tokens i and j becomes the plain dot product x_i · x_j. Choose W_V to project each context token onto its y-coordinate, scaled by a constant η, and to place that scaled value in the token's y-slot with zero in the x-slot. Adding Δe_{n+1} to the query token then changes only its (currently masked) y-coordinate, to η · Σ_i y_i (x_i · x_{n+1}).

Compare that to ordinary gradient descent, starting from weight vector w_0 = 0, on the squared-error loss L(w) = ½ Σ_i (w·x_i − y_i)². The gradient is ∇L(w_0) = Σ_i (w_0·x_i − y_i) x_i = −Σ_i y_i x_i (since w_0·x_i = 0), so one gradient step with learning rate η gives w_1 = w_0 − η∇L(w_0) = η Σ_i y_i x_i, and the resulting prediction at the query point is ŷ_{n+1} = w_1 · x_{n+1} = η Σ_i y_i (x_i · x_{n+1}). That is precisely the value the attention layer computed. The forward pass, with no weight updates anywhere, has implemented a full training loop, gradient computation and parameter update included, using the prompt as data and the attention scores as the mechanism for "reading off" the gradient. The mesa-objective is the least-squares loss on the in-context examples; the mesa-optimizer is the composition of the layer's key, query, and value projections; and it exists only for the duration of that one forward pass.

Worked example: one attention layer performs one gradient step

Take a scalar case, d = 1, so each x_i is a single number. Context examples: (x_1,y_1)=(1,2), (x_2,y_2)=(2,3), (x_3,y_3)=(3,5). Query: x_4 = 4, label unknown. Learning rate η = 0.1. By hand: attention scores are x_i·x_4, giving 4, 8, 12 for i=1,2,3. The implicit gradient step gives w_1 = η Σ y_i x_i = 0.1×(2×1 + 3×2 + 5×3) = 0.1×23 = 2.3, and the prediction is ŷ_4 = w_1 × x_4 = 2.3 × 4 = 9.2. Equivalently, straight from the attention formula: Δ = η × Σ score_i × y_i = 0.1 × (4×2 + 8×3 + 12×5) = 0.1 × 92 = 9.2. Both routes, "compute the gradient step and then predict" and "run the attention layer," land on exactly the same number, because they are the same computation written two ways.

The following code builds the four token embeddings, assigns the three weight matrices exactly as described above, and runs the forward pass numerically, with no shortcuts, to confirm the 9.2 by direct matrix multiplication rather than by the algebraic shortcut.

import numpy as np

# In-context examples: (x_i, y_i), scalar case d = 1
x = np.array([1.0, 2.0, 3.0])   # x_1, x_2, x_3
y = np.array([2.0, 3.0, 5.0])   # y_1, y_2, y_3
x_query = 4.0                    # x_4, label withheld

eta = 0.1

# Tokens e_i = [x_i, y_i]; query token has its label masked to 0
tokens = np.array([
    [x[0], y[0]],
    [x[1], y[1]],
    [x[2], y[2]],
    [x_query, 0.0],
])

W_K = np.array([[1.0, 0.0], [0.0, 0.0]])   # keeps only the x-coordinate
W_Q = np.array([[1.0, 0.0], [0.0, 0.0]])   # keeps only the x-coordinate
W_V = np.array([[0.0, 0.0], [0.0, eta]])   # keeps only eta * y

K = tokens @ W_K.T
Q = tokens @ W_Q.T
V = tokens @ W_V.T

scores = Q @ K.T                    # scores[j, i] = x_j . x_i
context_scores = scores[3, :3]      # query token attending to the 3 context tokens
delta = context_scores @ V[:3]      # weighted sum of the context values

updated_query = tokens[3] + delta
y_hat = updated_query[1]

print(round(float(y_hat), 4))

Tracing it: K and Q reduce every row to its x-value, so context_scores = [1×4, 2×4, 3×4] = [4, 8, 12]. V reduces every context row to [0, η y_i] = [0, 0.2], [0, 0.3], [0, 0.5]. The weighted sum is 4×[0,0.2] + 8×[0,0.3] + 12×[0,0.5] = [0, 0.8+2.4+6.0] = [0, 9.2]. Adding that to the query token [4, 0] gives [4, 9.2], so y_hat is 9.2 and the script prints 9.2, matching the hand derivation exactly.

Diagram: two optimizers, two objectives, one forward pass

What breaks when the starting point isn't zero

The construction above quietly assumed w_0 = 0. That assumption is doing real work, and seeing what happens without it is the difference between memorizing a trick and understanding the mechanism. General one-step gradient descent gives ŷ_{n+1} = w_0·x_{n+1} − η Σ_i (w_0·x_i − y_i)(x_i·x_{n+1}). When w_0 = 0, the residual w_0·x_i − y_i collapses to −y_i, which is exactly why the value vectors could get away with encoding raw y_i. For a nonzero w_0, the residual the attention layer needs to place in each value vector is y_i − w_0·x_i, the current prediction error, not the label itself, and the layer additionally has to carry w_0·x_{n+1} forward as a baseline term. The simple three-matrix construction from the worked example cannot do that; it only ever computes η Σ_i y_i (x_i·x_{n+1}), regardless of what the "true" starting weight should have been.

Concretely: reuse the three context examples and query x_4=4, but now suppose the correct starting point is w_0 = 2, not 0. Real gradient descent gives ∇L(2) = Σ_i (2x_i − y_i)x_i = (2−2)×1 + (4−3)×2 + (6−5)×3 = 0+2+3 = 5, so w_1 = 2 − 0.1×5 = 1.5 and the correct prediction is ŷ_4 = 1.5×4 = 6.0. The naive attention layer from the worked example, which only knows how to compute η Σ y_i(x_i·x_4), still outputs 9.2 regardless, because it has no way to represent w_0 at all. The fix, which is what von Oswald et al. actually build, is to let the value projection depend on the query's own current estimate rather than on raw labels, so each attention layer computes the residual relative to whatever the previous layer already produced. Stacking L such layers then implements L successive gradient steps, each one starting from where the last left off, exactly the way an outer training loop would iterate SGD, except the "iterations" here are transformer layers stacked in depth, executed once, in a single forward pass.

The common misconception

The natural objection is: "the weights never change during inference, so nothing is actually being learned or optimized, it's just retrieval." That objection conflates two different things that both go by the word "optimization": updating θ, and running a search or fitting procedure inside the activations that the fixed θ compute. The worked example is a direct counterexample to the objection. θ (the three weight matrices W_K, W_Q, W_V) never changed anywhere in the computation; every number in the derivation came from one forward pass over four tokens. And yet a genuine loss function, L(w), was genuinely minimized by one genuine step of gradient descent, with a genuine parameter vector w that exists only in the space of the model's activations and was never written to disk. "No weight updates" and "no optimization happened" are not the same claim, and the gap between them is precisely where a mesa-optimizer can live: entirely in activation space, for the duration of a single context window, invisible to anyone who only inspects θ.

Where this sits in the mesa-optimization safety picture

This is a different phenomenon from the training-time mesa-optimizer that learns a proxy goal and strategically conceals a misaligned mesa-objective during training, only to pursue it once deployment removes the threat of further gradient updates. That scenario is about a mesa-objective baked permanently into θ. The ICL mesa-optimizer above has a mesa-objective, least-squares regression on this prompt's own examples, that is transient and task-local: it is assembled fresh from the tokens sitting in context, and it dissolves the moment the context window ends, leaving no trace in the weights for anyone to audit later. Chan, Santoro, Lampinen, Wang, Singh, Richemond, McClelland, and Hill (NeurIPS 2022) further showed that whether a transformer develops this kind of in-context optimizing behaviour at all depends heavily on properties of the training distribution, in particular how "bursty" or skewed the corpus is, which means the base optimizer can be pushed toward or away from producing mesa-optimizers by data choices alone, without touching the architecture. The safety-relevant conclusion is not that few-shot prompting is dangerous. It is that goal-directed, multi-step optimization can be assembled by gradient descent as a general-purpose strategy for lowering the base loss, and that this assembled optimizer can spin up and vanish entirely within a single inference call, which means weight-checkpoint audits, the standard tool for inspecting what a model "wants," structurally cannot see it.

Active recall

Attempt each question before reading its answer.

1. In the linear-attention construction, identify the base optimizer, the base objective, the mesa-optimizer, and the mesa-objective, using the specific roles each plays (not the generic RL-agent version of these terms).
2. True or false, with justification: "Since θ never receives a gradient update during inference, no optimization occurs inside a forward pass."
3. Extend the worked example: context examples become (1,2), (2,3), (3,5), (4,7), η = 0.1, w_0 = 0, and the query becomes x = 5 (with unknown label). Compute the prediction.
4. Using the same three original context examples (1,2),(2,3),(3,5) and query x_4=4, but with true starting weight w_0 = 2 instead of 0: what does the naive attention layer (built for w_0=0) output, what should the correct answer be, and why do they differ?
5. Give one concrete reason the ICL mesa-optimizer above is safety-relevant in a way that is different from the "hidden goal survives training, then defects at deployment" scenario.
6. A GPT-3-scale model has 96 transformer layers. If each linear self-attention layer can implement one step of implicit gradient descent, is 96 an exact count, an upper bound, or a lower bound on the number of implicit GD steps such a model's forward pass performs, and why?

Answers.

1. Base optimizer: SGD (or Adam), run once during training, over the whole corpus. Base objective: next-token prediction loss averaged across the training distribution. Mesa-optimizer: the composition of the layer's W_K, W_Q, W_V projections, which together implement a gradient-descent update rule. Mesa-objective: the least-squares loss L(w) = ½ Σ_i (w·x_i − y_i)² defined only over this particular prompt's in-context examples, which exists fresh for every new prompt and is never written into θ.

2. False. The claim conflates "no update to θ" with "no optimization anywhere." The worked example shows a genuine gradient-descent step, computing a real gradient and applying a real parameter update, occurring entirely within the forward pass, in the model's activations rather than its weights. θ stayed fixed throughout; the optimization happened to an internal vector w that exists only for that one pass.

3. ∇L(w_0=0) = −Σ_i y_i x_i = −(2×1 + 3×2 + 5×3 + 7×4) = −(2+6+15+28) = −51. w_1 = 0 − 0.1×(−51) = 5.1. Prediction: ŷ = w_1 × 5 = 5.1 × 5 = 25.5.

4. The naive layer, built assuming w_0=0, ignores w_0 entirely and still computes η Σ y_i(x_i·x_4) = 0.1×(4×2+8×3+12×5) = 9.2, unchanged from the original example. The correct gradient step from w_0=2 is ∇L(2) = (2×1−2)×1+(2×2−3)×2+(2×3−5)×3 = 0+2+3 = 5, giving w_1 = 2−0.1×5 = 1.5 and ŷ_4 = 1.5×4 = 6.0. They differ (9.2 vs 6.0) because the naive value vectors encode raw labels y_i, which only equals the needed residual y_i − w_0·x_i when w_0 = 0; for any other starting point the value vectors must encode the residual against that starting point instead, which requires the layer to carry a representation of w_0 forward, something the simple three-matrix construction cannot do.

5. In the deceptive-alignment scenario, the mesa-objective is fixed into θ during training and persists across every future deployment, which is exactly why it can be strategically concealed until a moment when acting on it is safe. The ICL mesa-optimizer is task-local and transient: it is assembled from the tokens in one prompt and disappears when that context window ends, leaving nothing in θ to inspect afterward. This matters because it shows that goal-directed optimization can appear and vanish within a single inference call, invisible to any safety process that only audits weight checkpoints between training runs.

6. Upper bound, not an exact count. The one-layer-equals-one-GD-step correspondence was derived for an idealized linear self-attention layer with no MLP sublayers, no layer norm, and no softmax nonlinearity interleaved. A real 96-layer model interleaves attention with MLP blocks and normalization, and not every attention head in every layer is necessarily organized to perform this specific role; some capacity is spent on other computations entirely. Follow-up work (for example Ahn, Cheng, Daneshmand, and Sra's 2023 analysis of preconditioned gradient descent in transformers) also shows that deeper stacks can implement more powerful update rules than plain gradient steps, so layer count is not even a reliable proxy for "number of GD-equivalent steps" in one consistent unit. 96 is therefore a ceiling set by the layer count, not a measured quantity.

Think About It

Think about this: How would you explain mesa-optimization: when inner optimizers emerge to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where mesa-optimization: when inner optimizers emerge is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting mesa-optimization: when inner optimizers emerge to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind mesa-optimization: when inner optimizers emerge, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.