Continual Learning and Catastrophic Forgetting

Picture a fraud-detection model running inside a UPI payments backend. In January, the team trains a neural network on labelled transaction data to catch SIM-swap fraud — a criminal gets a victim's SIM re-issued, then drains the linked account through one large, sudden transfer from a freshly registered device. The model learns this pattern well: 94% recall on held-out SIM-swap cases. Then October arrives, and with it Diwali shopping season and a new fraud wave: QR-code merchant scams, where a fake "refund" QR code tricks a user into authorising many small outgoing payments. The engineering team does the obvious, cheap thing — they take the already-trained network and keep training it, feeding it only the new QR-scam examples, because re-running the full January dataset again feels wasteful. Three weeks later, someone checks SIM-swap recall on the current model. It has fallen from 94% to 21%. Nothing about SIM-swap fraud changed in the real world — the fraudsters using that technique are still doing exactly what they did in January. The model simply forgot how to catch them, because training on Task B silently overwrote what it had learned for Task A. This is catastrophic forgetting, and understanding exactly why gradient descent does this — and what can be done about it — is the subject of this chapter.

Why forgetting is not a bug, it is what gradient descent is built to do

A neural network's knowledge lives entirely inside its weight matrices — there is no separate "memory bank" tagged by task. When you train the SIM-swap detector, backpropagation nudges every weight in the direction that reduces the SIM-swap loss. Those weights settle into a configuration — call it θ*_A — that happens to work well for Task A. Nothing in that configuration is labelled "important, keep this." It is just a point in a very high-dimensional space where the Task-A loss happens to be low.

Now switch to training on Task B (QR-scam) data using the same weights as a starting point. The loss function the optimiser actually sees at this stage is only the Task-B loss. Task A's training examples are gone from the computation graph entirely — the optimiser has no term in its objective that says "and don't move too far from where Task A liked things." Gradient descent, doing exactly its job, walks θ downhill on the Task-B loss surface, and that path drags θ away from θ*_A toward a new point θ*_B that is good for Task B and, generally, bad for Task A. The psychologists Michael McCloskey and Neal Cohen gave this phenomenon its name in 1989, studying exactly this failure mode in early connectionist networks — long before deep learning existed — because it is a structural property of any model that represents multiple pieces of knowledge in a single shared, distributed set of weights. The formal name for the underlying tension is the stability–plasticity dilemma: a network needs plasticity (the ability for weights to change) to learn anything new, but that same plasticity is exactly what lets new learning erase old learning. Too much stability and the model can never learn Task B; too much plasticity and it forgets Task A. Catastrophic forgetting is what happens when a system has no mechanism at all for balancing the two — it defaults to pure plasticity.

A fully worked example: watching a weight get overwritten

To see this mechanically rather than just verbally, strip the problem down to a single trainable weight — as if the fraud detector had only one input feature (say, "transfer amount relative to account average") and one weight w multiplying it, with no bias term. Suppose Task A's training reduces to a single representative point (x=1, y=2), so the optimal weight for Task A is exactly w = 2 (since prediction = w·x = w, and we need w = 2 to hit y = 2 with zero loss). The model has already converged there: w = 2.0, Task-A loss = 0.

Task B's data reduces to a single point (x=1, y=5), so its optimum is w = 5. Using squared-error loss L(w) = 0.5·(w·x − y)², the gradient with respect to w at x=1 is dL/dw = (w − y). Running plain gradient descent on Task B only, starting from the Task-A-optimal weight, with learning rate η = 0.5:

def loss_A(w):
    return 0.5 * (w - 2) ** 2

def loss_B(w):
    return 0.5 * (w - 5) ** 2

w = 2.0          # already converged on Task A; loss_A(w) = 0 here
lr = 0.5

for step in range(1, 4):
    grad = (w - 5)          # dLoss_B/dw at the single point (x=1, y=5)
    w = w - lr * grad
    print(step, round(w, 4), round(loss_A(w), 4), round(loss_B(w), 4))

This prints, exactly:

1 3.5 1.125 1.125
2 4.25 2.5312 0.2812
3 4.625 3.4453 0.0703

Read the columns as (step, weight, Task-A loss, Task-B loss). In three gradient steps that only ever look at Task-B data, the weight has moved from 2.0 to 4.625, closing in on Task B's optimum of 5.0 — Task-B loss has dropped from 4.5 down to 0.07. But Task-A loss, which started at exactly 0, has climbed to 3.45. Nobody deleted anything, no weight was reset, no data was corrupted. Three ordinary gradient steps, each one individually correct for the objective it was given, are entirely sufficient to wreck a previously perfect solution to a different problem. This is catastrophic forgetting captured in miniature. A learning rate of η = 1.0 instead of 0.5 would make it even more dramatic — since the second derivative of this loss with respect to w is exactly 1, a single step of size η = 1 lands precisely on w = 5 in one move (verify: grad = 2 − 5 = −3, so w becomes 2 − 1·(−3) = 5), forgetting Task A completely after a single update.

Real deep networks make this worse, not better. A single scalar weight has nowhere to hide, but that also means the damage is easy to see. A convolutional network with fifty million shared parameters spreads Task-A's knowledge across features that are entangled with Task-B's features in ways no one designed on purpose — a small nudge to an early convolution layer, driven purely by Task-B gradients, can ripple through every downstream layer that depended on that layer's outputs, corrupting representations for inputs the new gradient never even saw. The one-weight example isolates the mechanism; production-scale forgetting is the same mechanism multiplied across millions of entangled coordinates simultaneously.

The mechanism as a picture

The blue curve is how wrong the model is on Task A for any weight value; the red curve is how wrong it is on Task B. They share the same weight axis because it is the same network. Plain fine-tuning only ever walks downhill on the curve it is currently being trained against — the green path slides down the red curve toward θ*_B, and because the two curves' minima sit at different weight values, sliding down one necessarily climbs the other. The yellow point marks where a regularised version of training settles instead, which the next section derives exactly.

Elastic Weight Consolidation: making forgetting expensive

If forgetting happens because the Task-B loss is the only thing the optimiser is asked to minimise, the fix is to put a second term into the objective that penalises moving away from θ*_A — but only in the directions that actually mattered for Task A. Elastic Weight Consolidation (EWC), proposed by Kirkpatrick et al. at DeepMind in 2017, does exactly this using the diagonal of the Fisher information matrix as a per-weight "importance" score, F_i. Intuitively, F_i is large for a weight if Task A's loss curved sharply around its optimal value in that direction (small changes badly hurt Task A) and small if the loss was flat there (Task A didn't care much what that weight did). The augmented objective for training on Task B becomes:

L_total(θ) = L_B(θ) + (λ/2) · Σ_i F_i · (θ_i − θ*_A,i)²

The second term is a spring anchoring each weight back toward its Task-A value, with stiffness proportional to how important that weight was for Task A. λ is a single dial for the whole network: it controls how much stability you are buying at the cost of plasticity.

Continue the one-weight example. Since there is only one weight, its Fisher information is just the curvature of Task A's loss at its optimum: d²L_A/dw² = x² = 1, so F = 1. Pick λ = 2. The gradient of the total loss is now (w − 5) + λF(w − θ*A) = (w − 5) + 2(w − 2):

lam, F, wA = 2.0, 1.0, 2.0
w, lr = 2.0, 0.5
for step in range(1, 6):
    grad = (w - 5) + lam * F * (w - wA)
    w = w - lr * grad
    print(step, round(w, 4), round(loss_A(w), 4), round(loss_B(w), 4))

1 3.5 1.125 1.125
2 2.75 0.2812 2.5312
3 3.125 0.6328 1.7578
4 2.9375 0.4395 2.127
5 3.0312 0.5317 1.938

Compare this to the unregularised run: instead of marching steadily toward w = 5, the weight now oscillates and settles down near w = 3 — pulled toward Task B, but held back by the Task-A anchor. You can find that equilibrium exactly, without iterating, by setting the total gradient to zero: (w − 5) + 2(w − 2) = 0 gives 3w = 9, so w* = 3, which matches what the code converges to. In general, for this one-weight setup, w* = (5 + 2λF) / (1 + λF). Setting λ = 0 recovers w* = 5 — no protection, full forgetting, exactly the plain fine-tuning case. Sending λ → ∞ forces w* → 2 — the weight is frozen at Task A's optimum and Task B is never learned at all. EWC's λ is a literal, tunable position on the stability–plasticity dial, and the equilibrium formula shows precisely where that position lands.

Other ways to fight forgetting

EWC belongs to a family called regularisation-based methods. Two other families are used just as widely in practice. Rehearsal (replay) methods keep a small buffer of old-task examples — for the fraud model, a few thousand representative SIM-swap transactions — and mix them back into every training batch while learning the QR-scam task, so the loss the optimiser actually sees always includes both distributions and gradient descent has no incentive to move away from θ*_A in directions Task A needs. Architectural (parameter-isolation) methods instead freeze the weights that already encode old knowledge and add new, small, task-specific parameters for the new task — a progressive network adds a fresh column of layers per task; a more modern equivalent is bolting a small low-rank adapter onto a frozen backbone. Nothing is ever overwritten because nothing shared is ever trained again. A fourth family, distillation-based methods such as Learning without Forgetting, use the old model's own predictions on new-task inputs as extra soft targets, so the new model is nudged to keep agreeing with its past self even where no old labels exist.

This is not a niche problem confined to fraud detection or robotics. Large language models are pretrained once on a huge, broad corpus and then fine-tuned (instruction-tuning, RLHF) on a much smaller, narrower dataset — and that second stage can measurably erode capabilities the base model had from pretraining, for exactly the reason worked out above: the fine-tuning loss only ever looks at the fine-tuning data. Production LLM training pipelines routinely mix a slice of the original pretraining data back into fine-tuning batches — rehearsal, applied at enormous scale — precisely to hold on to what the base model already knew.

The misconception to correct

A natural assumption, especially if you have only ever trained models on one fixed, shuffled dataset, is: "more training data over time should only make the model smarter — like a person doesn't unlearn arithmetic when they later learn calculus." This is false for a plain neural network, and the worked example proves exactly why: gradient descent has no built-in notion of "cumulative" knowledge. It only ever computes a gradient with respect to whatever loss it is currently given. If old-task data is not part of that loss, the optimiser has zero information telling it to preserve old-task behaviour, and it will happily walk straight through a previously perfect solution on its way to a new one. Humans resist this partly because biological memory consolidation is not simple backpropagation — but even human skills genuinely do decay with disuse, so the analogy is weaker than it feels. It is also worth separating catastrophic forgetting from a different, easily confused failure: concept drift. If SIM-swap fraudsters change their actual behaviour — different transfer amounts, different timing — the old model degrades because the real-world task itself moved; that model would have failed on the new pattern even if it had never been retrained at all. Catastrophic forgetting is the opposite: the old task's data distribution is completely unchanged, and the model still fails on it, purely because of what happened during a later training run. You can tell them apart empirically — evaluate the checkpoint saved before the second training run against current data from the old task. If that old checkpoint still performs well, the world didn't change; the training did, and that is forgetting.

Active recall

Attempt each question before reading its answer.

1. In the no-EWC toy example, if the learning rate were η = 1.0 instead of 0.5, what would the weight be after exactly one gradient step of Task-B training, starting from w = 2.0? What does this tell you about the relationship between learning rate and forgetting speed?
2. Using the EWC equilibrium formula w* = (5 + 2λF)/(1 + λF) with F = 1, find the value of λ that makes the equilibrium Task-A loss exactly 0.5.
3. In one or two sentences, explain why catastrophic forgetting is a fundamentally different failure from overfitting.
4. Why doesn't simply making the network bigger (more parameters) automatically solve catastrophic forgetting, even though a bigger network has "more room" to store two tasks' worth of knowledge?
5. A warehouse robot's vision model is first trained on well-lit daytime images, then continues training only on dim night-warehouse images, reusing the same weights with no replay buffer. Predict what happens to daytime picking accuracy, and name two concrete fixes.
6. A UPI fraud model's SIM-swap accuracy drops over three months. Scenario A: fraudsters change their tactics, so SIM-swap transactions now look different. Scenario B: fraud tactics are unchanged, but the model was retrained on new QR-scam data with no replay. Which scenario is concept drift and which is catastrophic forgetting, and how would you distinguish them using only data you already have?

Answers.

1. Gradient at w = 2 is (2 − 5) = −3, so w_new = 2 − 1.0 × (−3) = 5. With η = 1.0 the model lands exactly on Task B's optimum after a single step (this learning rate equals 1/second-derivative for this quadratic loss, so one step is a full Newton step) — and Task A is forgotten completely, immediately, rather than gradually. A larger learning rate makes forgetting faster and more total, not just faster convergence on the new task.

2. Task-A loss at equilibrium is 0.5(w* − 2)² . Since w* − 2 = 3/(1 + λ), loss_A = 4.5/(1 + λ)². Setting this to 0.5 gives (1 + λ)² = 9, so 1 + λ = 3 and λ = 2 — exactly the value used in the worked example, which is why its equilibrium loss_A came out to 0.5317 (converging toward the exact value 0.5).

3. Overfitting is a generalisation gap within a single task: the model memorises noise in its training set and fails on unseen data drawn from the same distribution. Catastrophic forgetting is a cross-task interference problem: the model fails on a different, previously learned task's data because a later training run on new data overwrote shared parameters — the old task's distribution is untouched and unseen during the failure.

4. Extra capacity only helps if training actually routes different tasks into different parts of that capacity. Plain gradient descent has no objective term that rewards leaving any subset of weights untouched — it updates whichever combination of weights reduces the current loss fastest, and because representations in a trained network are typically distributed and entangled across many neurons rather than cleanly separated by task, that update touches parameters Task A depended on too, regardless of how much spare capacity exists elsewhere in the network.

5. Daytime accuracy will very likely collapse for the same reason as the SIM-swap example: night-only gradients pull the shared convolutional weights toward night-optimised features, and nothing in the night-only loss protects daytime performance. Two fixes: (a) rehearsal — keep a buffer of daytime images and interleave them into every night-training batch, so both distributions are present in the loss; (b) architectural isolation — freeze the existing backbone and train a small night-specific adapter module on top of it, leaving the daytime-tuned weights untouched.

6. Scenario A is concept drift: the real-world SIM-swap pattern itself changed, so even a model that was never retrained again would start failing on new SIM-swap cases, because those cases genuinely look different now. Scenario B is catastrophic forgetting: SIM-swap fraud is identical to before, but training on QR-scam data overwrote the weights that used to catch it. To tell them apart, evaluate the checkpoint saved before the QR-scam retraining against today's SIM-swap transactions. If that old checkpoint still scores near 94%, the SIM-swap pattern hasn't moved — the failure is entirely due to the later training run, which is forgetting, not drift.

Think About It

Think about this: How would you explain continual learning and catastrophic forgetting to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind continual learning and catastrophic forgetting, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.