LSTMs and GRUs: Solving the Vanishing Gradient Problem

Picture a fraud model running on a UPI processor's transaction stream for a single account. Most days it sees the ordinary rhythm of a salaried customer: a ₹500 transfer to a friend, a ₹200 mobile recharge, a ₹3,000 rent split. Then, 300 transactions into the account's history, the pattern that matters happened — a small round-trip of transfers between three accounts that, on their own, look innocent, but that later feed a mule network laundering stolen funds. The model reading transaction #340 needs to connect it back to transaction #40. That is exactly what a recurrent neural network is supposed to do: carry a hidden state forward so that early information can still influence a much later prediction. But if you build that network the naive way, it cannot do this — not because it lacks capacity, but because the training signal that would teach it to remember transaction #40 is mathematically forced to zero before it ever gets there. This chapter derives why that happens, and shows exactly how the LSTM and the GRU — two gated recurrent architectures from the late 1990s and 2014 respectively — fix it, with a numeric trace you can follow by hand.

From Vanilla RNNs to the Vanishing Gradient

Recall the vanilla recurrent update from the previous chapter's sequence models: at each timestep the network combines the new input with its previous hidden state,

h_t = tanh(W_hh · h_{t-1} + W_xh · x_t + b)

Training this network with backpropagation through time (BPTT) means computing, for a loss L measured at the final timestep T, the gradient ∂L/∂h_1 — how much the very first hidden state should change to reduce the final error. By the chain rule, this gradient is a product of Jacobians, one factor per timestep the signal has to cross:

∂L/∂h_1 = ∂L/∂h_T · ∏_{t=2}^{T} ∂h_t/∂h_{t-1}
        = ∂L/∂h_T · ∏_{t=2}^{T} diag(tanh′(z_t)) · W_hh

where z_t is the pre-activation at step t. Two things make this product shrink. First, tanh′(z) = 1 − tanh²(z) has a maximum value of exactly 1, reached only when z = 0 — the moment the neuron is doing nothing interesting. Away from zero it falls fast: tanh′(1) ≈ 0.42, tanh′(2) ≈ 0.07. In a trained network, most units operate away from z = 0 most of the time (that is what makes them useful nonlinear feature detectors), so a typical per-step tanh′ is well under 0.5. Second, W_hh is a matrix whose eigenvalues are, for a stable network, generally kept near or below 1 (larger eigenvalues cause the complementary failure, exploding gradients, controlled in practice by gradient clipping — a fix for the opposite problem, not this one).

Multiply a sub-1 factor by itself across many timesteps and the product decays exponentially in the number of steps. Take a deliberately generous per-step factor of 0.25 — generous because it assumes W_hh contributes nothing worse than identity. Across the 300-step gap in the fraud example, or even a plain 100-step gap:

>>> 0.25 ** 100
6.223015277861142e-61

A 32-bit float's smallest representable positive number is about 1.4 × 10⁻⁴⁵, and 0.25ⁿ crosses below that around n = 75 (0.25⁷⁵ ≈ 7.0 × 10⁻⁴⁶). So on the 300-step gap back to transaction #40, the gradient has already underflowed to a literal, exact zero in the computer's arithmetic by step 75 — barely a quarter of the way there. There is no learning signal left — SGD sees ∂L/∂W = 0 for every weight that only affects the network through those early steps, and updates them by nothing. The network isn't choosing to ignore transaction #40; it is structurally unable to receive any evidence that transaction #40 existed. This is the vanishing gradient problem, identified by Hochreiter in 1991 and by Bengio, Simard and Frasconi in 1994, and it is the reason plain RNNs are reliable only over short sequences — tens of steps, not hundreds.

The LSTM Cell: An Additive Memory Highway

Hochreiter and Schmidhuber's 1997 fix does not try to make tanh′ bigger or tame W_hh. It changes what kind of operation carries information across time. Alongside the hidden state h_t, the LSTM (Long Short-Term Memory) maintains a separate cell state c_t, updated by addition rather than by repeated matrix-multiply-then-squash:

f_t = σ(W_f·x_t + U_f·h_{t-1} + b_f)      # forget gate
i_t = σ(W_i·x_t + U_i·h_{t-1} + b_i)      # input gate
c̃_t = tanh(W_c·x_t + U_c·h_{t-1} + b_c)   # candidate values
c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t           # additive cell update
o_t = σ(W_o·x_t + U_o·h_{t-1} + b_o)      # output gate
h_t = o_t ⊙ tanh(c_t)

Each gate is a vector of values between 0 and 1 (from the sigmoid σ), one per cell-state unit, learned like any other weight. In the fraud-model reading, the forget gate decides how much of the running summary of "what this account has done so far" to keep; the input gate decides how much of the current transaction's information to write into that summary; the candidate c̃_t is what that new information would look like; the output gate decides how much of the memory to expose for the current-step prediction versus keep hidden for internal bookkeeping. The account-level pattern from transaction #40 can be written into c_t once, given a small weight by the input gate on every irrelevant transaction in between, and left almost untouched by a forget gate close to 1 — surviving 300 steps not because the network is lucky, but because f_t ⊙ c_{t-1} is a direct elementwise multiply, not a multiply-then-saturate-through-tanh.

That distinction is the whole mechanism. Differentiate the cell update with respect to the previous cell state, holding the gates fixed for a moment: ∂c_t/∂c_{t-1} = f_t. There is no tanh′ in this path at all — the cell state's own recurrence is linear in c_{t-1}, scaled only by a gate the network learns. (The gates f_t, i_t and c̃_t do also depend on h_{t-1}, hence indirectly on c_{t-1}, contributing extra terms to the full derivative — but f_t is the dominant term, and it is the one that gives this design its historical nickname, the "constant error carousel": if the network sets f_t ≈ 1 for a unit it wants to preserve, error can circulate around that unit near-losslessly for as many steps as needed.)

Worked Example: Tracing an LSTM Cell by Hand

To see the additive path in action, trace a single-unit (scalar) LSTM cell for two timesteps with deliberately simple weights, chosen so the arithmetic is checkable by hand: set every recurrent weight U to 0, so the gates depend only on the input x_t and a bias, and give the forget gate a bias b_f = 2 (a standard initialization trick, discussed below). Concretely: W_f = 0, b_f = 2; W_i = 0, b_i = 0; W_c = 1, b_c = 0; W_o = 0, b_o = 0. Start with c_0 = 0, h_0 = 0, and feed x_1 = x_2 = 1.0.

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

Wf, bf = 0, 2
Wi, bi = 0, 0
Wc, bc = 1, 0
Wo, bo = 0, 0

c, h = 0.0, 0.0
for t, x in enumerate([1.0, 1.0], start=1):
    f = sigmoid(Wf * x + bf)      # forget gate
    i = sigmoid(Wi * x + bi)      # input gate
    c_tilde = math.tanh(Wc * x + bc)  # candidate
    c = f * c + i * c_tilde       # additive cell update
    o = sigmoid(Wo * x + bo)      # output gate
    h = o * math.tanh(c)
    print(f"t={t}: f={f:.4f} i={i:.4f} c~={c_tilde:.4f} "
          f"c={c:.4f} o={o:.4f} h={h:.4f}")

Trace it step by step. At t=1: f₁ = σ(2) = 1/(1+e⁻²) = 0.8808; i₁ = σ(0) = 0.5; c̃₁ = tanh(1) = 0.7616. The cell update is c₁ = f₁·c₀ + i₁·c̃₁ = 0.8808·0 + 0.5·0.7616 = 0.3808. With o₁ = σ(0) = 0.5, h₁ = 0.5·tanh(0.3808) = 0.5·0.3635 = 0.1817.

At t=2, the recurrent weights are still 0, so f₂, i₂, o₂ take the same values as before (0.8808, 0.5, 0.5) and c̃₂ = tanh(1) = 0.7616 again. But c₁ is no longer zero, so the cell update now actually exercises the additive path: c₂ = f₂·c₁ + i₂·c̃₂ = 0.8808·0.3808 + 0.5·0.7616 = 0.3354 + 0.3808 = 0.7162. Then h₂ = 0.5·tanh(0.7162) = 0.5·0.6145 = 0.3073. Running the code above prints exactly t=1: f=0.8808 i=0.5000 c~=0.7616 c=0.3808 o=0.5000 h=0.1817 and t=2: f=0.8808 i=0.5000 c~=0.7616 c=0.7162 o=0.5000 h=0.3073, matching this derivation to four decimal places.

The number to watch is f = 0.8808. It is the multiplier that would carry a gradient from c₂ back to c₁, and from c₁ back to c₀ if the sequence continued — a fixed, gate-controlled scalar, not a value squeezed through tanh′ at every step.

Why the Forget Gate Changes the Exponent, Not the Exponential

Here is the precise claim, and the misconception it corrects. A very common student summary is: "LSTMs solve the vanishing gradient problem." That overstates it. What the additive cell state actually does is hand the network a decay base it can choose, instead of forcing it to inherit tanh′ · W_hh. The gradient still decays geometrically across timesteps — ∂c_T/∂c_1 is still, to leading order, a product of T forget-gate values — but the network gets to pick that base by learning f_t, rather than being stuck with whatever tanh′ happens to be at each step.

Compare the three decay bases directly across 100 steps, all computed the same way as the vanishing-gradient calculation earlier:

>>> 0.25 ** 100        # plain-RNN-style decay
6.223015277861142e-61
>>> 0.8808 ** 100      # LSTM with our example's forget gate
3.0731695436807123e-06
>>> 0.99 ** 100        # LSTM with a forget gate pushed near 1
0.3660323412732292

With f ≈ 0.88, the 100-step product is about 3.1 × 10⁻⁶ — still small, but 55 orders of magnitude larger than the plain-RNN figure (in log₁₀ terms, −5.5 versus −60.2). That gap is the difference between a gradient that is numerically indistinguishable from zero and one that a modern optimizer can still act on. And if the network learns to push a unit's forget gate to f ≈ 0.99 for information worth keeping, the 100-step decay is only to 0.366 — barely a dent. This is exactly why the forget-gate bias is, by convention, initialized to a positive value like 1 or 2 rather than 0 (a practice validated empirically by Jozefowicz et al., 2015): at initialization, before the network has learned anything, a bias of 0 gives f = σ(0) = 0.5, and 0.5¹⁰⁰ ≈ 8 × 10⁻³¹ — already vanished. Starting from f ≈ 0.88 or higher gives gradients room to flow while training discovers, per unit, which memories are actually worth the near-1 forget gate and which should be closer to 0 and overwritten quickly.

The corollary is the part of the misconception that matters most in practice: an LSTM with poorly initialized or poorly trained gates — one where forget gates saturate toward 0 for units that should be remembering — reduces to the same exponential collapse as a vanilla RNN, because 0.1¹⁰⁰ vanishes just as completely as 0.25¹⁰⁰ does. The architecture makes long-range memory learnable; it does not make it automatic. And even with well-behaved gates, sequences of thousands of steps (a full news article, a genomic sequence, an hour of sensor readings) still show gradual information decay through this same geometric mechanism — which is precisely the gap that attention mechanisms and, later, Transformers were built to close by giving every timestep a direct, non-decaying connection to every other, rather than relying on a chain of gated multiplications at all.

GRU: A Leaner Alternative

The Gated Recurrent Unit, introduced by Cho et al. in 2014, keeps the same core idea — replace repeated saturating multiplication with a learned, gated additive update — but merges the cell state and hidden state into one, and uses two gates instead of three:

z_t = σ(W_z·x_t + U_z·h_{t-1} + b_z)               # update gate
r_t = σ(W_r·x_t + U_r·h_{t-1} + b_r)               # reset gate
h̃_t = tanh(W_h·x_t + U_h·(r_t ⊙ h_{t-1}) + b_h)    # candidate
h_t = (1 − z_t) ⊙ h_{t-1} + z_t ⊙ h̃_t              # update

The update gate z_t plays the combined role of the LSTM's forget and input gates: (1 − z_t) is how much of the old hidden state to keep, and z_t is how much of the new candidate to write in — the two always sum to exactly 1, so the GRU never independently decides to both "forget everything" and "write nothing," a combination the LSTM's separate f_t and i_t technically permit. The reset gate r_t controls how much of the previous hidden state is allowed to influence the new candidate itself, letting the unit effectively start fresh when the past is judged irrelevant to what comes next. Differentiating the update, ∂h_t/∂h_{t-1} again has (1 − z_t) as its direct term — the same additive, gate-controlled structure that fixed the vanilla RNN's derivative, and the same caveat: if z_t saturates toward 1 (fully overwrite every step), (1 − z_t) → 0 and the GRU vanishes too.

With one cell/hidden state instead of two and three weight matrices instead of four, a GRU has roughly 25% fewer parameters than an LSTM of the same hidden size, which usually means faster training and less data needed to fit it well. Empirical comparisons (Chung et al., 2014; Jozefowicz et al., 2015) find the two architectures trade wins depending on task, with no consistent overall champion — a strong reason, when compute and data are limited, to default to the GRU and reach for the LSTM's extra output gate and separate cell state when a task is known to need very long, precisely controlled memory (for instance, models that must retain a piece of state, like an account's running risk score, essentially unmodified across an unusually long and noisy sequence of unrelated events).

Active Recall

Attempt these before reading the answers.

1. In a vanilla RNN using tanh activations, why does the gradient ∂h_T/∂h_1 shrink exponentially with sequence length T, even if W_hh has no eigenvalues greater than 1?
2. What is the single structural change in the LSTM's cell-state update that removes the repeated tanh′ multiplication from the recurrent gradient path?
3. A trained LSTM has a forget gate that outputs 0.999 for the units carrying information about a customer's account-opening date. Estimate 0.999¹⁰⁰⁰ in order of magnitude, and explain what this means for a 1,000-step sequence.
4. Why is it standard practice to initialize the LSTM forget-gate bias to a positive value (e.g., 1 or 2) rather than 0?
5. Give one input on which a GRU's reset gate r_t would output a value near 0, and explain what that does to the candidate h̃_t.
6. True or false, with justification: "LSTMs completely eliminate the vanishing gradient problem."

Answers

1. Because ∂h_t/∂h_{t-1} = diag(tanh′(z_t))·W_hh, and tanh′(z) ≤ 1 with equality only at z = 0. For any unit doing real nonlinear work (z_t away from 0), tanh′(z_t) is well below 1 — often under 0.5, sometimes near 0.01 for a saturated unit. Multiplying T such factors together, even alongside a W_hh with eigenvalues capped at 1, produces a product that decays exponentially in T regardless of W_hh's exact spectrum, since the tanh′ terms alone already shrink it.

2. The cell update c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t is additive and elementwise-linear in c_{t-1} — the dominant term of ∂c_t/∂c_{t-1} is simply f_t, a value the network learns directly, with no tanh′ (or any other saturating derivative) multiplied in along that path.

3. 0.999¹⁰⁰⁰ ≈ e^(1000·ln 0.999) ≈ e^(−1.0005) ≈ 0.368 — essentially the same order of magnitude as the input, not vanished. A forget gate this close to 1 lets a piece of information survive a 1,000-step gap with only about a two-thirds retention factor, which is exactly the regime a well-trained LSTM can reach for memories it has learned matter, in sharp contrast to a vanilla RNN's 0.25¹⁰⁰⁰, which is zero in every sense a computer can represent.

4. At initialization the network has learned nothing, so without this trick the forget gate starts at σ(0) = 0.5. Across even 50–100 steps, 0.5ᵀ is already numerically negligible, so gradient signal from distant timesteps is lost before training even begins — there is nothing to learn from. Starting the bias at 1 or 2 gives f ≈ 0.73–0.88 from step zero, keeping the gradient alive long enough for the network to discover, through training, which units should keep an even higher forget gate and which should be overwritten quickly.

5. r_t ≈ 0 for a timestep where the previous hidden state is judged irrelevant to the next candidate value — for example, the first token after a sentence boundary or a topic change in a language model, or the first transaction after a long dormant period in an account-history model. When r_t ≈ 0, the term r_t ⊙ h_{t-1} inside the candidate computation goes to (near) zero, so h̃_t = tanh(W_h·x_t + b_h) is computed almost entirely fresh from the current input, effectively letting the unit ignore stale history rather than blend it in.

6. False. LSTMs mitigate the vanishing gradient problem by replacing a fixed, structurally shrinking decay (tanh′ · W_hh at every step) with a learned, gate-controlled decay (f_t at every step) that the network can push close to 1 for information worth keeping. The gradient along the cell-state path is still a product of per-step factors and still decays geometrically — it is just that the base of that geometric decay is now something the network optimizes rather than something imposed on it by the architecture. A poorly trained or poorly initialized LSTM, with forget gates saturated near 0, vanishes exactly as a vanilla RNN does, and even well-trained LSTMs show gradual decay over sequences of many thousands of steps.

Think About It

Think about this: How would you explain lstms and grus: solving the vanishing gradient problem to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.