Recurrent Neural Networks and LSTMs

Open Gboard or SwiftKey on an Android phone and start typing a message the way most Indians actually text: "mausam aaj bahut". Before the word is even finished, the keyboard is already offering "accha" or "kharab" as the next word. It did not look up "mausam" in a dictionary and guess a generic continuation — it read the whole sequence you typed, in the order you typed it, and used that history to narrow the prediction. Shuffle the same three words to "bahut aaj mausam" and a competent predictor's ranked suggestions shift too, even though the bag of words is identical. Order carries information. A model that only ever sees "which words are present," with no notion of sequence, throws that information away before it even starts.

This is exactly the wall a plain feedforward network hits on language. A feedforward classifier takes a fixed-size input vector and produces an output — that is the whole contract. To predict a next word from the previous three words, you would have to hard-code a window: concatenate the embeddings of exactly three previous words into one fixed-length vector, feed that in, and hope three is always the right amount of context. This "fixed-window neural language model" is a real, historically important architecture (Bengio et al., 2003), and it works — until the dependency you need is four words back, or forty. The window size is a hyperparameter you picked before training ever started; the sentence does not get a vote. What language modelling actually needs is a network that can consume a sequence of any length and carry forward a running summary of everything it has read so far. That is precisely what a recurrent neural network is built to do.

The recurrent idea: the same weights, called again and again

An RNN keeps a hidden state — a vector, call it h_t — that gets updated at every timestep by combining the current input with the previous hidden state:

h_t = tanh(W_xh · x_t + W_hh · h_(t-1) + b_h)
y_t = W_hy · h_t + b_y

Three things matter here more than the formula itself. First, W_xh, W_hh, W_hy and the biases are the same matrices at every timestep — there is exactly one set of weights, reused for a 5-word sentence or a 500-word document. Second, h_t is a fixed-size vector (say, 128 numbers), not a growing list — whatever the network wants to remember about everything it has seen so far has to fit in that one vector, every single step. Third, if you are comfortable with recursion, the structure should feel familiar: an RNN cell is the same "function" (the same weights) invoked again at every position in the sequence, threading an accumulator — the hidden state — from one call into the next, the way a tail-recursive function threads an accumulator through its calls. The difference is that there is no call stack and no branching: it is a straight-line loop over timesteps, and the "return value" you actually care about (y_t) can be read off at every step, or only at the last one, depending on the task — word-by-word tagging needs a y_t at every position, while sentence classification (say, tagging a product review as positive or negative) only needs the final h_T once the whole sentence has been read.

"Unrolling" an RNN just means drawing this loop out flat, one copy of the cell per timestep, so you can see how information flows forward through h_t and, during training, how error flows backward through the same path:

x_1 -> [RNN cell] -> h_1 -\
                          \
x_2 -> [RNN cell] -> h_2 -> ... -> h_T -> y_T
        ^                ^
        |________________|
     same W_xh, W_hh, W_hy reused at every step

Worked example: predicting one keystroke at a time

Take a toy vocabulary of three characters — U, P, I — one-hot encoded as x_U = [1,0,0], x_P = [0,1,0], x_I = [0,0,1]. The task: after reading "U" then "P", predict the next character. A well-trained model on enough "UPI"-shaped strings should push probability mass toward "I". Here is a hidden size-2 RNN with freshly initialised (untrained) weights, traced by hand, one multiplication at a time.

import numpy as np

x_U = np.array([1, 0, 0])
x_P = np.array([0, 1, 0])

W_xh = np.array([[ 0.5, -0.3,  0.1],
                  [ 0.2,  0.4, -0.5]])   # shape (hidden=2, vocab=3)
W_hh = np.array([[ 0.1, -0.2],
                  [ 0.3,  0.05]])        # shape (hidden=2, hidden=2)
b_h  = np.array([0.0, 0.0])

h = np.array([0.0, 0.0])                # h_0: nothing read yet

# timestep 1: read "U"
z = W_xh @ x_U + W_hh @ h + b_h          # z = [0.5, 0.2]
h = np.tanh(z)                           # h_1 = [0.4621, 0.1974]

# timestep 2: read "P"
z = W_xh @ x_P + W_hh @ h + b_h          # z = [-0.2933, 0.5485]
h = np.tanh(z)                           # h_2 = [-0.2851, 0.4994]

W_hy = np.array([[ 0.6, -0.1],
                  [-0.2,  0.5],
                  [ 0.3,  0.3]])          # shape (vocab=3, hidden=2)
logits = W_hy @ h                        # [-0.2210, 0.3067, 0.0643]
probs  = np.exp(logits) / np.exp(logits).sum()
# probs = [0.2484, 0.4211, 0.3305]  for  [U, P, I]

Trace the arithmetic yourself for timestep 1: W_xh·x_U just selects column 0 of W_xh, giving [0.5, 0.2]; W_hh·h₀ is zero because h₀ is zero; so z = [0.5, 0.2] and h₁ = [tanh(0.5), tanh(0.2)] = [0.4621, 0.1974]. At timestep 2, W_xh·x_P selects column 1, [-0.3, 0.4], and now W_hh·h₁ is no longer zero — it mixes the memory of "U" into the new state: [0.1(0.4621) + (-0.2)(0.1974), 0.3(0.4621) + 0.05(0.1974)] = [0.0067, 0.1485]. Adding gives z = [-0.2933, 0.5485], and h₂ = [-0.2851, 0.4994]. Push that through the output layer and softmax and you get roughly 25% U, 42% P, 33% I. The untrained network's top guess is "P", not "I" — which is exactly what you should expect from random weights that have never seen a training example. That gap between the current output (42% P) and the target (100% I) is what backpropagation-through-time exists to close, by nudging W_xh, W_hh and W_hy so that repeated exposure to "U", "P" → "I" gradually raises P(I) and lowers the others.

Why training breaks down: vanishing gradients

Training an RNN means running backpropagation through the unrolled graph — "backpropagation through time" (BPTT). To get the gradient of the loss at the final timestep with respect to the weights used at timestep 1, the chain rule has to walk backward through every intermediate hidden state, and at every single step it multiplies in a factor of roughly diag(tanh′(z_t))·W_hh. Two things make this dangerous. tanh′(x) = 1 − tanh(x)² has a maximum value of exactly 1, reached only at x = 0, and shrinks toward 0 as the neuron saturates in either direction — so it is almost always comfortably less than 1. Multiply a sequence of such factors together, once per timestep, and the product shrinks geometrically. If the typical per-step factor across a passage is around 0.5, then after just 20 timesteps the surviving gradient has been scaled by roughly 0.5²⁰ = 1/1,048,576 ≈ 9.5 × 10^-7. For a 20-word sentence, that means the gradient signal telling the network "the first word mattered for this outcome" has, for all practical purposes, vanished by the time it is computed. The network simply cannot learn a dependency that spans that many steps — the update to the weights that would fix it is numerically indistinguishable from zero. (The mirror-image failure, exploding gradients, happens when that per-step factor is consistently greater than 1; it is a real problem too, but it is comparatively easy to patch with gradient clipping. Vanishing gradients have no such cheap fix — they need a different architecture.)

A misconception worth killing now

Many students picture the RNN's hidden state as a buffer that literally stores the sequence — as if h_t remembered "first there was a U, then a P" the way a list remembers its elements. It does not. h_t is a fixed-size vector — 2 numbers in the worked example above, maybe 512 in a real model — and at every timestep it is entirely overwritten by a fresh tanh computation that mixes in the new input. Nothing about that formula preserves old values verbatim; it is a lossy, learned compression of everything seen so far, continuously rewritten. That is not an incidental detail — it is the same tanh squashing that overwrites h_t every step that is responsible for the vanishing-gradient decay above. The hidden state does not "hold onto" the first word; it is asked, at every single step, to re-derive a whole new summary from scratch, and information about distant history keeps getting diluted in that re-derivation. Fixing this requires an architecture that separates "what to carry forward unchanged" from "what to recompute," and that is the entire design idea behind the LSTM.

LSTM: an unobstructed highway for the cell state

A Long Short-Term Memory cell keeps two vectors instead of one: the familiar hidden state h_t, and a second vector — the cell state c_t — that is updated additively rather than by a full overwrite. Three sigmoid "gates" (each outputting values in (0,1), read as "how much of this to let through") and one tanh "candidate" control the update:

f_t = sigma(W_f · [h_(t-1), x_t] + b_f)   # forget gate
i_t = sigma(W_i · [h_(t-1), x_t] + b_i)   # input gate
g_t = tanh (W_g · [h_(t-1), x_t] + b_g)   # candidate values
c_t = f_t * c_(t-1) + i_t * g_t              # cell state update
o_t = sigma(W_o · [h_(t-1), x_t] + b_o)   # output gate
h_t = o_t * tanh(c_t)                        # hidden state (read-out)

The key line is c_t = f_t·c_t-1 + i_t·g_t. It is additive: the old cell state is scaled by the forget gate (not squashed through tanh) and a new contribution is added on top. If the network learns to push f_t close to 1 for a piece of information it needs to keep, that information rides down the cell-state path across many timesteps picking up almost no decay — no repeated tanh crunch, no forced multiplication by W_hh at every step. The gradient flowing backward through that additive path during BPTT is multiplied mainly by f_t itself, which the network controls and can keep near 1, rather than by an uncontrolled tanh derivative. The hidden state h_t is still bounded and still recomputed every step (it is the "read-out" the rest of the network sees), but the long-term memory now lives in c_t, on a path built to survive many steps rather than fight them.

Worked example: one LSTM step

Suppose at some timestep, given the current input and h_t-1, the four gates' pre-activations (after their own W·[h,x]+b) come out as below, with an incoming cell state c_t-1 = [1.2, -0.8]:

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

c_prev = np.array([1.2, -0.8])

z_f = np.array([3.0, 2.5])     # forget gate pre-activation
z_i = np.array([-1.0, 0.5])    # input gate pre-activation
z_g = np.array([0.8, -0.6])    # candidate pre-activation
z_o = np.array([1.5, 0.2])     # output gate pre-activation

f_t = sigmoid(z_f)             # [0.9526, 0.9241]
i_t = sigmoid(z_i)             # [0.2689, 0.6225]
g_t = np.tanh(z_g)             # [0.6640, -0.5370]
o_t = sigmoid(z_o)             # [0.8176, 0.5498]

c_t = f_t * c_prev + i_t * g_t # [1.3217, -1.0736]
h_t = o_t * np.tanh(c_t)       # [0.7090, -0.4348]

Dimension 1 of the cell state barely moved (1.2000 to 1.3217) because the forget gate there is 0.9526 — the network is deliberately preserving whatever that slot was already tracking, while blending in a modest amount of new information. Notice also that c_t is allowed to sit outside the (-1, 1) range that bounds tanh outputs — 1.3217 is not something a plain tanh could ever produce. That headroom exists on purpose: c_t is an accumulator, not a squashed output, and only gets passed through tanh at the very last moment, when computing h_t = o_t·tanh(c_t), to produce the bounded value the rest of the network actually consumes. The internal memory is never crushed; only the external read-out is.

Vanilla RNN versus LSTM, at a glance

Where this mattered, and where it stopped being enough

Stacked LSTMs were the backbone of production machine translation for years — Google's Neural Machine Translation system, which replaced Google Translate's older phrase-based pipeline in 2016, used deep stacks of LSTM encoders and decoders connected by attention. Sequence taggers built on LSTMs still show up in named-entity recognition, intent detection for support chatbots, and any setting where inputs are a variable-length stream read strictly in order. The architectural weakness that eventually mattered more than the vanishing-gradient fix, though, is a different one: an RNN or LSTM must process a sequence one timestep at a time, because h_t genuinely depends on h_t-1. There is no way to compute step 500 before step 499 finishes, which means training cannot be parallelised across the length of a sequence, only across separate sequences in a batch. That sequential bottleneck — not the memory problem this chapter solved — is precisely what the transformer architecture was designed to remove, by replacing recurrence with attention computed over the whole sequence at once.

Active recall

Attempt every question before reading its answer.

1. A vanilla RNN with hidden size 64 processes a 50-word sentence. How many distinct W_hh matrices does it use across the 50 timesteps, and why?
2. Using the worked RNN forward pass above, is the untrained network's top prediction after reading "U", "P" actually "I"? State the three probabilities and explain what that tells you about a freshly initialised network.
3. Name the two multiplicative factors that make an early timestep's gradient shrink as a vanilla RNN's sequence gets longer.
4. In the worked LSTM step, dimension 1 had f_t ≈ 0.95 and i_t ≈ 0.27. In plain language, what is the network doing to that memory slot at this timestep?
5. A classmate says: "LSTMs completely solve the vanishing gradient problem, so you can safely feed one a 10,000-token sequence." What's wrong with that claim?
6. Why is c_t allowed to take values outside (-1, 1), while h_t is always squashed into (-1, 1)?

Answers.

1. Exactly one. W_hh (and W_xh, b_h) are shared and reused at every timestep — that weight-sharing is what lets a fixed number of parameters handle a sequence of any length, and it is the defining property that makes the network "recurrent" rather than just "50 separate feedforward layers."
2. No. Softmax gives roughly [U: 0.2484, P: 0.4211, I: 0.3305] — the network's top guess is "P," not the correct "I." That is exactly what you should expect from random initial weights: the model has not yet seen a single training example, so its output distribution reflects the random weights, not the task. Backpropagation-through-time is what would gradually raise P(I) and lower the others across many training examples.
3. diag(tanh′(z_t)), whose entries are at most 1 and shrink toward 0 as neurons saturate, and W_hh (specifically its dominant eigenvalue). Chaining one such factor per timestep across a T-step backward pass multiplies T of these together, and the product decays geometrically in T when the typical factor is below 1.
4. It is keeping about 95% of whatever that memory slot already held from previous timesteps (barely forgetting it) while blending in only about 27% weight on the freshly computed candidate value — so that slot is dominated by carried-over history rather than by what just happened at this input.
5. LSTMs make it far easier for gradients to survive many steps — the cell-state path is additive, and a forget gate can learn to sit near 1 — but "far easier" is not "solved." Forget gates are still sigmoids that can saturate near 0 for a given dimension, and in practice LSTMs still degrade well before 10,000 steps; that residual limitation, not this one alone, is part of why attention-based architectures were later developed for very long contexts.
6. c_t is an accumulator built by addition across many timesteps and needs headroom to represent magnitude/confidence without being clipped every step; h_t = o_t·tanh(c_t) is the bounded, gated read-out used by the rest of the network. Squashing only happens at that final read-out, not on the stored memory itself — which is exactly what protects c_t from the every-step crushing that a vanilla RNN's h_t suffers.

Think About It

Think about this: How would you explain recurrent neural networks and lstms to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.