Recurrent Neural Networks and Sequence Models

Open WhatsApp and start typing a message to a friend: "Yaar, we are reaching the station in 10..." Before you tap another key, your phone's keyboard has already guessed the next word: minutes. Now type a completely different sentence that also ends in the number 10: "The bowler has picked up wickets in his last 10..." This time the suggestion is different — overs, or matches, or balls, depending on what you typed earlier in the sentence. The keyboard is not reacting to the digit 10 in isolation. It is holding on to everything typed before it — station, bowler, wickets — and using that accumulated context to guess what comes next. This chapter is about how a neural network can be built to do exactly that: read a sequence one piece at a time, remember what it has already seen, and use that memory to make sense of, or predict, what comes next.

Why a Feedforward Network Falls Short

Every network you have built so far — a plain multi-layer perceptron, a convolutional network for images — expects a fixed-size input and produces an output from it with no notion of "what came before." That works well for a photograph, where every pixel arrives at once, but it breaks down the moment order carries meaning. Consider two IRCTC search queries built from the same three words: "Delhi to Mumbai" and "Mumbai to Delhi." A bag-of-words representation — a vector that simply counts how often each word appears, ignoring the order they appeared in — is identical for both sentences: Delhi appears once, Mumbai appears once, to appears once. Yet one query is asking for a ticket originating in Delhi and the other in Mumbai. Feed either representation into an ordinary feedforward network and it cannot tell them apart, because order was discarded before the network ever saw the input. Sentences, speech signals, stock prices, and cricket scorecards all share this property: they are sequences, where the position of each element changes what the whole thing means, and where the length of the input is not fixed in advance — a review might be five words or five hundred. A network built for sequences needs two abilities a feedforward network does not have: a way to process inputs of varying length using the same set of weights, and a way to carry information from earlier positions forward to later ones.

The Recurrent Neuron: A Loop with Memory

The fix is conceptually simple: add a loop. A recurrent neural network (RNN) keeps a hidden state — a vector that acts as the network's running memory — and updates it one input at a time. At each time step t, the network looks at two things: the current input x(t) (say, the one-hot encoded vector for the current character or word) and the hidden state h(t-1) it computed at the previous step. It combines them into a new hidden state:

h(t) = tanh(W_xh · x(t) + W_hh · h(t-1) + b_h)

and, when the task calls for an output at this step, produces one from the current hidden state:

y(t) = W_hy · h(t) + b_y

The crucial detail is that W_xh, W_hh, W_hy, and the two bias vectors do not change from one time step to the next. The same three weight matrices are reused at every position in the sequence, a design called weight sharing. This is what lets the network handle a three-word query and a three-hundred-word product review with the same set of learned parameters: the loop simply runs three times or three hundred times, applying identical arithmetic at each step. "Unroll" this loop across time — draw a separate copy of the network for t = 1, 2, 3, and so on — and it looks like a very deep feedforward network, except every layer shares the same weights and takes a fresh external input at every depth. That is the whole idea: read one token, update memory, read the next token, update memory again, and keep going until the sequence ends.

Worked Example: Teaching a Tiny RNN to Spell INDIA

These equations are easiest to trust once real numbers have been pushed through them by hand. Build a minimal character-level RNN over a four-letter vocabulary — I, N, D, A — the letters needed to spell INDIA (I-N-D-I-A). Each character is one-hot encoded as a 4-dimensional vector with a single 1 marking its position:

I = [1,0,0,0]   N = [0,1,0,0]   D = [0,0,1,0]   A = [0,0,0,1]

Fix the hidden state size at 2 and choose small illustrative weights — not yet trained, just fixed numbers so the arithmetic is easy to trace by hand:

• W_xh = [[0.5, -0.3, 0.2, 0.1], [0.1, 0.4, -0.2, 0.3]]
• W_hh = [[0.3, -0.1], [0.2, 0.4]]
• W_hy = [[2.0, 1.0], [-1.0, 0.5], [0.5, -1.0], [-1.5, 1.5]] (rows ordered I, N, D, A)

with all biases at zero and initial hidden state h(0) = [0, 0]. Feed in the first three letters, I, N, D, and see what the network predicts for the next position — it should ideally be I, since INDIA continues I-N-D-I-A.

• Input I: the one-hot vector for I picks out column 0 of W_xh, which is [0.5, 0.1]. Since h(0) is all zeros, the W_hh term contributes nothing. So h(1) = tanh([0.5000, 0.1000]) = [0.4621, 0.0997].
• Input N: column 1 of W_xh is [-0.3, 0.4]. The recurrent term is W_hh · h(1) = [0.3(0.4621) - 0.1(0.0997), 0.2(0.4621) + 0.4(0.0997)] = [0.1287, 0.1323]. Adding gives a pre-activation of [-0.1713, 0.5323], so h(2) = tanh([-0.1713, 0.5323]) = [-0.1697, 0.4871].
• Input D: column 2 of W_xh is [0.2, -0.2]. The recurrent term is W_hh · h(2) = [0.3(-0.1697) - 0.1(0.4871), 0.2(-0.1697) + 0.4(0.4871)] = [-0.0996, 0.1609]. Adding gives [0.1004, -0.0391], so h(3) = tanh([0.1004, -0.0391]) = [0.1000, -0.0391].

Notice that h(3) is not simply a function of D. It carries a compressed trace of I and N as well, folded in through two earlier rounds of multiplication and tanh squashing — that compressed trace is the network's entire memory of the sequence so far. To predict the fourth character, pass h(3) through the output layer and a softmax, which turns any vector of numbers into a probability distribution that sums to 1:

• logits = W_hy · h(3) = [0.1610, -0.1196, 0.0891, -0.2087] for I, N, D, A respectively
• softmax(logits) = [0.2961, 0.2237, 0.2756, 0.2046]

The network assigns the highest probability, 29.6%, to I — correctly identifying that the letter after "IND" should be I. The margin over D (27.6%) is modest, which makes sense: these weights were picked by hand for illustration, not learned. Train the same network on real spelling data using backpropagation through time (covered next) and gradient descent would push W_hy and W_hh toward values that make this prediction sharper and more confident. Every number above can be reproduced exactly with a dozen lines of NumPy:

import numpy as np

vocab = ['I', 'N', 'D', 'A']
onehot = {
    'I': np.array([1,0,0,0.]), 'N': np.array([0,1,0,0.]),
    'D': np.array([0,0,1,0.]), 'A': np.array([0,0,0,1.]),
}

W_xh = np.array([[0.5, -0.3, 0.2, 0.1],
                  [0.1,  0.4, -0.2, 0.3]])
W_hh = np.array([[0.3, -0.1],
                  [0.2,  0.4]])
W_hy = np.array([[2.0, 1.0], [-1.0, 0.5],
                  [0.5, -1.0], [-1.5, 1.5]])

h = np.array([0.0, 0.0])          # h(0)
for ch in ['I', 'N', 'D']:
    x = onehot[ch]
    h = np.tanh(W_xh @ x + W_hh @ h)   # biases are zero here
    print(ch, h)

logits = W_hy @ h
probs = np.exp(logits) / np.exp(logits).sum()
print(dict(zip(vocab, np.round(probs, 4))))

Running this script prints the same h(1), h(2), h(3), and softmax probabilities computed above — the RNN forward pass is nothing more than a loop over the sequence, doing one matrix multiply and one tanh per step.

Learning Through Time: Backpropagation Through Time

Training an RNN means adjusting W_xh, W_hh, and W_hy so predictions like the one above get closer to the correct answer, across many training sequences. Because the network is just an unrolled feedforward computation once the sequence length is fixed, ordinary backpropagation still applies — this specific application of it is called backpropagation through time (BPTT). A loss is computed at each output time step (for the spelling example, the cross-entropy between the predicted distribution and the true next character), and gradients are propagated backward through the unrolled graph, from the last time step to the first. The key wrinkle is that W_hh contributed to the computation at every single time step, so its gradient is a sum of contributions from all of them: the gradient at time step 3 flows back through h(3), then h(2), then h(1), and each of those paths adds its own term to the gradient of the loss with respect to W_hh before a single gradient-descent update is applied. This summing-across-time is exactly what lets one shared weight matrix learn a pattern that holds across an entire sequence — but it is also the source of the RNN's most famous weakness.

The Vanishing Gradient Problem

Follow that backward path carefully. Each step back through time multiplies the gradient by (roughly) W_hh and by the derivative of tanh at that step. The derivative of tanh is at most 1, reached only at input 0, and smaller everywhere else, so in practice each backward step scales the gradient signal by some factor comfortably below 1. The trouble is what happens when many such factors are multiplied together. If each of ten time steps contributed a shrinking factor of just 0.25, the gradient reaching the earliest step would be scaled by 0.25 to the power 10, which is about 0.00000095 — roughly one-millionth of its original size. Even a gentler shrink of 0.9 per step compounds to roughly 0.0052 after fifty steps. A gradient that small carries essentially no learning signal: the earliest inputs in a long sequence stop influencing the weight updates at all, and the network cannot learn a dependency that spans that distance. This difficulty, formally analyzed by Yoshua Bengio, Patrice Simard, and Paolo Frasconi in 1994, is called the vanishing gradient problem, and it means a plain RNN is, in practice, only reliable on short-range patterns. It can finish "the sky is ___" without trouble, because "sky" sits right next to the blank, but it struggles with something like a product review that opens with "Would not recommend this phone to anyone" and then spends the next eighty words on detailed, neutral-sounding technical specifications before ending — by the time the network reaches the end of the review, the opening sentiment has all but vanished from its hidden state.

The mirror-image failure is the exploding gradient problem: when the repeated multiplicative factors are greater than 1 instead of less, the same chain-rule product grows exponentially rather than shrinking, and a single weight update can throw the network's parameters to nonsensical values. Exploding gradients are comparatively easy to control with gradient clipping — rescaling the gradient vector whenever its size exceeds some fixed threshold, before it is used in the update. Vanishing gradients have no such easy fix, which is why the architectural solution below became the standard approach.

Long Short-Term Memory: A Separate Channel for Memory

Long Short-Term Memory (LSTM) networks, introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997, address this by giving the network two streams instead of one: the usual hidden state h(t), and a cell state C(t) that acts like a conveyor belt running alongside it. Information can ride along the cell state with only gentle, mostly linear modifications — multiplied and added to, but not squashed through a nonlinearity at every step — which is precisely what lets gradients survive a long journey backward through time. Three gates, each a small sigmoid-activated layer producing values between 0 and 1, control what happens to that conveyor belt at every time step:

• Forget gate: f(t) = sigmoid(W_f · [h(t-1), x(t)] + b_f) — decides what fraction of the old cell state to keep.
• Input gate: i(t) = sigmoid(W_i · [h(t-1), x(t)] + b_i) — decides how much of a new candidate value to add.
• Candidate values: g(t) = tanh(W_g · [h(t-1), x(t)] + b_g) — proposes new content.
• Output gate: o(t) = sigmoid(W_o · [h(t-1), x(t)] + b_o) — decides how much of the cell state to reveal as output.

These combine into the cell state update and the new hidden state:

• C(t) = f(t) × C(t-1) + i(t) × g(t)
• h(t) = o(t) × tanh(C(t))

Here × denotes element-wise multiplication — multiplying matching entries of two vectors together, not a matrix product. Think of it the way a student might manage a running set of notes through a school year: the forget gate is the decision about which old notes are no longer relevant and can be crossed out; the input gate is the decision about which new fact from today's lesson is worth writing down; and the output gate is the decision, when asked a question right now, about which parts of the notes are actually relevant to answer it. The gate values themselves are learned through the same backpropagation through time described above, so the network is trained to open and close each gate at the right moments for its task. Because the forget gate can learn to sit close to 1 for information that matters over a long stretch, the cell state lets that information survive across dozens or hundreds of steps largely undamaged, sidestepping the repeated squashing that causes vanishing gradients in a plain RNN. The forget gate itself was not part of Hochreiter and Schmidhuber's original 1997 design; it was added by Felix Gers, Schmidhuber, and Fred Cummins in 2000, and the three-gate version described here is the one used almost universally today.

Gated Recurrent Units: A Leaner Alternative

Gated Recurrent Units (GRU), proposed by Kyunghyun Cho and colleagues in 2014, achieve much the same effect with a simpler design. A GRU drops the separate cell state entirely and merges the forget and input gates into a single update gate:

• z(t) = sigmoid(W_z · [h(t-1), x(t)]) — update gate: how much of the past to carry forward versus overwrite.
• r(t) = sigmoid(W_r · [h(t-1), x(t)]) — reset gate: how much of the past hidden state to ignore when proposing new content.
• g(t) = tanh(W · [r(t) × h(t-1), x(t)]) — candidate hidden state.
• h(t) = (1 - z(t)) × h(t-1) + z(t) × g(t) — final hidden state, a blend of old and new.

Look closely at the last line: h(t) is an interpolation between the previous hidden state and the new candidate, controlled entirely by z(t). When z(t) is close to 0, the GRU almost exactly copies h(t-1) forward — an additive, gradient-friendly path very similar in spirit to the LSTM's cell state, achieved with one fewer gate and no separate memory channel. With fewer parameters to learn, a GRU trains a little faster and needs somewhat less data than an LSTM of comparable hidden size, and on many practical NLP tasks the two perform comparably; very long-range dependencies are where the LSTM's dedicated cell state most often keeps an edge. In practice, both are reasonable defaults, and the choice is usually settled by trying both on a validation set rather than by theory alone.

Four Shapes of Sequence Problems

Not every sequence task looks the same, and RNNs — or LSTMs and GRUs used as drop-in replacements for the plain recurrent cell — can be arranged to match the shape of the problem:

• Many-to-one: read an entire sequence, produce a single output at the end. Feeding the network one word at a time from a Flipkart or Amazon product review and reading a single positive or negative label off the final hidden state is a many-to-one task — sentiment classification.
• One-to-many: a single input unfolds into a sequence of outputs. Generating a caption, one word at a time, from a single image's feature vector is a classic one-to-many setup.
• Many-to-many, synced: one output per input position, aligned in time. Tagging every word of a sentence with its part of speech — noun, verb, adjective — produces exactly as many tags as there are words, each depending on the words around it.
• Many-to-many, encoder-decoder: the input and output sequences can have different lengths and are not aligned position for position. In an encoder-decoder system, one RNN (the encoder) reads an entire input sentence and compresses it into a final hidden state, and a second RNN (the decoder) unrolls that hidden state back out, one word at a time, stopping only when it generates an end-of-sentence token, regardless of whether the source sentence had six words or sixteen. Machine translation is the standard example.

This last pattern is not just theoretical. In 2016, Google switched Google Translate to a neural encoder-decoder system built from stacked LSTM layers with an attention mechanism, and reported that human evaluators judged its translations to contain, on average, 60% fewer errors than the older phrase-based system it replaced. The underlying idea — read the whole input first, then generate the whole output — is exactly the many-to-many pattern above, run at the scale of billions of sentence pairs across languages including Hindi, Bengali, Tamil, and Telugu.

Reading Both Directions: Bidirectional RNNs

Every RNN considered so far reads left to right, using only past context to interpret the current position. That is unavoidable when generating text one word at a time — there is no way to use words that do not exist yet — but for tasks where the entire input is already available before an answer is needed, restricting the network to past context throws away useful information. Consider the word "current" in two banking-related sentences: "She opened a current account at the bank" versus "She read the current affairs section of the newspaper." A left-to-right reader sees only "She opened a current" or "She read the current" at the moment it processes the word "current" — identical context so far, with no way to tell the two apart. The word that follows, account or affairs, is what actually resolves the meaning. A bidirectional RNN handles this by running two separate RNNs over the same input: one left to right as usual, and a second right to left, over the reversed sequence. The hidden states from both directions are combined at each position, so the representation for "current" in either sentence includes everything that came before it and everything that came after it. Bidirectional LSTMs and GRUs are standard for tasks like named-entity recognition or part-of-speech tagging, where the whole sentence is available upfront and every scrap of context helps; they are not used for tasks like live speech transcription or text generation, where the future simply has not happened yet.

Building a Review Classifier with Keras

Putting these pieces together, here is a complete architecture for classifying product reviews as positive or negative, using an LSTM layer in place of the plain recurrent cell derived earlier:

import tensorflow as tf
from tensorflow.keras import layers, models

vocab_size = 10000      # size of the word vocabulary
max_length = 100        # reviews are padded/truncated to 100 tokens
embedding_dim = 64

model = models.Sequential([
    layers.Input(shape=(max_length,)),
    layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim),
    layers.LSTM(64),
    layers.Dense(32, activation='relu'),
    layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.summary()

Each layer maps directly onto an idea from this chapter. Embedding turns each word index into a dense 64-dimensional vector, a learned representation rather than a one-hot vector, but conceptually playing the same "x(t) at each time step" role that the one-hot letters played in the INDIA example. LSTM(64) is exactly the gated recurrent cell derived above, unrolled internally across all 100 positions and returning only its final hidden state, because sentiment classification is a many-to-one task where there is no need to keep every intermediate hidden state. The two Dense layers turn that final 64-dimensional summary of the whole review into a single probability that the review is positive, and binary_crossentropy compares that probability against the true label during training. Swap layers.LSTM(64) for layers.GRU(64) and nothing else in the architecture or training loop needs to change — only the internal gating mechanics differ, which is exactly the drop-in relationship described earlier.

Back to the Keyboard

Return to the WhatsApp message from the start of this chapter. A predictive keyboard guessing the word after "10" is doing, at enormous scale, exactly what the four-letter INDIA example did by hand: reading a sequence one token at a time, folding each new token into a hidden state that summarizes everything seen so far, and using that hidden state to produce a probability distribution over what comes next. For years, Gboard's on-device word suggestions ran on compact LSTM language models, small enough to run on a phone, trained in part using many Android devices collaborating through federated learning without any individual's typed message ever leaving that phone. The specific architectures keep evolving — attention-based Transformer models now handle much of the sequence-modeling work that LSTMs and GRUs used to carry alone, including in the newest keyboards and translation systems. But the core insight built from first principles here — that meaning in a sequence depends on order, that a network needs a running memory to capture it, and that gates are what keep that memory from decaying over long distances — is the idea every later sequence model, attention included, is still built on top of. The next time a keyboard finishes a sentence correctly, or a translation app turns a Hindi voice note into readable English, there is a hidden state, updated one token at a time, doing the quiet work underneath.

Think About It

Think about this: How would you explain recurrent neural networks and sequence models to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.