Recurrent Neural Networks and LSTMs

Introduction: The Problem of Sequential Data

Most real-world data is sequential: text (words in order), speech (sounds over time), stock prices (values in time), DNA sequences (nucleotides in order). Standard MLalgorithms treat each sample independently and identically — they ignore the crucial fact that position matters. A recurrent neural network (RNN) maintains memory of past information to inform current decisions.

Today, Transformers have largely replaced RNNs in production (faster, more parallelizable), but understanding RNNs is critical because:

1. Transformers are built on attention, which evolved from RNN attention mechanisms
2. Many competitive exams (KVPY, JEE Advanced) still ask RNN questions
3. Certain domains still use RNNs: time series forecasting, online learning, streaming data
4. Understanding RNNs' limitations explains why Transformers were necessary

At IIT-Delhi and IIIT-Hyderabad, RNNs are the gateway to understanding sequence modeling, essential for building speech recognition systems for Indian languages and automatic speech translation.

Basic RNN: Hidden State as Memory

A vanilla RNN processes a sequence x₁, x₂, ..., xₜ by maintaining a hidden state h that summarizes past information:

hₜ = tanh(Wₕ hₜ₋₁ + Wₓ xₜ + b)

yₜ = softmax(W_y hₜ + b_y) [if generating output at each step]

Key insight: The same weights (Wₕ, Wₓ) are reused at every timestep. This parameter sharing is what makes RNNs work on variable-length sequences.

The Critical Problem: Vanishing and Exploding Gradients

When training RNNs via backpropagation through time (BPTT), gradients flow backward through many timesteps. The chain rule repeatedly multiplies gradients by W_h:

∂hₜ/∂hₜ₋₁ = Wₕ × (derivative of tanh)

If |Wₕ| < 1, gradients shrink exponentially: gradient ∝ (0.9)^t → 0 for large t. RNN forgets what happened at t=0 by t=100. This is vanishing gradient — the network can't learn long-range dependencies.

If |Wₕ| > 1, gradients explode: gradient → ∞, causing numerical overflow. This is exploding gradient — training becomes unstable.

This fundamental problem limits vanilla RNNs to learning only ~5-20 timestep dependencies in practice.

LSTM: The Solution to Long-Term Memory

Long Short-Term Memory (LSTM), invented by Hochreiter & Schmidhuber (1997), solves vanishing gradients through a clever architecture with four gates that regulate information flow:

Forget Gate (fₜ): How much of the past cell state to forget?

fₜ = sigmoid(W_f · [hₜ₋₁, xₜ] + b_f) ∈ [0, 1]

Input Gate (iₜ): How much of the new input to save?

iₜ = sigmoid(W_i · [hₜ₋₁, xₜ] + b_i) ∈ [0, 1]

Cell Update (C̃ₜ): What new information to add?

C̃ₜ = tanh(W_c · [hₜ₋₁, xₜ] + b_c) ∈ [-1, 1]

New Cell State (Cₜ): Combine old memory with new input

Cₜ = fₜ ⊙ Cₜ₋₁ + iₜ ⊙ C̃ₜ [element-wise multiplication]

Output Gate (oₜ): What part of cell state to expose?

oₜ = sigmoid(W_o · [hₜ₋₁, xₜ] + b_o) ∈ [0, 1]

Hidden State (hₜ): Output for this timestep

hₜ = oₜ ⊙ tanh(Cₜ)

Why this fixes vanishing gradients:**

The cell state Cₜ updates via addition (not multiplication), so ∂Cₜ/∂Cₜ₋₁ = fₜ + (1-fₜ)×(other terms). Even if forget gate is small, the additive connection preserves gradients. Gradients can flow unchanged across hundreds of timesteps!

Complete LSTM Implementation with Visualization

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt

class LSTMCell(nn.Module):
    """Single LSTM cell for one timestep.

    Equations:
    f_t = sigmoid(W_f * [h_{t-1}, x_t] + b_f)  — forget gate
    i_t = sigmoid(W_i * [h_{t-1}, x_t] + b_i)  — input gate
    C̃_t = tanh(W_c * [h_{t-1}, x_t] + b_c)     — cell candidate
    C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t            — new cell state
    o_t = sigmoid(W_o * [h_{t-1}, x_t] + b_o)  — output gate
    h_t = o_t ⊙ tanh(C_t)                       — hidden state
    """

    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size

        # Forget gate
        self.W_f = nn.Parameter(torch.randn(hidden_size, input_size + hidden_size))
        self.b_f = nn.Parameter(torch.zeros(hidden_size))

        # Input gate
        self.W_i = nn.Parameter(torch.randn(hidden_size, input_size + hidden_size))
        self.b_i = nn.Parameter(torch.zeros(hidden_size))

        # Cell update
        self.W_c = nn.Parameter(torch.randn(hidden_size, input_size + hidden_size))
        self.b_c = nn.Parameter(torch.zeros(hidden_size))

        # Output gate
        self.W_o = nn.Parameter(torch.randn(hidden_size, input_size + hidden_size))
        self.b_o = nn.Parameter(torch.zeros(hidden_size))

    def forward(self, x_t, h_prev, C_prev):
        """
        Args:
            x_t: [batch_size, input_size] — current input
            h_prev: [batch_size, hidden_size] — previous hidden state
            C_prev: [batch_size, hidden_size] — previous cell state

        Returns:
            h_t: [batch_size, hidden_size] — current hidden state
            C_t: [batch_size, hidden_size] — current cell state
        """
        # Concatenate previous hidden with current input
        combined = torch.cat([h_prev, x_t], dim=1)
        # combined: [batch_size, input_size + hidden_size]

        # Forget gate: which parts of previous cell state to keep
        f_t = torch.sigmoid(combined @ self.W_f.t() + self.b_f)

        # Input gate: which parts of new input to keep
        i_t = torch.sigmoid(combined @ self.W_i.t() + self.b_i)

        # Cell candidate: new information
        C_tilde = torch.tanh(combined @ self.W_c.t() + self.b_c)

        # New cell state: combination of old and new
        C_t = f_t * C_prev + i_t * C_tilde

        # Output gate: which parts of cell state to expose
        o_t = torch.sigmoid(combined @ self.W_o.t() + self.b_o)

        # New hidden state
        h_t = o_t * torch.tanh(C_t)

        return h_t, C_t


class LSTM(nn.Module):
    """Full LSTM for processing a sequence."""

    def __init__(self, input_size, hidden_size, num_layers=1, dropout=0.0):
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        # Create LSTM cells for each layer
        self.cells = nn.ModuleList([
            LSTMCell(input_size if layer == 0 else hidden_size, hidden_size)
            for layer in range(num_layers)
        ])

        self.dropout = nn.Dropout(dropout)

    def forward(self, x, h_prev=None, C_prev=None):
        """
        Args:
            x: [batch_size, seq_length, input_size]
            h_prev: [num_layers, batch_size, hidden_size] or None (init to zeros)
            C_prev: [num_layers, batch_size, hidden_size] or None (init to zeros)

        Returns:
            outputs: [batch_size, seq_length, hidden_size]
            h_final: [num_layers, batch_size, hidden_size]
            C_final: [num_layers, batch_size, hidden_size]
        """
        batch_size, seq_length, _ = x.shape

        # Initialize hidden and cell states
        if h_prev is None:
            h_prev = torch.zeros(self.num_layers, batch_size, self.hidden_size, device=x.device)
        if C_prev is None:
            C_prev = torch.zeros(self.num_layers, batch_size, self.hidden_size, device=x.device)

        outputs = []

        for t in range(seq_length):
            x_t = x[:, t, :]  # [batch_size, input_size]

            # Process through all layers
            for layer in range(self.num_layers):
                h_t, C_t = self.cells[layer](x_t, h_prev[layer], C_prev[layer])
                h_prev = h_prev.clone()
                h_prev[layer] = h_t
                C_prev = C_prev.clone()
                C_prev[layer] = C_t
                x_t = self.dropout(h_t)  # Output of this layer is input to next

            outputs.append(h_t)

        # Stack outputs
        outputs = torch.stack(outputs, dim=1)  # [batch_size, seq_length, hidden_size]

        return outputs, h_prev, C_prev


# Example: Predict next word in a sentence
print("="*70)
print("LSTM EXAMPLE: Language Modeling")
print("="*70)

# Simple dataset: word sequences
vocab_size = 100
embedding_dim = 32
hidden_dim = 64
seq_length = 10

# Random input
batch_size = 4
x = torch.randn(batch_size, seq_length, embedding_dim)

# Create LSTM
lstm = LSTM(input_size=embedding_dim, hidden_size=hidden_dim, num_layers=2, dropout=0.2)
outputs, h_final, C_final = lstm(x)

print(f"Input shape: {x.shape}")  # [4, 10, 32]
print(f"Output shape: {outputs.shape}")  # [4, 10, 64]
print(f"Final hidden state shape: {h_final.shape}")  # [2, 4, 64]

# Use final hidden state for classification
classifier = nn.Linear(hidden_dim, 2)  # Binary classification
logits = classifier(outputs[:, -1, :])  # Use last timestep
print(f"\nClassification logits shape: {logits.shape}")  # [4, 2]

Bidirectional LSTM: Processing Sequences in Both Directions

In many NLP tasks, you need context from both past and future (e.g., "the ___ crossed the river" — without seeing "bank" in context, you don't know if it's a financial bank or riverside). Bidirectional LSTM runs two LSTMs: one forward (left-to-right) and one backward (right-to-left), then concatenates hidden states.

Bidirectional LSTMs work for tasks with full sequence visibility (sentiment analysis, NER, POS tagging) but NOT for generation (where you can only see past tokens).

GRU: A Simpler Alternative

Gated Recurrent Unit (GRU) is a simplified LSTM with 2 gates instead of 3, fewer parameters, and often similar performance. Useful when computational budget is tight or data is limited.

Real-World Application: Hindi Sentiment Analysis

Platforms like BookMyShow and IRCTC classify customer reviews (Hindi and Hinglish mix) as positive/negative. An LSTM approach:

1. Tokenize: "yeh restaurant bahut achcha hai" → [yeh, restaurant, bahut, achcha, hai]
2. Embed: Each word → 300-dim vector (using Hindi embeddings)
3. LSTM: Process sequence, track sentiment shifts ("achcha" = good reverses negativity)
4. Classify: Use final hidden state for binary sentiment classification

Challenge: Code-mixing (Hinglish). Solution: Train on mixed Hindi-English corpus so LSTM learns to handle both seamlessly.

Why Transformers Replaced RNNs

Practice Problems for Advanced Students

1. (Theory): Prove that LSTM's additive cell state update prevents vanishing gradients. Show that ∂Cₜ/∂Cₜ₋₁ ≥ forget_gate, which can be close to 1.

2. (Implementation): Implement LSTM from scratch (like in code above). Train on character-level language modeling (predict next character). Verify that LSTM learns longer dependencies than vanilla RNN.

3. (Analysis): Train bidirectional LSTM on Hindi sentiment classification. Compare with unidirectional. Show that bidirectional achieves higher accuracy.

4. (Visualization): Visualize forget gate, input gate, output gate values over a sequence. Which gates activate for which types of information?

5. (Real-World): Design a bidirectional LSTM for named entity recognition (NER) in Hindi. Label: PERSON, ORG, LOC, O (other). How would you handle code-mixing?

Key Takeaways — Internalize These Concepts

• RNN fundamental: Hidden state hₜ summarizes history; same weights reused across timesteps
• Vanishing gradient catastrophe: Gradients shrink exponentially backward through time, preventing learning of long-range dependencies
• LSTM gates solve this: Forget gate, input gate, cell update, output gate together enable memory of relevant information over hundreds of timesteps
• Cell state as highway: Additive cell update (not multiplicative) allows gradients to flow unchanged across timesteps
• Bidirectional: Process forward and backward, concatenate for tasks with full context
• GRU: Simplified LSTM (2 gates), often similar performance with fewer parameters
• Why Transformers won: Parallelizable, better long-range attention, faster training, dominated industry
• Still relevant: Time series, streaming data, domains where you can't wait for full sequence

Deep Dive: Recurrent Neural Networks and LSTMs

At this level, we stop simplifying and start engaging with the real complexity of Recurrent Neural Networks and LSTMs. In production systems at companies like Flipkart, Razorpay, or Swiggy — all Indian companies processing millions of transactions daily — the concepts in this chapter are not academic exercises. They are engineering decisions that affect system reliability, user experience, and ultimately, business success.

The Indian tech ecosystem is at an inflection point. With initiatives like Digital India and India Stack (Aadhaar, UPI, DigiLocker), the country has built technology infrastructure that is genuinely world-leading. Understanding the technical foundations behind these systems — which is what this chapter covers — positions you to contribute to the next generation of Indian technology innovation.

Whether you are preparing for JEE, GATE, campus placements, or building your own products, the depth of understanding we develop here will serve you well. Let us go beyond surface-level knowledge.

ML Pipeline: From Raw Data to Production Model

At the advanced level, machine learning is not just about algorithms — it is about building robust pipelines that handle real-world messiness. Here is a production-grade ML pipeline pattern used at companies like Flipkart and Razorpay:

# Production ML Pipeline Pattern
import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

def build_ml_pipeline(model, X_train, y_train, X_test):
    """
    A standard ML pipeline with validation.
    Works for classification, regression, or clustering.
    """
    # Step 1: Create pipeline (preprocessing + model)
    pipe = Pipeline([
        ('scaler', StandardScaler()),
        ('model', model)
    ])

    # Step 2: Cross-validation (5-fold) — prevents overfitting
    cv_scores = cross_val_score(pipe, X_train, y_train, cv=5)
    print(f"CV Score: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")

    # Step 3: Train on full training set
    pipe.fit(X_train, y_train)

    # Step 4: Evaluate on held-out test set
    test_score = pipe.score(X_test, y_test)
    print(f"Test Score: {test_score:.4f}")
    return pipe

# Longest Common Subsequence — classic DP problem
# Used in: diff tools, DNA sequence alignment, version control

def lcs(s1, s2):
    m, n = len(s1), len(s2)
    dp = [[0] * (n + 1) for _ in range(m + 1)]

    for i in range(1, m + 1):
        for j in range(1, n + 1):
            if s1[i-1] == s2[j-1]:
                dp[i][j] = dp[i-1][j-1] + 1
            else:
                dp[i][j] = max(dp[i-1][j], dp[i][j-1])

    return dp[m][n]

# Dijkstra's Shortest Path — used by Google Maps!
import heapq

def dijkstra(graph, start):
    dist = {node: float('inf') for node in graph}
    dist[start] = 0
    pq = [(0, start)]  # (distance, node)

    while pq:
        d, u = heapq.heappop(pq)
        if d > dist[u]:
            continue
        for v, weight in graph[u]:
            if dist[u] + weight < dist[v]:
                dist[v] = dist[u] + weight
                heapq.heappush(pq, (dist[v], v))

    return dist

# Real use: Google Maps finding shortest route from
# Connaught Place to India Gate, considering traffic weights