Sequence-to-Sequence Models and Translation

When Translation Is Not a Word Swap

Open Google Translate and type in this English sentence: "Send ₹500 to Ramesh." Ask it to translate the sentence into Hindi. A natural Hindi rendering is "रमेश को ₹500 भेजें" — word by word, that reads "Ramesh-to ₹500 send." Notice what happened: "Send," the very first English word, has jumped to the very end of the Hindi sentence, and "to Ramesh," which closed the English sentence, now opens it. Hindi, like several other Indian languages, typically places the verb at the end of a sentence (a Subject-Object-Verb order), while English places the verb in the middle (Subject-Verb-Object). No dictionary lookup, however good, can fix this by translating one word at a time in place — the entire input sentence has to be read before any word of the output can safely be written, and the output words do not line up position by position with the input words.

This is the problem that motivates sequence-to-sequence (usually shortened to seq2seq) models: given an input sequence of arbitrary length, produce an output sequence of arbitrary, possibly different, length, where the mapping between input and output positions can be reordered, expanded, or compressed. Machine translation is the example this chapter builds around, but the same blueprint shows up anywhere a variable-length input must become a variable-length output:

• Machine translation — an English sentence in, a Hindi, Tamil, or Kannada sentence out.
• Text summarization — a long news article in, a two-line summary out.
• Conversational AI — a user's message in, a chatbot's reply out.
• Speech-to-text — an audio waveform in, a written transcript out.

India's own Bhashini mission — a Government of India initiative to build AI-based translation across the 22 languages listed in the Eighth Schedule of the Constitution — and everyday tools like Google Translate are both built on exactly the ideas developed in this chapter.

The Encoder-Decoder Blueprint

By 2014, researchers already had a good tool for processing sequences: the LSTM (Long Short-Term Memory) cell you met in the previous chapter, which reads a sequence one token at a time while carrying a hidden state forward. The open question was how to connect two different sequences — a source sentence and a target sentence — through a single network. Two papers, published within months of each other in 2014, gave the field its answer. Kyunghyun Cho and his colleagues proposed an RNN encoder-decoder that scored candidate phrase translations inside a traditional statistical translation pipeline; Ilya Sutskever, Oriol Vinyals, and Quoc Le at Google proposed using an RNN encoder-decoder as a complete, standalone translator, with no traditional pipeline at all. Both designs share the same core architecture, now simply called the encoder-decoder model.

The encoder is an LSTM (or a related cell, the GRU, short for Gated Recurrent Unit) that reads the source sentence one token at a time — "Send," then "₹500," then "to," then "Ramesh" — updating its hidden state after each word. After the last token, the encoder keeps only its final hidden state. This vector, often called the context vector (or, more evocatively, the "thought vector"), is meant to be a compressed summary of the entire input sentence's meaning, regardless of whether the sentence had four words or forty.

The decoder is a second LSTM that receives this context vector as its own starting hidden state. It then generates the output sentence one token at a time: it produces a probability distribution over the target vocabulary, picks the most likely (or a sampled) word, feeds that word back in as its own next input, and repeats — a process called autoregressive generation, since each output depends on the outputs generated before it. Generation stops when the decoder produces a special end-of-sequence token. Crucially, in this basic design the decoder never looks at the source sentence directly — every fact it knows about "Send ₹500 to Ramesh" has to survive being squeezed through that one context vector.

import torch
import torch.nn as nn

class Encoder(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)

    def forward(self, source_tokens):
        embedded = self.embedding(source_tokens)          # (batch, src_len, embed_dim)
        outputs, (hidden, cell) = self.lstm(embedded)      # outputs: (batch, src_len, hidden_dim)
        return outputs, hidden, cell


class Decoder(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.fc_out = nn.Linear(hidden_dim, vocab_size)

    def forward(self, input_token, hidden, cell):
        embedded = self.embedding(input_token)              # (batch, 1, embed_dim)
        output, (hidden, cell) = self.lstm(embedded, (hidden, cell))
        prediction = self.fc_out(output.squeeze(1))          # (batch, vocab_size)
        return prediction, hidden, cell

A full training script would still need a start-of-sequence token to kick off the decoder, and padding to batch sentences of different lengths together — both left out here to keep the skeleton readable. Notice, too, that the encoder's outputs tensor — one hidden state per source word — is computed but never used by the decoder above; only the final hidden and cell states are passed on. Keep that unused tensor in mind. The next section is about why discarding it causes trouble, and the section after that is about how attention puts it back to work.

The Bottleneck: One Vector to Remember an Entire Sentence

A four-word sentence like "Send ₹500 to Ramesh" compresses into one context vector without much loss. But what about a thirty-word sentence — an IRCTC refund-policy clause, or a paragraph from a school notice? The context vector stays a fixed size (say, 512 numbers) no matter how long the input sentence is, so every additional word has to fight for space inside that same fixed-size vector, and the words read first are the ones most likely to fade — by the time the encoder reaches word thirty, its hidden state has been overwritten twenty-nine times. This is the bottleneck problem: a single fixed-length vector is being asked to do a job — losslessly summarizing an arbitrarily long sentence — that no fixed-size representation can do perfectly.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio measured the practical cost of this bottleneck directly: they plotted translation quality (BLEU score, which you will meet properly later in this chapter) against sentence length, and found that a basic encoder-decoder translated short sentences reasonably well but got steadily worse on longer ones, while their proposed fix held its quality far more evenly across sentence lengths. That fix, published in 2015, was almost embarrassingly simple in hindsight: stop forcing the decoder to work from memory alone. Instead, let it look back at the source sentence, word by word, every single time it generates an output word.

Attention: Letting the Decoder Look Back

Concretely, attention works like this: instead of discarding the encoder's per-word hidden states, the decoder keeps all of them around — one vector per source word. At every decoding step, before producing the next output word, the decoder effectively asks, "given what I have generated so far, which source words matter most right now?" It answers this by comparing its current hidden state against each encoder hidden state, turning the comparison into a set of weights that sum to 1, and blending the encoder states together using those weights. The result is a fresh context vector, custom-built for this one decoding step, rather than one static vector reused for the whole sentence.

Formally, for encoder hidden states h1, h2, ..., hn and the decoder's current hidden state s, attention computes an alignment score for each source position. Bahdanau's original design used a small feed-forward network to score each pair; Minh-Thang Luong's 2015 follow-up showed that a simple dot product, score(s, hi) = s · hi, works just as well when the encoder and decoder share a hidden size. Either way, the scores are passed through a softmax so they become attention weights a1, ..., an that are all positive and sum to 1, and the step's context vector is the weighted sum c = a1·h1 + a2·h2 + ... + an·hn.

Let's trace this by hand for one word of our running example. Suppose the encoder has read the three-word phrase "UPI payment successful" and produced one small hidden state per word — kept to just two numbers each here, purely so the arithmetic stays visible: h(UPI) = [1.0, 0.0], h(payment) = [0.0, 1.0], and h(successful) = [-1.0, 0.5]. The decoder is about to generate the second Hindi word, "भुगतान" ("payment"), and its current hidden state happens to be s = [0.0, 1.0].

Using dot-product attention, each alignment score is s · hi: against h(UPI) the score is (0.0)(1.0) + (1.0)(0.0) = 0.0; against h(payment) it is (0.0)(0.0) + (1.0)(1.0) = 1.0; against h(successful) it is (0.0)(-1.0) + (1.0)(0.5) = 0.5. The code below runs exactly this calculation end to end, then finishes the two steps still needed — softmax, then the weighted sum — to get from three raw scores to one context vector:

import numpy as np

# Encoder hidden states for "UPI", "payment", "successful" (toy 2-D vectors)
h = np.array([
    [ 1.0, 0.0],   # h(UPI)
    [ 0.0, 1.0],   # h(payment)
    [-1.0, 0.5],   # h(successful)
])

# Decoder hidden state while generating the Hindi word "भुगतान" (payment)
s = np.array([0.0, 1.0])

# Step 1: alignment scores -- dot product of s with each encoder state
scores = h @ s
print(scores)                # [0.  1.  0.5]

# Step 2: softmax turns scores into attention weights that sum to 1
weights = np.exp(scores) / np.exp(scores).sum()
print(np.round(weights, 4))  # [0.1863 0.5065 0.3072]

# Step 3: context vector -- weighted sum of the encoder states
context = weights @ h
print(np.round(context, 4))  # [-0.1209  0.6601]

As expected, "payment" receives the largest share of attention (about 50.6%), since its alignment score was highest — but "successful" still contributes a meaningful 30.7%, and "UPI" contributes a smaller 18.6%. This graded, soft mixing, rather than an all-or-nothing choice, is what "soft attention" means. The resulting context vector is concatenated with the decoder's hidden state and fed into the output layer to help predict "भुगतान." One decoding step later, generating "सफल" ("successful"), the decoder's hidden state will have moved on, the alignment scores will be recalculated from scratch, and attention will most likely shift its weight toward h(successful) instead. Every output word gets its own custom-mixed view of the whole input — which is exactly how a decoder can correctly place a word like "Ramesh" early in a Hindi sentence even though "Ramesh" appeared last in the English one: attention lets it reach directly back to that word without carrying the information forward through several intermediate hidden states.

Training with Teacher Forcing, Translating with Beam Search

During training, the model sees both the source sentence and the correct target sentence, and learns by predicting each target word given the words before it. A subtlety matters here: while generating the third Hindi word, should the decoder be fed the second word it actually predicted — which might well be wrong, especially early in training — or the second word from the correct human translation? Feeding back its own possibly-wrong guesses means one early mistake can throw off every word that follows, making training slow and unstable. The standard fix, called teacher forcing, feeds the correct previous target word as the decoder's input during training, regardless of what the decoder itself predicted a moment ago. It resembles a language teacher who corrects each word immediately rather than waiting until the end of a mispronounced sentence: the student practises producing the next word from a known-good starting point, every single time.

At inference — actually translating a new sentence the model has never seen a correct answer for — there is nothing to fall back on, so the decoder must feed its own output back into itself at every step. The simplest strategy, greedy decoding, always picks the single highest-probability word at each step. It is fast, but short-sighted: a slightly-lower-probability first word might open the door to a much better complete sentence, and greedy decoding can never go back and reconsider a choice it has already made. Beam search hedges against this by tracking the k most promising partial translations (the "beam width") in parallel at every step, instead of committing to just one. At each step, every surviving partial sequence is extended by every possible next word, and only the k highest cumulative-probability sequences survive into the next round. A beam width of 1 is identical to greedy decoding; real translation systems commonly use beam widths in the small single digits up to around ten, trading extra computation for measurably better translations, especially on longer sentences.

To see why a wider beam can win, imagine a toy vocabulary where, immediately after reading the source sentence, the decoder assigns these probabilities to its very first output word: P(A) = 0.6 and P(B) = 0.4. Greedy decoding commits to A immediately, since 0.6 > 0.4. But suppose the best available continuation after A is weak, P(next word | A) = 0.3, giving a best-case full-sequence probability of 0.6 × 0.3 = 0.18. If B had been chosen instead, its best continuation might be strong, P(next word | B) = 0.8, giving 0.4 × 0.8 = 0.32 — a better sentence overall that greedy decoding could never discover, because it discarded B in the very first step. Beam search with width k = 2 keeps both A and B alive after step one, evaluates both continuations, and correctly walks away with the higher-probability B-branch sentence.

How Do We Know a Translation Is Good? BLEU

Once a model can produce translations, how do you score it automatically, across thousands of test sentences, without paying a human translator to judge every single one? The most widely used metric, introduced by Kishore Papineni and colleagues at IBM in 2002, is BLEU (Bilingual Evaluation Understudy). BLEU compares a candidate machine translation against one or more human reference translations by measuring how many overlapping word sequences they share — not just single words, but consecutive pairs, triples, and quadruples of words, called n-grams.

The n-gram idea matters because single-word overlap alone is easy to fool. Suppose the reference translation is "The UPI payment was successful" and a candidate contains exactly the same five words, just in the wrong order: "UPI payment was successful the". Checked word by word (unigram precision), every candidate word appears somewhere in the reference, so unigram precision is a perfect 5/5 = 100% — despite the sentence reading badly. Checked as consecutive pairs (bigram precision), the story changes. The reference's bigrams are (The,UPI), (UPI,payment), (payment,was), and (was,successful); the candidate's bigrams are (UPI,payment), (payment,was), (was,successful), and (successful,the). Three of the candidate's four bigrams appear in the reference list; the fourth, (successful,the), does not. Bigram precision comes out to 3/4 = 75%, correctly flagging that something is wrong even though every word, in isolation, was "correct." BLEU's full score combines precision across several n-gram lengths — typically unigrams through 4-grams — using a geometric mean, together with a brevity penalty that punishes translations that are suspiciously short. (A very short candidate can otherwise rack up artificially high precision just by only ever offering "safe," high-confidence words.) The exact formula is a technical detail; the idea worth remembering is this chapter's theme in miniature — a good translation is not just about using the right words, it is about producing them in the right sequence.

From Recurrence to Attention-Only

The recipe in this chapter — read the whole input with one RNN, generate the output with a second RNN, and use attention to bridge them — dominated machine translation research from around 2015 onward, and dominated real-world translation too after Google adopted a deep, multi-layer LSTM version of it (an eight-layer encoder paired with an eight-layer decoder, a system called GNMT) in production in 2016. But the recurrence itself turned out to be replaceable. In 2017, a team at Google published a paper called "Attention Is All You Need," showing that the recurrent encoder and decoder could each be rebuilt entirely out of attention layers, with no LSTM or GRU anywhere — a design now called the Transformer. You will study the Transformer in detail separately, but its name is not a coincidence: it kept the encoder-decoder blueprint and the attention mechanism from this chapter almost unchanged, and simply removed the recurrence around them. Every large translation and language system you are likely to have used — Google Translate, government platforms like Bhashini, and general-purpose chatbots — descends from the same encoder-decoder-plus-attention blueprint you traced by hand a few paragraphs ago, whether or not recurrence is still in the mix.

Back to the UPI Message

• Encoder-decoder: one RNN reads the whole source sentence into a sequence of hidden states; a second RNN generates the target sentence one word at a time.
• Attention: at every decoding step, the model recomputes which source words matter most right now, instead of relying on one fixed summary vector for the whole sentence.
• Teacher forcing, beam search, and BLEU: the training trick, the decoding trick, and the evaluation metric that make the first two ideas trainable, usable, and measurable.

Return to the phone in your hand. When Google Translate or Bhashini turns "Send ₹500 to Ramesh" into "रमेश को ₹500 भेजें," it is not running a dictionary and reshuffling words with hand-written grammar rules. It is running an encoder that reads your whole sentence into a sequence of hidden states, a decoder that generates the Hindi sentence one word at a time, and an attention mechanism that, for every Hindi word it writes, freshly decides which English words deserve its focus right now. You traced that exact decision by hand for one word of one sentence in this chapter: three alignment scores, one softmax, one weighted sum. Scale that same arithmetic up to a vocabulary of tens of thousands of words, sentences of arbitrary length, and a network trained on hundreds of millions of translated sentence pairs, and you have the sequence-to-sequence architecture — RNN-based here, Transformer-based in its modern form — that quietly translates messages, forms, and notifications for a whole country, every single day.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where sequence-to-sequence models and translation is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting sequence-to-sequence models and translation to at least 3 other topics you have studied.