Sequence-to-Sequence Models with Attention

Bhashini, India's National Language Translation Mission, has to turn "मैं स्कूल जाता हूँ" into "I go to school." Four Hindi tokens, four different problems packed into one short sentence. Hindi is subject-object-verb: मैं (I, subject), स्कूल (school, object), जाता (go, verb stem), and हूँ, which glues on at the end to carry the first-person-present sense of the verb. English is subject-verb-object, and it fuses that auxiliary meaning straight into the verb form — "go" already carries what हूँ was doing on its own. A system that reads this sentence strictly left to right and forgets each token once it has moved past it cannot produce a correct translation. By the time it needs to decide "school" comes right after "go," it must reach back across the sentence to a word it read three steps earlier. And to decide the verb needs no extra auxiliary word in English, it must connect the very last token it read (हूँ) to the very first output word it produces ("I"). This chapter is about the architecture pattern that makes that kind of reaching-back mechanically possible: a sequence-to-sequence (seq2seq) model with attention.

The encoder-decoder pattern, and where it breaks

A seq2seq model, as introduced by Sutskever, Vinyals and Le in 2014, is two recurrent networks wired end to end. The encoder — an LSTM or GRU, the same recurrent cells from the deep learning chapters — reads the source sentence one token at a time, updating a hidden state h_t at every step. After the last source token it hands a hidden state to the second network. The decoder is a separate recurrent network that starts from that handed-off state and generates the target sentence one token at a time, feeding each token it produces back in as the next input, until it emits an end-of-sequence marker.

The flaw is in what gets handed off. In the original architecture, the only channel between encoder and decoder is a single fixed-size vector — the encoder's last hidden state. It carries the same number of floating-point values whether the source sentence has 4 words or 40. Every fact the decoder will ever need about the source — who did what to whom, which noun each pronoun refers back to, where the sentence's clauses split — has to be squeezed into that one vector before generation even starts. This is the context bottleneck, and it is not a minor inefficiency: Cho et al. (2014) showed that translation quality holds up reasonably on short sentences and degrades sharply as source length grows, exactly the pattern you would expect from compressing more information into a container of unchanging size. A 4-word sentence and a 40-word sentence get handed the same size suitcase; the second one simply cannot fit everything in.

Attention: stop compressing, start pointing

The fix, introduced by Bahdanau, Cho and Bengio in 2014 and simplified by Luong, Pham and Manning in 2015, is almost embarrassingly direct: stop throwing away the encoder's intermediate hidden states. Keep all T of them — h₁, h₂, …, h_T, one per source token — and let the decoder, at every single generation step, decide on the fly which ones matter right now. Concretely, at decoder step t:

1. Score. Compute an alignment score e_t,i between the decoder's current state and every encoder hidden state h_i. Bahdanau's version runs both through a small feed-forward layer before combining them ("additive" attention: e_t,i = v^Ttanh(W_ss_t + W_hh_i)). Luong showed a plain dot product, e_t,i = s_t · h_i, works almost as well and is far cheaper to compute — this "multiplicative" or "dot" form is what most modern systems, including the attention inside Transformers, actually use.
2. Normalize. Pass the T raw scores through softmax across the source positions, producing weights α_t,1, …, α_t,T that are non-negative and sum to 1 — a probability distribution over "where in the source sentence should I be looking right now."
3. Combine. Take the weighted average of the encoder states using these weights: the context vector c_t = Σ_i α_t,i h_i.
4. Generate. Feed c_t together with the decoder's own hidden state into the output layer to produce a distribution over the target vocabulary, and pick (or sample) the next token from it.

The critical detail is that steps 1–3 repeat from scratch at every decoder timestep, using that timestep's own decoder state. Nothing about the weights from step t=1 carries over to step t=2. That is exactly what lets the decoder attend almost entirely to मैं while emitting "I", then shift attention almost entirely to जाता while emitting "go" two steps later — even though जाता appears earlier in the source than स्कूल, which the decoder needs to emit in between.

A worked example — computing one attention step by hand

Real encoder hidden states have hundreds of dimensions and come from trained weights, which makes the arithmetic opaque. Here is the identical computation with toy 3-dimensional vectors, chosen so every step can be checked by hand. Take the encoder outputs for our four Hindi tokens (मैं, स्कूल, जाता, हूँ):

h1 (मैं)  = [ 1,  0,  1]
h2 (स्कूल) = [ 0,  1,  1]
h3 (जाता)  = [ 1,  1,  0]
h4 (हूँ)   = [-1,  0,  1]

and suppose the decoder, having just processed the start token, has produced its first hidden state s₁ = [1, 0, 0]. Using Luong's dot-product form, the alignment score with each encoder state is e_1,i = s₁ · h_i:

e1 = (1)(1) + (0)(0) + (0)(1) =  1
e2 = (1)(0) + (0)(1) + (0)(1) =  0
e3 = (1)(1) + (0)(1) + (0)(0) =  1
e4 = (1)(-1) + (0)(0) + (0)(1) = -1

Run these four numbers through softmax. exp(1) ≈ 2.71828, exp(0) = 1, exp(-1) ≈ 0.36788; the denominator is their sum, 2.71828 + 1 + 2.71828 + 0.36788 = 6.80444:

α1 = 2.71828 / 6.80444 ≈ 0.3995
α2 = 1.00000 / 6.80444 ≈ 0.1470
α3 = 2.71828 / 6.80444 ≈ 0.3995
α4 = 0.36788 / 6.80444 ≈ 0.0541
                          (sum ≈ 1.0001, rounding)

Two sanity checks confirm this before trusting it. First, e1 and e3 were tied at 1, so α1 and α3 must come out equal — they do. Second, softmax is monotonic: the ranking of scores was e1 = e3 (1) > e2 (0) > e4 (−1), and the ranking of weights is α1 = α3 > α2 > α4, which matches. Now form the context vector as the weighted sum c₁ = Σ α_ih_i:

c1_x = 0.3995(1) + 0.1470(0) + 0.3995(1) + 0.0541(-1) ≈ 0.7449
c1_y = 0.3995(0) + 0.1470(1) + 0.3995(1) + 0.0541(0)  ≈ 0.5464
c1_z = 0.3995(1) + 0.1470(1) + 0.3995(0) + 0.0541(1)  ≈ 0.6005

c1 ≈ [0.7449, 0.5464, 0.6005]

That vector, not h₁ alone and not the encoder's final state h₄ alone, is what gets concatenated with s₁ and pushed through the output layer to produce the first English word. In this toy example it leans hardest on मैं and जाता (0.3995 each) and almost ignores हूँ (0.0541) — a plausible pattern for a step that is about to emit "I", since a real trained model would also weight the subject heavily at the first output step. The vectors above were chosen only to keep the arithmetic clean, not fit from real training data, so treat the specific split between मैं and जाता as illustrative rather than a claim about what a trained network would actually learn.

Here is the same computation as code, with the printed values matching the hand trace exactly:

import numpy as np

H = np.array([
    [ 1,  0,  1],   # h1 : मैं
    [ 0,  1,  1],   # h2 : स्कूल
    [ 1,  1,  0],   # h3 : जाता
    [-1,  0,  1],   # h4 : हूँ
], dtype=float)

s1 = np.array([1, 0, 0], dtype=float)   # decoder state, step 1

scores = H @ s1                          # dot-product alignment
alpha  = np.exp(scores) / np.exp(scores).sum()
c1     = alpha @ H                       # weighted sum of encoder states

print("scores:", scores)
print("alpha :", np.round(alpha, 4))
print("c1    :", np.round(c1, 4))

# scores: [ 1.  0.  1. -1.]
# alpha : [0.3995 0.147  0.3995 0.0541]
# c1    : [0.7449 0.5464 0.6005]

The mechanism, drawn out

The misconception this diagram corrects

Students who first meet attention often assume it lets the decoder "look at the original words," the way a human translator glances back at the source text. It does not. The decoder never sees Hindi characters or the source word-embeddings again after the encoder has finished. It only ever sees the encoder's hidden states h₁, …, h_T — and because the encoder is itself a recurrent network, each h_i is already a contextualized summary of token i mixed with everything the encoder had read up to that point, not a raw lookup of the word itself. Attention is a weighted average over these summaries, not a table mapping output tokens to input tokens.

A second, closely related error is assuming the T attention weights are computed once for the whole sentence and reused at every output step, the way a single alignment table would work. They are not. Score computation in step 1 depends on the current decoder state s_t, so a completely fresh set of T weights is recomputed at every decoder timestep. That is exactly what lets the model attend almost entirely to मैं while emitting "I", then shift attention almost entirely to जाता while emitting "go" — a different word, computed at a different internal step, using a freshly recomputed α distribution rather than one fixed at the start.

From here to self-attention

This recurrent attention mechanism is the direct ancestor of the self-attention inside Transformers, which the next chapters build on. The renaming makes the lineage explicit: the decoder's state s_t becomes the query, and the encoder's hidden states h_i serve double duty as both the keys (used to compute the alignment scores) and the values (used to build the weighted sum). A Transformer generalizes exactly this recipe in two ways: it lets every position attend to every other position within the same sequence — self-attention, rather than only decoder-to-encoder — and it removes the recurrence altogether, computing all the queries, keys and values for a whole sequence in parallel instead of one token at a time. The score-softmax-weighted-sum computation you traced by hand above, however, is identical in spirit: it survives unchanged as the scaled dot-product attention at the core of every Transformer layer. Google's production translation system (GNMT, 2016) was itself an LSTM encoder-decoder with exactly this attention mechanism bolted on, before Transformers replaced the recurrent layers entirely a year later — attention, not the recurrence, was the part that stuck.

Active recall

Attempt each question before reading its answer.

1. Why does a plain (attention-free) encoder-decoder degrade on long sentences specifically, rather than performing uniformly badly on all sentence lengths?
2. In the worked example, why must α1 and α3 come out exactly equal without doing any softmax arithmetic?
3. Bahdanau attention is called "additive" and Luong's dot-product form is called "multiplicative." Looking at their score formulas, why do those names fit?
4. If the decoder hidden state at some later step were s₂ = [0, 0, 1] instead of [1, 0, 0], which encoder position would receive the single highest attention weight? (Use the h_i vectors from the worked example.)
5. What does the context vector c_t get combined with before producing the next output token, and why can't the model skip straight from c_t to the output word?
6. A classmate says: "With attention, the encoder doesn't need to be recurrent anymore, since the decoder can just attend to whichever word it needs." What is wrong with this claim about the architecture described in this chapter (as opposed to a full Transformer)?

Answers.

1. Without attention, the entire source sentence must be compressed into one fixed-size vector before decoding starts. That vector's capacity does not grow with sentence length, so a 40-word sentence has to fit into the same number of floats as a 4-word one — information is necessarily discarded, and more of it is discarded as the sentence grows. A uniformly bad model would fail equally at all lengths; this one specifically loses information proportional to how much had to be squeezed in, so degradation tracks length.

2. Softmax weights are a strictly increasing function of the raw scores, and increasing functions preserve equality: whatever score value two inputs share, their outputs after applying the same function must also match. Since e1 = s1·h1 = 1 and e3 = s1·h3 = 1 are numerically identical (both are dot products of s1 = [1,0,0] with a vector whose first coordinate is 1), softmax must assign them identical weights — no computation needed beyond spotting the tie in the scores.

3. Bahdanau's formula v^Ttanh(W_ss_t + W_hh_i) sums two separately-projected vectors before squashing them — an addition is the core operation. Luong's dot product s_t·h_i = Σ_k s_t,kh_i,k is a sum of products, i.e., built from multiplication between corresponding coordinates. The names describe which operation each formula leans on to combine the two vectors.

4. Compute s2·h_i for each: s2·h1 = (0)(1)+(0)(0)+(1)(1) = 1. s2·h2 = (0)(0)+(0)(1)+(1)(1) = 1. s2·h3 = (0)(1)+(0)(1)+(1)(0) = 0. s2·h4 = (0)(-1)+(0)(0)+(1)(1) = 1. Three positions tie at score 1 (h1, h2, h4) and h3 scores 0, so h1, h2 and h4 would share the single highest weight rather than any one of them winning alone — a reminder that "highest weight" can be a tie, not always a unique peak.

5. c_t is concatenated with the decoder's own hidden state s_t before being passed through the output (softmax-over-vocabulary) layer. c_t alone only encodes "what is relevant in the source right now" — it carries no information about what the decoder has already generated in the target sentence, so grammatical target-side context (verb agreement, what pronoun to use next, whether "a" or "the" fits) would be lost if the model skipped straight from c_t to the output word.

6. The claim conflates two independent design choices. In the architecture this chapter covers, attention only replaces the single fixed handoff vector between encoder and decoder — the encoder is still a recurrent network, because that recurrence is exactly what produces the contextualized h_i vectors attention operates over; a non-recurrent encoder here would just hand attention a set of context-free word embeddings. Dropping recurrence from the encoder entirely, and replacing it with attention among the source positions themselves, is a separate and later idea — self-attention, which is what the Transformer architecture does, not the seq2seq-with-attention model described here.

Think About It

Think about this: How would you explain sequence-to-sequence models with attention to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where sequence-to-sequence models with attention is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting sequence-to-sequence models with attention to at least 3 other topics you have studied.