In 2016, a customer support pipeline built for a large Indian food-delivery platform needed to route a Hindi-language complaint, "मेरा ऑर्डर एक घंटे से लेट है और खाना ठंडा आया" ("my order is an hour late and the food arrived cold"), to an English-speaking support queue, in under half a second, at a scale of several million messages a day. The state-of-the-art approach at the time was an encoder-decoder recurrent neural network: an LSTM reads the Hindi sentence one word at a time, compresses everything it has seen into a fixed-size vector, and a second LSTM unrolls that vector back out into English, one word at a time. Two properties of this design were fighting the deployment. First, correctness: by the time the encoder LSTM reaches "ठंडा" (cold), the vector it is carrying has already been overwritten nine or ten times, and the information about "ऑर्डर" (order) at the start of the sentence has been diluted almost to nothing, so long sentences systematically lost their early content, a problem the field called the bottleneck. Second, speed: an LSTM processes a sequence strictly in order, word two cannot be computed until word one is done, so training on a GPU with thousands of parallel cores was using a sliver of the hardware. You could not fix the accuracy problem and the throughput problem by buying more GPUs, because the algorithm itself refused to parallelize.
The bottleneck problem already had a partial fix by 2015: Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio's "Neural Machine Translation by Jointly Learning to Align and Translate" let the decoder look back at every encoder state, not just the final compressed one, weighting each by relevance: the first widely used attention mechanism. It fixed accuracy on long sentences. It did nothing for the parallelization problem, because the LSTM underneath was still sequential. In June 2017, eight researchers, most of them at Google Brain and Google Research, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez (University of Toronto; the work was completed during a Google Brain internship), Łukasz Kaiser, and Illia Polosukhin, posted "Attention Is All You Need" to arXiv, and asked a more radical question: what if you delete the recurrent network entirely, and keep only the attention mechanism? The paper was presented at NeurIPS 2017 in December. The architecture that resulted, the Transformer, is the direct ancestor of BERT, GPT, and every large language model taught in this course. This chapter builds it from the ground up.
The core idea: attention as a lookup, not a summary
An RNN encoder is forced to summarize a sentence into one vector before the decoder can use any of it. Self-attention throws that constraint out: every token is allowed to look directly at every other token in the same sequence and decide, numerically, how much each one matters to it. Concretely, every token's embedding is projected through three separate learned weight matrices into three different vectors:
- Query (Q): "what am I looking for?", computed as
Q = X·WQ - Key (K): "what do I contain, as a label other tokens can match against?", computed as
K = X·WK - Value (V): "what information do I actually hand over if selected?", computed as
V = X·WV
where X is the matrix of token embeddings (one row per token) and WQ, WK, WV are learned matrices shared across all positions. A token's query is compared against every token's key via a dot product; a large dot product means "this key matches what I'm looking for." The resulting scores are turned into a probability distribution with softmax, and that distribution is used to take a weighted average of every token's value. The paper's exact formula, "scaled dot-product attention":
Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · VRead the pieces in order. Q·KT produces an n×n matrix of raw compatibility scores, one row per query token, one column per key token. Dividing by √d_k (the square root of the key dimension) is not cosmetic: it is a variance correction. If each component of Q and K is an independent random value with mean 0 and variance 1, the dot product of two d_k-dimensional vectors has variance d_k (variance of a sum of d_k independent terms adds up). At d_k = 64, the base model's setting, the raw dot products land around magnitude 8, standard deviation. Fed directly into softmax, differences of that size push the exponentials so far apart that softmax saturates: one weight rounds to 1 and the rest to 0, and the gradient with respect to the inputs of a saturated softmax is essentially zero everywhere, so training stalls. Dividing by √d_k rescales the scores back to unit variance regardless of d_k, keeping softmax in a regime where it still has usable gradient. Softmax then turns each row of scores into weights that sum to 1, and the final matrix multiply by V produces, for every token, a new vector that is a weighted blend of every other token's value, with the weights chosen by content, not by fixed position.
Multi-head attention: several lookups, not one
A single attention head has to pack every kind of relationship, subject-verb agreement, coreference, adjacent-word syntax, long-range topical relevance, into one similarity score per token pair. The paper instead runs h independent attention computations in parallel, each with its own, separately learned WQ, WK, WV, each projecting into a smaller dimension d_k = d_v = d_model / h. The base model uses d_model = 512 and h = 8 heads, so each head works in a 64-dimensional space. The h resulting output vectors, one per head, are concatenated back to 512 dimensions and passed through one more learned linear layer, WO. Because the total parameter count and the total compute are held roughly constant, splitting into 8 narrower heads costs almost nothing over one wide head; empirically, different heads learn to specialize: some track adjacent-token syntax, some track long-range coreference, some attend almost uniformly. This is the paper's second core contribution, distinct from attention itself.
The problem attention creates, and how the paper solves it: positional encoding
Self-attention, as just defined, is a set operation: it computes a weighted average over the other tokens using only their content, with no notion of which token came first. Feed it "order was late" and "late was order" and, absent any other signal, self-attention produces the exact same set of pairwise scores: permute the input rows and the output rows permute identically, nothing about the computation itself changes. An RNN never had this problem, because reading tokens in sequence order encodes position for free. Removing the RNN removes that free signal, so the Transformer has to reintroduce it explicitly. The paper's choice is a fixed (non-learned) sinusoidal encoding, added directly to each token's embedding before the first layer:
PE(pos, 2i) = sin( pos / 10000^(2i/d_model) )
PE(pos, 2i+1) = cos( pos / 10000^(2i/d_model) )where pos is the token's position in the sequence and i indexes the embedding dimension. Each dimension pair oscillates at a different frequency, from very fast (low i) to very slow (high i, wavelength near 10000·2π), so the resulting 512-dimensional vector is a unique fingerprint for every position. Because sine and cosine of a sum expand into linear combinations of sine and cosine of the parts, the encoding for position pos+k is always a fixed linear function of the encoding for position pos, which the authors argue makes it easy for the model to learn to attend by relative offset. This is exactly one of two design choices apart from attention itself that make the architecture work at all: the other is the residual connections and layer normalization wrapped around every sub-layer, without which stacking six or more attention blocks would not train stably.
The full architecture
The encoder is a stack of N = 6 identical layers (base model). Each layer has exactly two sub-layers: multi-head self-attention, then a position-wise feed-forward network (two linear layers with a ReLU between them, expanding to d_ff = 2048 and back down to 512). Each sub-layer is wrapped as LayerNorm(x + Sublayer(x)), a residual connection followed by normalization, which is what allows gradients to flow cleanly through a six-layer (or, in modern LLMs, ninety-plus layer) stack. The decoder is also a stack of N = 6 layers, but each decoder layer has three sub-layers: masked multi-head self-attention over the tokens generated so far, cross-attention where the queries come from the decoder but the keys and values come from the encoder's output (this is the direct descendant of Bahdanau's original attention), and the same feed-forward network. The mask in the first decoder sub-layer sets every score for a future position to −∞ before the softmax, forcing each position to attend only to itself and earlier positions. This is necessary because at inference time the model genuinely does not have future tokens yet, and training has to respect that same constraint, or the model would learn to cheat.
Because none of this, self-attention, the feed-forward layers, layer norm, has any sequential dependency across positions within a layer, every token's computation for a given layer can run simultaneously on separate GPU cores. This is the paper's central empirical payoff: the base model trained in about 12 hours on 8 NVIDIA P100 GPUs (100,000 training steps), and the larger model in about 3.5 days on the same hardware, reaching 28.4 BLEU on WMT 2014 English-to-German and 41.8 BLEU on WMT 2014 English-to-French, both new state-of-the-art results at the time, at a fraction of the training cost of the best prior models. Removing recurrence did not just simplify the architecture; it converted an inherently sequential algorithm into an inherently parallel one, which is the property that later let researchers scale these models by throwing more GPUs at bigger data, producing GPT, BERT, and everything downstream of them.
Worked example: attention weights and outputs by hand
Take a 3-token sentence, tokens labelled order, was, late, with toy 2-dimensional embeddings (small on purpose, so every step can be checked by hand) and, to isolate the attention mechanism itself, identity projection matrices, so Q = K = V = X:
x_order = [1, 0]
x_was = [0, 1]
x_late = [1, 1]
d_k = 2, √d_k = 1.41421356Step 1: raw dot-product scores, every query against every key (Q·Kᵀ). For the query token late = [1,1]: score against order = 1×1 + 1×0 = 1; against was = 1×0 + 1×1 = 1; against itself = 1×1 + 1×1 = 2.
Step 2: scale by √d_k = 1.41421356. Scores become [1/1.41421356, 1/1.41421356, 2/1.41421356] = [0.707107, 0.707107, 1.414214].
Step 3: softmax. e0.707107 = 2.028115, e1.414214 = 4.113250, sum = 2.028115 + 2.028115 + 4.113250 = 8.169480. Weights: 2.028115/8.169480 = 0.248255, 2.028115/8.169480 = 0.248255, 4.113250/8.169480 = 0.503490, a valid probability distribution, summing to 1.000000.
Step 4: weighted sum of values: out_late = 0.248255·[1,0] + 0.248255·[0,1] + 0.503490·[1,1] = [0.751745, 0.751745].
So late ends up attending 50.3% to itself, and 24.8% each to order and was, the token with the largest dot product with itself (it shares both active dimensions with itself, [1,1]·[1,1]=2) pulls the most weight, exactly as the mechanism is designed to do: high similarity, high attention. Running the identical four steps for the other two query rows (verified numerically) gives out_order = [0.802224, 0.598888] and out_was = [0.598888, 0.802224]: three genuinely different output vectors from three genuinely different tokens, each a content-weighted blend of all three, computed with zero sequential dependency between rows.
Diagram: one attention head, end to end
The misconception worth correcting explicitly
Students who have heard the paper's title before reading it almost always assume that "Attention Is All You Need" means attention mechanisms were invented in this paper. They were not. Bahdanau, Cho, and Bengio's 2015 attention was already published and already in production translation systems two years earlier; the Transformer paper's own related-work section is explicit that it builds on that line of work. What the title actually claims, and what the paper actually contributes, is that you do not need recurrence or convolution at all once you have attention; that attention alone, stacked with feed-forward layers and positional encoding, is sufficient. The word "all" in the title is doing double duty: it says attention is the only mechanism needed, not that it is a new mechanism. Confusing "attention was invented here" with "recurrence was proven unnecessary here" misses the paper's actual empirical contribution, which is about a resource (training parallelism enabled by removing sequential dependency), not about a new similarity function.
A second, quieter misconception is worth naming while you're here: it is tempting to read a large attention weight as "the model considers this token important" in some general, interpretable sense. All the softmax weight guarantees is that this key's dot product with this particular query, at this particular layer and head, was large relative to the other keys in the same row. Later layers can, and routinely do, redistribute that "importance" completely: a token attended to strongly in layer 2 can be nearly ignored by layer 5. Attention weights are evidence about the mechanism, not a certified explanation of the model's overall reasoning.
Complexity: why removing recurrence actually pays off
For a sequence of length n and representation dimension d, a self-attention layer costs O(n²·d) per layer (computing all pairwise dot products) but is O(1) sequential operations: every pairwise score is independent of every other, so with enough parallel hardware the whole layer runs in constant time regardless of n. A recurrent layer costs O(n·d²) per layer but is O(n) sequential operations, one full step per token, with no way to shortcut that regardless of hardware. For the sentence lengths typical of translation (n well under d), self-attention's per-layer FLOP count is actually competitive with or cheaper than a recurrent layer's, but the sequential-operations column is where the real story is: O(1) versus O(n) is the difference between an operation that finishes in one parallel step on a GPU with thousands of cores and one that takes n strictly sequential steps no matter how many cores you own. There is a second, independent win: in a recurrent network, information from token 1 has to pass through n-1 intermediate hidden states to influence token n, a maximum path length of O(n), with gradient signal weakening at every hop. In self-attention, token 1 and token n are one dot product apart, a maximum path length of O(1), which is the mechanistic reason self-attention handles long-range dependencies, like the sentence-initial "ऑर्डर" governing the sentence-final "ठंडा" from the opening example, far more reliably than recurrence does.
Active recall
Attempt each question before reading its answer.
1. Why does the paper divide by √d_k instead of, say, d_k, or not scaling at all?
2. Using the worked example's numbers, recompute out_late if the scaling step is skipped entirely (divide by 1 instead of √2). What changes, and why does the direction of the change make sense?
3. In the worked example, suppose the embedding for was changes from [0,1] to [0,2], everything else (including Wq=Wk=Wv=I) stays fixed. Trace every row of the score matrix, every softmax, and every output vector that this touches, and explain why token order's attention weights stay identical even though its output vector changes.
4. Why split one 512-dimensional attention head into 8 heads of 64 dimensions each, instead of using one wide head?
5. If self-attention has no built-in sense of word order, why does the Transformer need positional encoding at all: what concretely breaks without it?
6. Why does the decoder's self-attention need a mask during training, given that the full target sentence is already available in the training data?
Answers.
1. If Q and K have independent, roughly unit-variance components, the dot product of two d_k-dimensional vectors has variance d_k (variances of independent terms sum). At d_k = 64, raw dot products swing over a range wide enough to push softmax into a near-one-hot regime, where its gradient is close to zero almost everywhere and learning stalls. Dividing by √d_k, not d_k, exactly cancels the variance growth (variance divided by (√d_k)² = d_k restores unit variance) without over-correcting; dividing by d_k itself would shrink the scores too aggressively and make every attention distribution close to uniform, destroying the model's ability to focus sharply when it needs to.
2. Unscaled, the raw dot products for query late are [1, 1, 2] (unchanged, scaling happens after this step). Softmax of [1,1,2]: e¹=2.718282 (twice), e²=7.389056, sum = 12.825620, weights = [0.211942, 0.211942, 0.576117]. Compare to the scaled weights [0.248255, 0.248255, 0.503490]: removing the scale makes the distribution more peaked on the largest score (57.6% vs 50.3% on itself), because the gap between scores 1 and 2 is proportionally larger than the gap between the scaled scores 0.707 and 1.414 relative to softmax's exponential sensitivity: softmax amplifies larger absolute gaps more sharply. The output becomes out_late = 0.211942·[1,0] + 0.211942·[0,1] + 0.576117·[1,1] = [0.788058, 0.788058], pulled further toward late's own value than the scaled version's [0.751745, 0.751745]. This is the concrete, small-scale version of the saturation problem the earlier "why scale" discussion describes: at realistic d_k = 64 the same effect is drastic enough to stall training, not just shift an output by a few hundredths.
3. With Q=K=V=X, changing was's embedding to [0,2] affects every row and column where token was appears as either query or key, and every output where it appears as a value. Row order (query [1,0]): its dot product with the new was key is 1×0+0×2=0, identical to before: order's first component is 0, so it was never sensitive to was's second component in the first place, whether that component is 1 or 2. Its score row and softmax weights are therefore unchanged: [0.401112, 0.197776, 0.401112]. But its output vector still changes, to [0.802224, 0.796664], because the output is a weighted sum that includes 0.197776×[0,2] instead of 0.197776×[0,1]: the weight on was didn't move, but the value being weighted did. Row was (query now [0,2]) is hit hardest, since both its query and its key changed: raw scores become [0, 4, 2], scaled [0, 2.828427, 1.414214], softmax [0.045388, 0.767918, 0.186694], output [0.232082, 1.722530], a sharp swing toward attending to itself, because doubling its own embedding doubled its self dot-product from 1 to 4. Row late (query [1,1]): its dot product with the new was key becomes 1×0+1×2=2 (up from 1), scaled to 1.414214; its score row becomes [0.707107, 1.414214, 1.414214], softmax [0.197776, 0.401112, 0.401112], output [0.598888, 1.203336], late now attends to was as strongly as to itself, and its output shifts substantially in the second dimension because that's the dimension was's value dominates. The general lesson: an attention weight can be completely insulated from a change (row order's weights) while the output is not, because weights and values are separate channels: always check both before concluding nothing changed.
4. One 512-dimensional head produces exactly one similarity score per token pair, forcing every kind of relationship (short-range syntax, long-range coreference, whatever else the data contains) to be blended into that single number before it's used. Eight 64-dimensional heads compute eight independent similarity scores per pair, each free to specialize, at almost the same total compute and parameter count as one wide head (h heads at d_model/h dimensions each cost about the same as one head at d_model dimensions for the QKᵀ step, and the concatenation plus WO restores the full d_model width). You get representational diversity essentially for free.
5. Self-attention computes a set of pairwise scores based purely on content; permuting the input rows permutes the output rows identically, with no change to any individual score. Without positional encoding, "order was late" and "late was order", or, more dangerously, "the driver hit the pedestrian" and "the pedestrian hit the driver", would produce token representations built from exactly the same multiset of pairwise interactions, because the mechanism has no way to distinguish "which word came first" from "which word came second." Sentence meaning that depends on word order would be structurally invisible to the model.
6. Training uses teacher forcing: the true target sentence is fed to the decoder all at once for parallel training, which is exactly the speed advantage the architecture exists to deliver. But at inference, the model generates one token at a time and genuinely cannot see future tokens; token 5 has to be generated using only tokens 1 through 4. If training let the decoder's self-attention see the whole target sentence at once, unmasked, it would learn to "cheat" by partly copying from future tokens it will never have access to at inference time, and the trained weights would be systematically miscalibrated for the actual generation setting. The mask makes the training-time computation structurally match the inference-time constraint, even though training itself is fully parallel.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where attention is all you need: the paper that changed everything is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting attention is all you need: the paper that changed everything to at least 3 other topics you have studied.