Attention Mechanisms: Focus in the Noise

The sentence that broke the bottleneck

In 2014, researchers building neural machine translation systems ran into an odd failure. Their encoder-decoder RNNs translated short sentences well, but the moment a source sentence crossed roughly fifteen to twenty words, translation quality fell off a cliff. This was not a training bug. It was architectural. An encoder-decoder RNN reads the entire source sentence word by word and compresses everything it has seen into one fixed-length vector — the final hidden state. The decoder then generates the entire translation using only that single vector as its memory of the source. A five-word sentence and a fifty-word sentence both get squeezed into the same size box. Ask a court stenographer to listen to a two-hour deposition and then summarize the whole thing from memory in one paragraph before writing the transcript, and you get the same problem: the further back the information sits, the more of it gets crushed or lost in the squeeze.

This is exactly the situation an Indian Railways enquiry system faces when a passenger describes a long, meandering complaint before finally stating the actual question buried in the last sentence. If the system reads the whole complaint and compresses it into a single fixed-size summary before deciding on a response, the crucial detail near the end has to compete for space with everything before it, and it frequently loses. The fix that researchers Bahdanau, Cho and Bengio proposed was direct: stop forcing the decoder to work from one compressed vector. Instead, let it look back at every hidden state the encoder produced, at every decoding step, and decide — freshly, each time — which of those hidden states matter most right now. That "deciding which parts matter right now" operation is attention. It replaced a single fixed memory with a dynamic, weighted view over the entire input, and it is the single architectural idea that made everything from modern translation to transformers possible.

From one vector to a weighted mixture

Formally, suppose an encoder has produced hidden states h1, h2, ..., hn for the n tokens of the input. Instead of handing the decoder only hn (the last one), attention constructs, at each decoding step, a context vector c that is a weighted sum of all of them:

c = a1*h1 + a2*h2 + ... + an*hn,  where a1 + a2 + ... + an = 1

Every hi contributes something; the weights ai just control how much. The question is where the weights come from. Bahdanau's original formulation computed an "alignment score" eij between the decoder's current state and each encoder hidden state using a small feed-forward network, then normalised those scores into weights with a softmax:

eij = v^T * tanh(Wa*s(i-1) + Ua*hj)
aij = softmax(eij) over j

This is called additive attention, because the score is built by adding two projected vectors before squashing them through tanh. It works, but it needs an extra learned network just to produce a single number per pair of positions. Two years later, Luong showed that a simple dot product between the decoder state and each encoder state works just as well and is far cheaper to compute: eij = s(i-1) . hj. This is multiplicative attention, and it is the ancestor of the mechanism every transformer uses today, including the Query-Key-Value formulation you are about to work through by hand.

Query, Key, Value: attention as a differentiable lookup

The Query-Key-Value (QKV) framing, introduced by Vaswani et al. in "Attention Is All You Need" (2017), generalises the idea into something closer to a soft dictionary lookup. Every token produces three vectors, each via its own learned linear projection of the token's embedding:

• Query (q) — "what am I looking for?", generated by the token that is currently trying to gather information.
• Key (k) — "what do I contain?", generated by every token that could be attended to, used only for matching.
• Value (v) — "what do I actually offer, once picked?", generated by every token, used only for the payload once a match is found.

In an ordinary dictionary lookup, you compare a query key against stored keys, find the exact match, and return that one value. Attention softens every part of this. Instead of an exact match, it computes a similarity between the query and every key using a dot product. Instead of returning one value, it returns a weighted blend of all values, where the weight is proportional to how well each key matched. For a query vector q and a set of key/value vectors {ki, vi}, the full scaled dot-product attention formula is:

score_i  = (q . ki) / sqrt(dk)
weight_i = softmax(score_i)  =  exp(score_i) / sum_j exp(score_j)
output   = sum_i weight_i * vi

Here dk is the dimensionality of the key vectors. The division by sqrt(dk) is not decoration — it fixes a real numerical problem. If the components of q and k are independent random values with mean 0 and variance 1, then each of the dk products that make up the dot product also has variance 1, and the variance of their sum grows to dk, so the standard deviation of the raw score grows as sqrt(dk). For large dk (transformers commonly use 64 or more per attention head), unscaled dot products swing wildly, softmax saturates towards a near one-hot output, and the gradient with respect to the losing keys shrinks to almost nothing — learning stalls. Dividing every score by sqrt(dk) restores the variance to 1 regardless of dimensionality, keeping softmax in a range where it still has usable gradient everywhere.

Worked example: three tokens, one query

Take the sentence "ISRO launched Chandrayaan" as three tokens. To keep every arithmetic step checkable by hand, use toy 2-dimensional query, key and value vectors for each token — in a real transformer these would come from learned projection matrices, but the mechanism that combines them is identical:

token 1 "ISRO":        q1 = [1, 0]   k1 = [1, 0]   v1 = [1, 2]
token 2 "launched":    q2 = [0, 1]   k2 = [0, 1]   v2 = [3, 0]
token 3 "Chandrayaan": q3 = [1, 1]   k3 = [1, 1]   v3 = [0, 4]

Compute the output vector for token 3, "Chandrayaan", acting as the query — that is, ask what information "Chandrayaan" should gather from the whole sentence, including itself. With dk = 2, so sqrt(dk) = 1.4142:

Step 1 — raw dot-product scores.

q3 . k1 = (1)(1) + (1)(0) = 1
q3 . k2 = (1)(0) + (1)(1) = 1
q3 . k3 = (1)(1) + (1)(1) = 2

Step 2 — scale by 1/sqrt(2).

score1 = 1 / 1.4142 = 0.707
score2 = 1 / 1.4142 = 0.707
score3 = 2 / 1.4142 = 1.414

Step 3 — softmax. Subtract nothing (values are small enough not to need the usual max-subtraction stabiliser, but a real implementation always includes it):

exp(0.707) = 2.028
exp(0.707) = 2.028
exp(1.414) = 4.113
sum = 2.028 + 2.028 + 4.113 = 8.169

weight1 = 2.028 / 8.169 = 0.248
weight2 = 2.028 / 8.169 = 0.248
weight3 = 4.113 / 8.169 = 0.503

Notice the weights sum to 1.000 (0.248 + 0.248 + 0.503), as any softmax output must.

Step 4 — weighted sum of values.

output = 0.248*[1,2] + 0.248*[3,0] + 0.503*[0,4]
       = [0.248, 0.496] + [0.744, 0] + [0, 2.012]
       = [0.993, 2.510]

The output context vector for "Chandrayaan" is [0.993, 2.510]. Half of it (weight 0.503) comes from its own value vector, and the remaining half is split almost exactly evenly between "ISRO" and "launched" (0.248 each) — because both have a raw dot product of exactly 1 with the query, they are equally, and only moderately, relevant. This is the actual mechanism: not a hard choice of one token, but a proportioned mixture of all three.

The same computation in code, verified step by step against the arithmetic above:

import numpy as np

# Query, Key, Value vectors for the 3 tokens: ISRO, launched, Chandrayaan
Q = np.array([[1, 0], [0, 1], [1, 1]], dtype=float)
K = np.array([[1, 0], [0, 1], [1, 1]], dtype=float)
V = np.array([[1, 2], [3, 0], [0, 4]], dtype=float)

d_k = K.shape[1]                     # d_k = 2
scores = (Q @ K.T) / np.sqrt(d_k)    # raw similarity, scaled

def softmax(x):
    e = np.exp(x - np.max(x))
    return e / e.sum()

row = softmax(scores[2])             # attention row for token index 2: "Chandrayaan"
out = row @ V

print(np.round(scores[2], 3))        # [0.707 0.707 1.414]
print(np.round(row, 3))              # [0.248 0.248 0.503]
print(np.round(out, 3))              # [0.993 2.51 ]

Run this mentally line by line: Q @ K.T produces a 3×3 matrix of every query dotted with every key; row index 2 picks out "Chandrayaan" as the query; dividing by np.sqrt(2) applies the scaling; softmax normalises that row into weights; and row @ V performs the weighted sum. The printed values match the hand derivation exactly. The diagram below traces this same computation visually, column by column.

One line of depth beyond the arithmetic: masking

The worked example let "Chandrayaan" attend freely to every token, including the one after it in the sentence, which is fine for an encoder that sees the whole input at once. A decoder generating text one token at a time cannot be allowed to do this: at the moment it is producing the third word, it must not be able to peek at hidden states derived from a fourth word it has not generated yet, or the model would be trivially cheating during training and would fail completely at inference, when future tokens simply do not exist. The fix is a causal mask: before the softmax step, every score for a "future" key is set to negative infinity, so its softmax weight becomes exactly zero without changing the relative weights among the allowed positions. Everything else in the computation — the dot products, the scaling, the weighted sum — is identical to what you just traced by hand.

The misconception that trips almost everyone

The most common misreading of attention is treating it as a hard selection — as if the model looks at the weights, picks the single highest one, and reads off just that one value, the way an array index v[argmax(weights)] would. In the worked example, a student thinking this way would say "Chandrayaan mostly attends to itself, so effectively the output is just v3 = [0, 4]." That is wrong, and the actual output, [0.993, 2.510], is visibly not equal to v3. Attention is a soft, continuous blend: every value vector contributes in proportion to its weight, no matter how small that weight is, and none is ever fully discarded. This distinction is not cosmetic. Because the weighted sum is a smooth, differentiable function of the scores, gradients can flow backward through every single key and value during training, adjusting all of them a little on every step. A genuinely hard selection — picking one token and ignoring the rest, as in the earliest "hard attention" models for image captioning — breaks this chain: choosing an index is not a differentiable operation, so those models had to fall back on reinforcement-learning tricks just to get a training signal through the selection step, which made them slower and harder to train than the soft, weighted-average attention used almost everywhere today.

Active recall

Attempt every question before reading the worked answers below.

1. Given q = [1, 1] and keys k1 = [1, 0], k2 = [0, 1], k3 = [1, 1], which key produces the highest raw dot-product score, and why, without recomputing every value first?
2. Why is the scaling factor 1/sqrt(dk) used instead of 1/dk or no scaling at all?
3. Using the same K and V from the worked example, compute the full attention output for query token 2, "launched" (q2 = [0, 1]). Show every step.
4. True or false: if two tokens have identical key vectors, they must always receive identical attention weight from any given query, even if their value vectors differ. Justify your answer.
5. Why is attention usually described as a "differentiable soft alignment" rather than a lookup table, and what training problem would a true hard lookup create?

Worked answers

1. k3. The dot product q . ki is largest when ki points in a similar direction to q and has large magnitude; k3 = [1, 1] is exactly parallel to q = [1, 1], so its dot product (2) is double that of k1 or k2 (each 1), which are only partially aligned with q.

2. If the components of q and k are independent, mean-zero, unit-variance values, each of the dk terms in the dot product has variance 1, so the dot product itself has variance dk and standard deviation sqrt(dk). Dividing by sqrt(dk) restores unit variance regardless of dimensionality. Dividing by dk instead would over-shrink the scores as dk grows (variance would fall to 1/dk), flattening softmax into a near-uniform distribution and destroying the model's ability to discriminate between keys. No scaling at all leaves variance growing with dk, which for the large head dimensions used in real transformers pushes softmax into a saturated, near one-hot region with vanishing gradients.

3. q2 = [0, 1]. Dot products: q2.k1 = 0, q2.k2 = 1, q2.k3 = 1. Scaled by 1/sqrt(2): [0, 0.707, 0.707]. Exponentials: exp(0) = 1, exp(0.707) = 2.028 (twice); sum = 5.056. Weights: 1/5.056 = 0.198, 2.028/5.056 = 0.401, 2.028/5.056 = 0.401 (sums to 1.000). Output = 0.198*[1,2] + 0.401*[3,0] + 0.401*[0,4] = [0.198+1.203+0, 0.396+0+1.604] = [1.401, 2.000].

4. True. The attention weight for key i depends only on softmax(q . ki / sqrt(dk)), which is a function of q and ki alone. Two tokens sharing the same key vector will always produce the same score against any given query, and therefore the same softmax weight, regardless of what their value vectors contain. The value vectors only affect what gets carried into the output once the (identical) weight is applied to each — they do not affect the weight itself.

5. A lookup table returns a single stored entry for an exact key match, and "which entry was returned" is a discrete decision with no meaningful derivative — you cannot ask how the output changes as the query changes by a tiny amount, because the answer is a step function that jumps between entries. Attention instead computes a smooth weighted average over every value, where the weights themselves are smooth functions (softmax of dot products) of the query and keys. This means the entire operation, including the "which key mattered most" decision, is differentiable, so backpropagation can adjust the query, key and value projections directly from the training loss. A true hard lookup would require some other mechanism, such as sampling and a reinforcement-learning-style gradient estimator, just to get any training signal through the selection step, which is exactly the extra machinery early hard-attention image captioning models needed and modern transformers avoid entirely.

Think About It

Think about this: How would you explain attention mechanisms: focus in the noise to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where attention mechanisms: focus in the noise is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting attention mechanisms: focus in the noise to at least 3 other topics you have studied.