The grievance queue problem
Every large Indian bank runs a text classification system you never see. When a UPI transfer fails silently, a card gets blocked without explanation, or a loan EMI is debited twice, the customer's complaint lands as free-form text — an email, an app ticket, a chatbot transcript — and a downstream team has a regulatory clock running against them: RBI's turnaround-time norms for failed digital payments and grievance redressal give the bank only a few days to acknowledge and route the complaint to the correct desk. A bank like SBI or HDFC receives on the order of tens of thousands of such messages a day. Nobody reads them one by one and picks a folder. A model reads the text and predicts one of a fixed set of labels — UPI Dispute, Card Fraud, Loan Query, KYC Update, Other — and routes accordingly. That prediction step, text in, category out, is text classification, and the model doing the predicting in every serious deployment since roughly 2019 has been some variant of a transformer.
Before transformers, the standard approach was to represent a sentence as a bag of words — a vector counting which words appear, ignoring order — and feed that vector to Naive Bayes or logistic regression. This works surprisingly well for topic detection ("this email mentions 'EMI' and 'overdue', so: Loan Query") but it has a precise, provable blind spot. Consider two complaint fragments built from the exact same three words:
"India defeated Australia" and "Australia defeated India."
As multisets these are identical: {India, Australia, defeated}. A bag-of-words vector is a count over the vocabulary, and counting is commutative — the vector for both sentences is exactly the same, bit for bit, no matter how the counts are computed. Any classifier reading only that vector must output the identical prediction for both sentences, even though they report opposite outcomes. Now translate this into the grievance queue: "UPI failed, amount deducted" and "amount deducted, UPI failed to reverse it" share almost the same bag of words as "UPI amount was deducted but not failed — refund pending," yet one is a live fraud-adjacent case and the other is a routine reversal-in-progress case. A system that cannot use word order, negation position, or which entity is the subject versus the object is structurally blind to exactly the distinctions that matter most for correct routing. This chapter is about the architecture that fixed this — not by adding more hand-built order-sensitive features, but by making every word's representation depend on every other word in the sentence, through a mechanism called self-attention, and then using that mechanism as the backbone of a classifier.
Text classification as a formal problem
Formally, a classifier is a function f: X → Y where X is the space of token sequences and Y is a finite label set (here, the five grievance categories). Given a labelled training set of (text, label) pairs, we parameterize f with weights θ, and for each example compute a probability distribution over Y using a softmax output layer, then minimize cross-entropy loss L = −Σ_c y_c log(p_c) summed over classes c, where y_c is 1 for the true class and 0 otherwise. You already met this exact loss function training a feedforward classifier on numeric features. Nothing about the loss or the training loop changes when the input is text — what changes entirely is how raw text becomes a fixed-size numeric vector that the softmax layer can consume. That conversion step is where bag-of-words, recurrent networks, and transformers diverge, and it is the actual subject of this chapter.
From bag-of-words to contextual embeddings
Recurrent networks (LSTMs) were the first serious fix for order-blindness: process the sentence one token at a time, carry a hidden state forward, and let each word's representation depend on everything seen so far. This solves the India/Australia problem — the hidden state after "India defeated" differs from the hidden state after "Australia defeated" because the subject was read first. But it introduces two costs you can already reason about from asymptotic analysis. First, information from token 1 has to survive n−1 sequential updates to influence token n, and gradients shrink or explode across that many hops — long complaints (a customer explaining a chain of events over several sentences) are exactly where this fails. Second, and just as important for a bank processing lakhs of messages a day, the recurrence is inherently sequential: hidden state h_t cannot be computed until h_{t-1} exists, so an LSTM layer over a sequence of length n takes n sequential steps no matter how many GPU cores you throw at it. Throughput does not scale with hardware.
The transformer, introduced by Vaswani et al. in 2017, replaces recurrence with self-attention: a mechanism that lets every token look at every other token directly, in one parallel computation, regardless of distance. No hidden state is carried step by step. This is the architectural choice that made both BERT-style classifiers and today's large language models practical to train at scale.
Self-attention: the core mechanism
Each input token starts as a vector x_i ∈ ℝ^d (a word embedding). Self-attention builds three different projections of every token using three learned weight matrices, shared across all positions:
Q = X W_Q (queries — "what is this token looking for?")
K = X W_K (keys — "what does this token offer, as a label?")
V = X W_V (values — "what content does this token actually contribute?")
Token i's new representation is a weighted average of every token's value vector, where the weight given to token j is how well token i's query matches token j's key:
Attention(Q, K, V) = softmax( Q Kᵀ / √d_k ) V
The division by √d_k (the dimension of each key vector) is not cosmetic. If the components of q and k are roughly independent with unit variance, their dot product q·k is a sum of d_k such terms and has variance d_k, so its standard deviation grows as √d_k. Feed an unscaled, large-magnitude score into softmax and it saturates — the output collapses to nearly one-hot, gradients through it vanish, and training stalls. Dividing by √d_k restores unit variance regardless of the embedding size, which is why the formula scales with dimension rather than using a fixed constant.
In practice this whole operation is run h times in parallel with independently learned W_Q, W_K, W_V per head — multi-head attention — so one head can learn to track subject–verb relationships while another tracks negation or entity co-reference, and their outputs are concatenated and linearly projected back to dimension d. Because the attention formula only ever multiplies and sums vectors — it has no notion of "first" or "second" — self-attention by itself is permutation-equivariant: if you shuffle the input tokens, the output vectors are exactly the same multiset, just reshuffled to match. This is fixed by adding a fixed or learned positional encoding to each token embedding before the first layer (the original design uses PE(pos,2i) = sin(pos/10000^{2i/d}), PE(pos,2i+1) = cos(pos/10000^{2i/d})), which injects position as content the query/key dot products can respond to.
Worked example: computing self-attention by hand
Take the toy sentence "UPI failed instantly" reduced to 3 tokens with an embedding dimension d = 2 — small enough to trace every multiplication by hand. These embeddings and weight matrices are deliberately simple illustrative numbers chosen so the arithmetic is checkable, not real trained parameters:
x₁ (UPI) = [1, 0], x₂ (failed) = [0, 1], x₃ (instantly) = [1, 1]
W_Q = [[1,0],[0,1]] (identity), W_K = [[0,1],[1,0]] (swap), W_V = [[1,1],[0,1]]
Step 1 — project to Q, K, V. Since W_Q is the identity, Q = X: q₁=[1,0], q₂=[0,1], q₃=[1,1]. W_K swaps a row's two components, so k₁=[0,1], k₂=[1,0], k₃=[1,1]. For W_V, a row [a,b] maps to [a, a+b], giving v₁=[1,1], v₂=[0,1], v₃=[1,2].
Step 2 — score query 1 against every key. q₁·k₁ = 1·0+0·1 = 0; q₁·k₂ = 1·1+0·0 = 1; q₁·k₃ = 1·1+0·1 = 1.
Step 3 — scale by √d_k = √2 ≈ 1.4142. Scores become [0, 0.7071, 0.7071].
Step 4 — softmax. e⁰ = 1, e^0.7071 ≈ 2.0281 (twice). Sum ≈ 5.0563. Weights: [0.1978, 0.4011, 0.4011] — token "UPI" ends up drawing roughly 20% of its new representation from itself and 40% each from "failed" and "instantly."
Step 5 — weighted sum of V. z₁ = 0.1978·[1,1] + 0.4011·[0,1] + 0.4011·[1,2]. First component: 0.1978·1 + 0.4011·0 + 0.4011·1 = 0.5989. Second component: 0.1978·1 + 0.4011·1 + 0.4011·2 = 1.4011. So z₁ ≈ [0.599, 1.401] — this vector is "UPI" no longer in isolation, but blended with context from the rest of the sentence.
The same five steps applied to queries 2 and 3 (left as Active Recall below for query 2) give the complete output matrix. Verifying with code:
import numpy as np
# Toy 3-token sentence, embedding dimension d = 2 (kept tiny so the
# arithmetic can be checked by hand)
X = np.array([
[1, 0], # "UPI"
[0, 1], # "failed"
[1, 1], # "instantly"
], dtype=float)
# Illustrative projection weights (not real trained values)
Wq = np.array([[1, 0], [0, 1]], dtype=float)
Wk = np.array([[0, 1], [1, 0]], dtype=float)
Wv = np.array([[1, 1], [0, 1]], dtype=float)
Q = X @ Wq
K = X @ Wk
V = X @ Wv
d_k = K.shape[1]
scores = (Q @ K.T) / np.sqrt(d_k)
weights = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)
Z = weights @ V
print(np.round(Z, 3))
# Output:
# [[0.599 1.401]
# [0.802 1.401]
# [0.752 1.503]]Row 1 matches the hand-derivation exactly. A real BERT-base encoder runs the identical five steps with d_model = 768 and 12 parallel heads per layer, stacked 12 times — the mechanics never change, only the size.
From token representations to a class label
A classification-tuned transformer (BERT and its relatives) prepends a special [CLS] token to the sequence before the first layer. After N encoder layers — each one self-attention, then residual Add & LayerNorm, then a position-wise feed-forward network, then Add & LayerNorm again — every token has a contextualized vector, including [CLS]. Because [CLS] attends to and is attended by every real token at every layer, its final vector z_CLS is trained during pretraining to summarize the whole sequence. A classification head is just one more linear layer plus softmax on top of it: ŷ = softmax(W_c · z_CLS + b_c). Fine-tuning the grievance router means taking a transformer already pretrained on general text — for Indian-language or code-mixed complaints, models like Google's MuRIL or AI4Bharat's IndicBERT, pretrained specifically on Indian languages, are the practical starting point — and continuing training on a few thousand labelled complaint examples with this classification head attached, using ordinary cross-entropy and backpropagation.
Now the India/Australia example resolves precisely. Self-attention alone is permutation-equivariant: swap two tokens' positions and the set of output vectors is the same set, just relabeled by content, not by position — CLS's attention weights are computed purely from content-based query–key matching, so without positional encoding, [CLS] would attend to "India"-content and "Australia"-content identically regardless of which one came first, producing the literal same z_CLS for "India defeated Australia" and "Australia defeated India." This is not a training artifact to be fixed with more data — it is a structural property of the un-augmented formula. Positional encoding is not a convenience feature; it is the only thing standing between a transformer and the exact bag-of-words blindness this chapter opened with.
A common misconception
Misconception: "A transformer reads a sentence left to right, one word at a time, updating some running memory as it goes — like an LSTM, just with attention bolted on." This is false, and the confusion usually comes from diagrams that draw tokens in a left-to-right row and conflate reading order with computation order.
Correction: There is no running memory and no sequential dependency between positions inside a self-attention layer. Q, K, V for every token are computed independently and simultaneously from the input embeddings; the score matrix QKᵀ is one matrix multiplication covering all n² token pairs at once; softmax and the weighted sum follow in the same parallel step. Position enters only through the additive positional encoding baked into each token's embedding before layer 1 — not through the order in which computation happens, because computation order for all tokens is simultaneous. This is precisely why transformers train faster on GPUs than RNNs of comparable size: the entire sequence's attention layer is one batched matrix operation, not n sequential steps.
Efficiency trade-off
Self-attention's per-layer cost is O(n²·d) — the n×n score matrix, each entry an inner product over d dimensions. An LSTM layer costs O(n·d²) total but as n strictly sequential steps. For a complaint of n = 500 tokens and d = 768 (BERT-base): LSTM ≈ 500 × 768² ≈ 2.95×10⁸ operations, forced sequential; transformer ≈ 500² × 768 ≈ 1.92×10⁸ operations, but every one of the n² pairs is independent and runs in parallel on a GPU. Comparable operation counts, wildly different wall-clock time, because the transformer's work can actually use the hardware's parallelism. The trade-off flips for very long documents: at n = 100{,}000 (a scanned loan-agreement PDF, say), the n² term dominates and attention becomes the bottleneck — which is exactly why sparse and linear-attention variants (Longformer and similar) exist for long-document classification.
Diagram: text classification through a transformer encoder
Active recall
Attempt each question before reading its answer.
1. Why do "India defeated Australia" and "Australia defeated India" get identical bag-of-words vectors? Name one grievance-routing scenario this blind spot would break.
2. Write the scaled dot-product attention formula and explain, in terms of variance, why the √d_k
3. Using the toy example's W_Q, W_K, W_V and embeddings, hand-derive the attention output z₂ for query token "failed" (q₂ = [0,1]), and confirm it matches the code's second output row, [0.802, 1.401].
4. Suppose a fourth token "again," with embedding x₄ = [−1, 1], is appended to the toy sentence (now length 4), using the same W_Q, W_K, W_V. Recompute z₁ for query token "UPI." Does anything besides the new token's own direct contribution change, and why?
5. Why does BERT-style fine-tuning read off the [CLS] token's final vector for classification instead of, say, averaging every token's final vector?
6. For a 500-token complaint processed with d = 768, compare the per-layer operation count order of an LSTM versus a transformer encoder layer, and explain why the transformer is faster in practice despite a similar count.
Worked answers
1. A bag-of-words vector counts word occurrences; counting is commutative, so any two sentences built from the same multiset of words — regardless of order — produce the identical vector. Any classifier reading only that vector must therefore predict the same label for both. In grievance routing this breaks cases distinguished only by argument order or negation position, e.g. "UPI failed, amount not refunded" versus "UPI amount refunded, not failed" — a resolved case and an open dispute sharing nearly the same words.
2. Attention(Q,K,V) = softmax(QKᵀ/√d_k)V. If query and key components are roughly independent with unit variance, the dot product q·k is a sum of d_k such terms, so its variance is d_k and its standard deviation is √d_k. Left unscaled, larger d_k produces larger-magnitude scores, which push softmax into a near-one-hot regime where gradients vanish almost everywhere except at the single largest score. Dividing by √d_k keeps the pre-softmax variance at roughly 1 regardless of the embedding size, keeping softmax in its well-behaved region.
3. q₂ = [0,1]. Scores: q₂·k₁ = [0,1]·[0,1] = 1; q₂·k₂ = [0,1]·[1,0] = 0; q₂·k₃ = [0,1]·[1,1] = 1. Scaled by √2: [0.7071, 0, 0.7071]. Exponentials: [2.0281, 1, 2.0281], sum ≈ 5.0563, weights [0.4011, 0.1978, 0.4011]. Weighted sum of V = {[1,1],[0,1],[1,2]}: first component 0.4011·1+0.1978·0+0.4011·1 = 0.8022; second component 0.4011·1+0.1978·1+0.4011·2 = 1.4011. So z₂ ≈ [0.802, 1.401] — matches.
4. New key and value for token 4: k₄ = x₄W_K = [-1,1] → swap → [1,-1]; v₄ = x₄W_V: first component -1·1+1·0=-1, second component -1·1+1·1=0, so v₄=[-1,0]. q₁ is unchanged (it only depends on x₁). New score: q₁·k₄ = [1,0]·[1,-1] = 1, scaled 0.7071, exponential 2.0281. The softmax denominator is no longer 5.0563 — it now sums over four terms: 1 + 2.0281 + 2.0281 + 2.0281 = 7.0844. New weights: [0.1412, 0.2863, 0.2863, 0.2863]. New z₁: first component 0.1412·1+0.2863·0+0.2863·1+0.2863·(-1) = 0.1412; second component 0.1412·1+0.2863·1+0.2863·2+0.2863·0 = 1.0001. So z₁_new ≈ [0.141, 1.000], a substantial shift from the original [0.599, 1.401] — not merely the addition of a fourth term. The reason: softmax renormalizes over the entire key set for every query, so appending one token changes the shared denominator and therefore reweights every existing token's contribution to every query's output, not only the new token's own score. This is why transformer models fix a maximum sequence length and apply consistent padding/truncation at train and inference time — attention's output for a given position is not independent of how many other tokens are present.
5. [CLS] is inserted specifically so that, across every layer, it attends to and is attended by all real tokens; pretraining objectives (masked-language modelling and next-sentence-style tasks in BERT) push its final vector to already function as a whole-sequence summary before fine-tuning even starts. Reading it off keeps the classifier consistent with the interface the pretrained model was built around. Mean-pooling all token vectors is a legitimate alternative used by some models (Sentence-BERT, for instance), but it is not what a stock BERT checkpoint was pretrained to produce, so it is not the default choice.
6. LSTM: n·d² = 500 × 768² ≈ 2.95×10⁸, but forced into n = 500 strictly sequential steps — step t cannot start before step t−1 finishes. Transformer self-attention: n²·d = 500² × 768 ≈ 1.92×10⁸, similar order of magnitude, but every one of the n² score entries is an independent computation that a GPU can execute concurrently across thousands of cores in one batched matrix multiplication. The transformer wins on wall-clock time not because it does asymptotically less arithmetic here, but because its arithmetic has no sequential dependency to serialize.
Think About It
Think about this: How would you explain text classification and transformer models to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.