Transformer Architecture: The Engine Behind GPT and BERT

Every year during the Tatkal booking window, IRCTC's servers absorb a spike of a few hundred thousand simultaneous requests in under sixty seconds. Suppose an on-call engineer's incident-summarizer bot is fed this log line and asked to identify what failed: "IRCTC's server crashed during Tatkal booking because it could not handle the traffic." The word "it" has two grammatically valid antecedents in the sentence — "IRCTC" and "server" — and only one is correct. A human resolves this instantly using world knowledge (servers handle traffic; organisations don't, directly). A machine has to resolve it purely from the statistical structure of language, and it has to do so while reading a sentence that might be thirty tokens away from the pronoun, or three hundred. This is the exact problem that forced a rewrite of how neural networks process sequences, and the rewrite is the Transformer — the architecture underneath both GPT and BERT.

Why the previous approach broke down

Before 2017, sequence models were recurrent: an RNN or LSTM read one token at a time, updating a fixed-size hidden state vector, and by the time it reached "it" at the end of the sentence, the information about "server" had to have survived being compressed and overwritten repeatedly across every intermediate token. Two problems fall out of this directly. First, a fixed-size hidden state is an information bottleneck — cramming an arbitrarily long sentence into one vector of, say, 512 numbers means older tokens get diluted, which is exactly the vanishing-gradient problem that makes long-range dependencies like our "it" hard to learn. Second, and just as damaging in practice, recurrence is inherently sequential: you cannot compute the hidden state at position 30 without first computing positions 1 through 29. On a GPU built for parallel arithmetic, this serialises training and throws away almost all of the hardware's throughput. Training a model on the hundreds of billions of tokens that GPT-3 eventually saw would have been computationally infeasible with an RNN backbone at any reasonable training budget.

The Transformer, introduced by Vaswani et al. in "Attention Is All You Need" (2017), removes recurrence entirely. Instead of reading the sentence one token at a time, it lets every token look directly at every other token in a single matrix operation, computes all of those look-ups simultaneously, and stacks several such layers to build up increasingly abstract representations. "It" can attend straight to "server" in one step, regardless of how many words sit between them, and every token's attention computation for a given layer happens in parallel on the GPU. This single design choice is what made both bigger contexts and bigger training runs practical, and it is the mechanism every subsequent large language model, GPT and BERT included, is built from.

Self-attention from first principles: Query, Key, Value

Self-attention gives every token three learned projections of its embedding vector, playing three different roles:

• Query (Q) — what this token is currently looking for in the rest of the sentence.
• Key (K) — what this token advertises about itself, to be matched against other tokens' queries.
• Value (V) — the actual content this token contributes if another token decides to attend to it.

Each is produced by multiplying the input embedding matrix X (one row per token) by a learned weight matrix: Q = XW_Q, K = XW_K, V = XW_V. Relevance between a query and a key is measured by their dot product — large when the two vectors point in a similar direction, near zero when they are unrelated. Stacking every token's query against every token's key gives the matrix QK^T, and a row-wise softmax turns each row into a probability distribution over "how much should this token attend to every other token." The full formula is:

Attention(Q, K, V) = softmax( QK^T / sqrt(d_k) ) V

where d_k is the dimensionality of each query/key vector. The division by sqrt(d_k) is not cosmetic. If the components of q and k are independent with mean 0 and variance 1, their dot product q·k is a sum of d_k independent zero-mean terms, each with variance 1, so the dot product itself has variance d_k and standard deviation sqrt(d_k). As d_k grows, raw dot products grow in magnitude, pushing the softmax into a region where one entry dominates and the gradient with respect to every other entry collapses toward zero — the network stops learning from those comparisons. Dividing by sqrt(d_k) rescales the dot product back to unit variance before the softmax, keeping the gradient signal alive regardless of how large d_k is chosen to be.

Worked example: resolving "it" by hand

To make this mechanical rather than abstract, here is the full computation on a 3-token toy version of the incident sentence, using hand-picked 4-dimensional embeddings and 2-dimensional Q/K/V projections small enough to trace by hand. This is a simplified illustration, not weights from a trained model, but every arithmetic step below was independently computed and verified in code.

import numpy as np

# Toy embeddings, one row per token (d_model = 4)
X = np.array([
    [1.0, 0.0, 1.0, 0.0],   # IRCTC
    [0.0, 2.0, 0.0, 2.0],   # server
    [1.0, 1.0, 1.0, 1.0],   # it
])

Wq = np.array([[1,0],[0,1],[1,0],[0,1]], dtype=float)
Wk = np.array([[1,0],[0,1],[1,0],[0,1]], dtype=float)
Wv = np.array([[0,1],[1,0],[0,1],[1,0]], dtype=float)

def scaled_dot_product_attention(Q, K, V):
    d_k = Q.shape[-1]
    scores = (Q @ K.T) / np.sqrt(d_k)
    scores = scores - scores.max(axis=-1, keepdims=True)  # stability
    weights = np.exp(scores)
    weights = weights / weights.sum(axis=-1, keepdims=True)
    return weights @ V, weights

Q, K, V = X @ Wq, X @ Wk, X @ Wv
output, attn_weights = scaled_dot_product_attention(Q, K, V)

Running the projections first: Q = K = [[2,0],[0,4],[2,2]] and V = [[0,2],[4,0],[2,2]] (rows in order IRCTC, server, it). The raw score matrix QK^T works out to:

          IRCTC  server   it
IRCTC   [   4     0       4  ]
server  [   0    16       8  ]
it      [   4     8       8  ]

Dividing every entry by sqrt(d_k) = sqrt(2) ≈ 1.4142 and applying a row-wise softmax gives the attention-weight matrix:

          IRCTC   server    it
IRCTC   [ 0.4856  0.0287  0.4856 ]
server  [ 0.0000  0.9965  0.0035 ]
it      [ 0.0287  0.4856  0.4856 ]

Read the bottom row: it is the query "it" asking every token in the sentence "how relevant are you to me?" The answer is 48.56% relevant to "server," 48.56% relevant to itself, and only 2.87% relevant to "IRCTC." Multiplying this row against V gives the contextual output vector for "it": 0.0287·[0,2] + 0.4856·[4,0] + 0.4856·[2,2] = [2.914, 1.029]. That output vector is now a blend dominated by "server" and "it" itself, carrying almost none of "IRCTC" — exactly the disambiguation the sentence requires, produced by nothing more exotic than three matrix multiplications and a softmax.

Multi-head attention

A single attention computation forces every token to express all of its relationships — grammatical role, coreference, topical similarity — through one shared Q/K/V geometry. Multi-head attention runs h independent copies of the mechanism above, each with its own learned W_Q, W_K, W_V, typically projecting into a smaller subspace of size d_k = d_model/h so the total compute stays comparable to one full-size head. Each head is free to specialise — one might learn to track subject-verb agreement, another coreference, another local adjacency — and their outputs are concatenated and passed through one more learned matrix W_O to mix them back into a single d_model-dimensional vector per token:

MultiHead(Q, K, V) = Concat(head_1, ..., head_h) W_O
head_i = Attention(X W_Q_i, X W_K_i, X W_V_i)

The original Transformer's base configuration used d_model = 512 with h = 8 heads, giving d_k = 64 per head. GPT-3's largest configuration (Brown et al., 2020) scales this to d_model = 12288 across 96 heads (d_k = 128 per head) and 96 stacked layers, 175 billion parameters in total. BERT-base (Devlin et al., 2018) uses 12 layers, d_model = 768, and 12 heads; BERT-large uses 24 layers, d_model = 1024, and 16 heads. The mechanism scales without changing shape — only the numbers get bigger.

The misconception: attention alone does not know word order

A very natural assumption, once you see that every token can attend to every other token, is that the Transformer must therefore understand sequence order the way an RNN does — after all, RNNs read left to right, so surely a "more powerful" architecture built on top of them keeps that sense of order too. This is wrong, and it is worth proving rather than just asserting. Self-attention as defined above is permutation equivariant: if you shuffle the rows of the input matrix X (i.e., reorder the tokens), the rows of Q and K shuffle identically, QK^T just has its rows and columns permuted the same way, softmax is applied independently per row so it permutes along with them, and the final weighted sum of V permutes in lockstep. Nowhere in Q = XW_Q, K = XW_K, or V = XW_V does a token's row-index in X ever enter the computation. Concretely: with nothing but raw self-attention, "IRCTC's server crashed because it could not handle the traffic" and a scrambled bag of the same words in any other order would produce the exact same set of attention outputs, just relabelled. The mechanism has zero built-in notion of position.

This is precisely why Vaswani et al. add a positional encoding to every token's embedding before the first layer, using fixed sinusoids of different frequency per dimension:

PE(pos, 2i)   = sin( pos / 10000^(2i/d_model) )
PE(pos, 2i+1) = cos( pos / 10000^(2i/d_model) )

and adds this vector directly to the token embedding, so the input to layer one already encodes "I am token 3 of this sequence" as part of its numeric content, before any attention is computed. GPT and BERT both need this step (GPT-style models typically use a simpler learned positional embedding table rather than fixed sinusoids, but the purpose is identical): without it, self-attention would be blind to word order, and "dog bites man" would be computationally indistinguishable from "man bites dog."

Residual connections, layer norm, and the feed-forward sublayer

Multi-head attention is only half of one Transformer block. Its output is added back to its own input via a residual connection and passed through layer normalisation — Add & Norm — which stabilises the scale of activations and gives gradients a direct path backward through however many blocks are stacked (6 in the original paper, 96 in GPT-3), the same anti-vanishing-gradient purpose residual connections serve in deep CNNs. The result then passes through a position-wise feed-forward network applied identically and independently to each token's vector:

FFN(x) = max(0, x W1 + b1) W2 + b2

with an inner dimension d_ff conventionally four times d_model (2048 versus 512 in the base configuration). This sublayer is also wrapped in its own residual connection and layer norm. Unlike the attention sublayer, the FFN does not mix information across token positions at all — every position is transformed by the exact same two matrices independently, so all cross-token reasoning in the block happens strictly inside the attention sublayer that precedes it.

GPT and BERT: identical building block, opposite wiring

Every ingredient above — multi-head self-attention, residual connections, layer norm, position-wise feed-forward — is shared by GPT and BERT. What differs is which half of the original encoder-decoder Transformer each one keeps, which direction attention is permitted to look, and what pretraining task shapes the resulting weights.

BERT is an encoder-only stack: bidirectional self-attention with no restriction, so every token attends freely to tokens both before and after it, exactly as in our worked example above where "it" could look forward and backward across the whole sentence. It is pretrained with masked language modelling — roughly 15% of input tokens are hidden, and the model must predict them using the full bidirectional context on both sides. That objective is only meaningful because BERT can see the whole sentence at once; it is built for understanding tasks — classification, named-entity recognition, extractive question answering — where the complete input is available up front.

GPT is a decoder-only stack, and it enforces a causal mask on top of the identical attention formula: before the softmax, every score where the key position is later than the query position is set to negative infinity, so softmax assigns it exactly zero probability. A query token at position i is mathematically permitted to attend only to keys at positions 1 through i. This is not an incidental restriction; it is required by GPT's pretraining objective of predicting the next token from everything generated so far, and it must hold at inference time too, since the model has literally not generated the later tokens yet when producing token i.

The causal mask's effect is visible directly in our worked example. The unmasked attention-weight matrix computed above is what BERT would use. Applying GPT-style causal masking to the same three tokens changes it token by token: "IRCTC" sits at position 1, so it is only permitted to attend to itself — its distribution collapses from [0.4856, 0.0287, 0.4856] to exactly [1, 0, 0], discarding the very information about "server" that a bidirectional model would keep. "server" at position 2 may attend to itself and "IRCTC" but not "it"; since its original weight on "it" was already only 0.0035, masking it out and renormalising leaves the row almost unchanged, at roughly [0.00001, 0.99999, masked]. "it," sitting last at position 3, is untouched by the mask altogether, since every other token is already at or before it — its row stays exactly [0.0287, 0.4856, 0.4856]. The same three matrix multiplications produce a materially different model depending only on which entries are zeroed out before the softmax, which is the entire architectural distance between an encoder and a decoder.

Diagram: one Transformer block, and how the mask splits GPT from BERT

Active recall

Attempt each question before reading its answer.

1. Why does scaled dot-product attention divide by sqrt(d_k) rather than by d_k itself, or not scaling at all?
2. Suppose the three tokens in the worked example were reordered to [it, server, IRCTC] — same embeddings, same W_Q/W_K/W_V — but positional encoding was switched off. What happens to the numeric attention weight between "it" and "server," and why?
3. In the worked example, BERT-style (unmasked) attention gave "it" a weight of 0.4856 on "server." Under a GPT-style causal mask on the same three-token sequence, does this particular weight change? Justify from the mask's definition, not just by recalling the answer above.
4. The worked example used server's embedding [0, 2, 0, 2]. Suppose that third component is changed from 0 to 2, so server becomes [0, 2, 2, 2], with everything else in the model held fixed. Trace every quantity this changes — not only the "it → server" attention weight.
5. Why can BERT not be used directly for autoregressive text generation the way GPT is, even though both are built from the same self-attention formula?

Answers

1. For query/key vectors with independent, zero-mean, unit-variance components, the dot product q·k is a sum of d_k independent zero-mean terms, so its variance is d_k and its standard deviation is sqrt(d_k). Dividing by d_k would over-correct, shrinking the standard deviation to 1/sqrt(d_k) and making the softmax output nearly uniform regardless of true relevance; not scaling at all leaves the standard deviation growing with sqrt(d_k), which for large d_k (64, 128, or more in real models) saturates the softmax into a near one-hot distribution with vanishing gradients everywhere except the single largest entry. Dividing by exactly sqrt(d_k) is the one scaling that restores unit variance to the dot products before the softmax, whatever d_k is chosen to be.

2. Nothing changes about the relationship itself, only its row/column position in the matrix. Self-attention is permutation equivariant: reordering the input rows reorders Q, K, and V rows identically, which reorders the rows and columns of QK^T the same way, and softmax and the final weighted sum follow along row by row. The numeric weight attached to the "it→server" relationship is still exactly 0.4856 — it just now lives at a different matrix coordinate (row 1, column 2, since "it" is listed first). Without positional encoding, the model has no way to tell this reordered sentence apart from the original one; the attention computation genuinely cannot distinguish "it...server" appearing early in the sequence from appearing late.

3. No change. The causal mask only zeroes out entries where the key's position is later than the query's position (column index > row index). "It" is the third and last token, so every other key (IRCTC at position 1, server at position 2) is at or before it; no entry in its row gets masked. Its attention distribution stays exactly [0.0287, 0.4856, 0.4856]. The rows that do change are the earlier tokens' rows — "IRCTC" (position 1) loses access to both "server" and "it," collapsing to [1, 0, 0].

4. Changing one embedding component ripples through far more than the single number the question names. Because server's embedding feeds Q = XW_Q, K = XW_K, and V = XW_V all from the same row of X, all three of Q_server, K_server, and V_server change: Q_server goes from [0,4] to [2,4], K_server from [0,4] to [2,4], and V_server from [4,0] to [4,2]. K_server is a column in every row's score computation, so the scores IRCTC·server, server·server, and it·server all shift — not just the "it" row. Recomputing exactly: the new attention matrix is [[0.333, 0.333, 0.333], [0.0000, 0.9965, 0.0035], [0.0033, 0.9411, 0.0556]] against the original [[0.4856, 0.0287, 0.4856], [0.0000, 0.9965, 0.0035], [0.0287, 0.4856, 0.4856]]. IRCTC's row changes substantially (it now spreads attention evenly across all three tokens instead of favouring itself and "it"), and "it"'s own row shifts too (0.4856 → 0.9411 on server, at the expense of both other tokens). server's own row of weights is unchanged — a consequence of softmax being invariant to adding the same constant to every entry in a row, which is exactly what happened here since both Q_server and K_server shifted by the same amount along the same axis. But server's row of output still changes, from [3.993, 0.007] to [3.993, 2.000], purely because V_server itself changed while the weights multiplying it did not. A single embedding coordinate therefore touches three projected vectors, three rows of the score matrix, two of three attention distributions, and the output of every row that attends to "server" at all — including the one row whose weights look untouched.

5. Generation is inherently sequential: producing token t+1 requires conditioning only on tokens 1 through t, because tokens after t+1 do not exist yet at that point in decoding. BERT's bidirectional attention lets every token see the entire sequence, including tokens that would not yet exist during generation, so a BERT layer applied at generation time would be looking at information it cannot legitimately have. GPT's causal mask is precisely what makes token-by-token generation well-defined: at every step, the model's attention pattern already matches what will actually be available at inference, because it was trained under that identical restriction from the first pretraining step onward.

Think About It

Think about this: How would you explain transformer architecture: the engine behind gpt and bert to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind transformer architecture: the engine behind gpt and bert, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.