GPT Architecture: Autoregressive Language Modeling

Open the keyboard on any Android phone in India — Gboard, SwiftKey, the stock keyboard on a budget device — and type "Please send the payment to". A row of suggested next words appears above the space bar: "him", "her", "account". Tap one. The sentence grows by one word, and the suggestion row instantly updates based on the new, longer sentence. Tap again. Grow again. This loop — look at everything typed so far, propose one next word, append it, repeat — is not a crude approximation of language modeling. It is language modeling, in miniature. GPT is the same loop, run by a much larger network, over much smaller pieces of words, looking at a much longer history. Understanding GPT's architecture is really the answer to one question: how do you build a network that can be trusted to make that single "what comes next" decision, over and over, well enough that the result reads like a person wrote it?

The phone keyboard's version of this decision is shallow — often just a lookup over the last two or three words, sometimes a small on-device model. It has no way to reach back to something said several sentences ago. GPT's version uses self-attention to look at the entire sequence generated so far, weigh every earlier token by relevance, and use that weighted evidence to predict the next one. That difference — a fixed short window versus attention over the whole prefix — is most of what separates a keyboard suggestion from a system that can hold a name mentioned in paragraph one and use it correctly in paragraph five. Everything in this chapter builds toward explaining exactly how.

The chain rule: turning "write a sentence" into "predict one token, forever"

Formally, a piece of text is a sequence of tokens x₁, x₂, …, x_T (a token is a word or, more often in GPT, a sub-word piece — you covered tokenization in the NLP foundations chapter). "Language modeling" means assigning a probability to that whole sequence, P(x₁, …, x_T). Computing that directly — one number for an entire sentence out of every possible sentence — is intractable; the space of sequences is astronomically large. The chain rule of probability rewrites the joint probability as a product of conditionals, and this rewrite is the single most important design decision in GPT's architecture:

P(x₁, x₂, …, x_T) = ∏_t=1^T P(x_t | x₁, …, x_{t−1})

Read the right-hand side literally: the probability of the whole sentence equals the probability of the first token, times the probability of the second token given the first, times the probability of the third given the first two, and so on. No term ever needs to know what comes after it — only what came before. This is exactly the constraint the phone keyboard obeys and exactly the constraint a live text generator must obey: at the moment it commits to a word, the future genuinely does not exist yet. GPT's entire job, at every single step, is to compute one factor of that product: P(x_t | x₁, …, x_{t−1}), a probability distribution over the whole vocabulary for what token comes next. "Generating a paragraph" is just calling that one function repeatedly and feeding each output back in as new input — this is what the word autoregressive means: the model regresses (predicts) using its own prior outputs as inputs.

From tokens to vectors: embedding and position

Before any of this probability estimation can happen, each token id has to become a vector the network can compute with. GPT looks up a learned embedding for every token in a table W_tok (one row per vocabulary entry, typically fifty thousand or more rows), giving a vector W_tok[x_t]. But self-attention, the mechanism at the heart of every block, computes weighted sums over token vectors regardless of their order — swap two tokens and, on their content alone, attention treats the sequence identically. Word order carries most of the meaning in language ("dog bites man" is not "man bites dog"), so GPT adds a second, position-dependent vector W_pos[t] to every token embedding before anything else happens:

E_t = W_tok[x_t] + W_pos[t]

Stack these row vectors for every position and you get a matrix E ∈ ℝ^(seq_len × d_model) — this matrix is what actually enters the decoder stack. (Later GPT-style models often replace this additive table with relative positional schemes such as rotary embeddings, but the purpose is identical: inject order information that attention alone cannot see.)

Inside a decoder block: masked self-attention

GPT is a stack of identical decoder blocks. Each block does two things to its input matrix, in order: it lets every position gather information from earlier positions (self-attention), then it processes each position's gathered information independently (a feed-forward network). The attention step is where the chain-rule constraint from earlier gets physically enforced.

Ordinary self-attention (the kind used in an encoder, as in BERT) lets every position attend to every other position, before and after. GPT cannot allow this — if the token at position 3 were allowed to look at position 5, the model would be "predicting" position 5's word using position 5's own word as evidence, which is cheating during training and meaningless during generation, since position 5 doesn't exist yet. So GPT applies a causal mask: before the softmax that turns attention scores into weights, every score for "position i attending to position j" where j > i is forced to negative infinity, so its softmax weight becomes exactly zero. Position i can only ever draw evidence from positions 1…i.

import torch

def causal_mask(seq_len):
    # True marks a position that must be blocked (future, j > i)
    return torch.triu(torch.ones(seq_len, seq_len), diagonal=1).bool()

mask = causal_mask(3)
print(mask)
# tensor([[False,  True,  True],
#         [False, False,  True],
#         [False, False, False]])

Row i of this matrix is query position i; column j is key position j. Row 0 (the first token) can see only itself — both later columns are True, meaning blocked. Row 2 (the third token) can see all three — nothing after it exists yet, so nothing is blocked. Inside the attention computation, this mask is applied as scores = scores.masked_fill(mask, float('-inf')) right before the softmax, so a blocked position's contribution vanishes to zero probability rather than merely being small.

Feed-forward, residuals, and stacking N blocks

After masked attention produces a context-aware vector for every position, each block adds a residual connection (the attention output is added back to its own input, not used to replace it) and a layer normalization, then passes the result through a position-wise feed-forward network — a two-layer MLP applied independently to every position's vector, typically expanding to four times the model's working dimension before projecting back down: GELU(x·W₁+b₁)·W₂+b₂. Another residual-add-and-norm follows. The residual connections matter architecturally, not just cosmetically: without a direct path for the original signal to skip past each transformation, gradients struggle to reach the earliest blocks in a stack that is dozens of layers deep, and training becomes unstable. GPT-1 used 12 such blocks; GPT-2's largest variant used 48; GPT-3 used 96, with each position's working vector 12,288 numbers wide. Whatever the count, every block takes in a matrix of shape (seq_len, d_model) and returns a matrix of the identical shape — which is exactly what lets you stack them arbitrarily deep and feed one block's output directly into the next.

Worked example: one autoregressive step by hand

Toy numbers make the mechanism checkable rather than just plausible. Take the three-token sequence "Kohli hits six" and, purely to keep the arithmetic visible, use an embedding dimension of 2 and set the query, key, and value projection matrices all equal to the identity — so Q = K = V = E exactly. Suppose, after adding positional encoding, the three token vectors are:

e₁ ("Kohli") = [1, 0]    e₂ ("hits") = [0, 1]    e₃ ("six") = [1, 1]

Step 1 — attention for position 3 ("six"), which may legally see all three positions. Raw scores are dot products scaled by 1/√d_k with d_k = 2, so the scale factor is 1/√2 ≈ 0.7071:

score(3,1) = e₃·e₁ = 1 → scaled 0.7071    score(3,2) = e₃·e₂ = 1 → scaled 0.7071    score(3,3) = e₃·e₃ = 2 → scaled 1.4142

Softmax over [0.7071, 0.7071, 1.4142]: exponentials are 2.0281, 2.0281, 4.1132, summing to 8.1694, giving weights 0.2482, 0.2482, 0.5035. The output for position 3 is the weighted sum of the value vectors: 0.2482·[1,0] + 0.2482·[0,1] + 0.5035·[1,1] = [0.7518, 0.7518].

Step 2 — the same computation for position 2 ("hits"), which may only see positions 1 and 2. Position 3 is masked to −∞ before the softmax:

score(2,1) = e₂·e₁ = 0 → scaled 0    score(2,2) = e₂·e₂ = 1 → scaled 0.7071    score(2,3) = masked → −∞

Softmax over [0, 0.7071, −∞]: exponentials are 1, 2.0281, and exactly 0, summing to 3.0281, giving weights 0.3302, 0.6698, 0. Notice the third weight is not merely small — it is exactly zero, because e^(−∞) = 0. Position 2's output is 0.3302·[1,0] + 0.6698·[0,1] + 0·[1,1] = [0.3302, 0.6698], a vector that has never had any contact with e₃, the embedding of a word that, from position 2's point of view, does not exist yet.

Step 3 — turning position 3's context vector into a next-token prediction. After the feed-forward sub-layer (skipped here for brevity — assume it passes the vector through unchanged for this toy example), the final hidden state is projected through an unembedding matrix onto a toy four-word vocabulary {A, B, C, D}, with rows W_out = A:[1,0], B:[0,1], C:[1,1], D:[−1,−1]. Using h₃ = [0.7518, 0.7518]:

logit_A = 0.7518    logit_B = 0.7518    logit_C = 1.5035    logit_D = −1.5035

Softmax turns these into probabilities: exponentials 2.1209, 2.1209, 4.4974, 0.2223, summing to 8.9615, giving P(A)=0.237, P(B)=0.237, P(C)=0.502, P(D)=0.025. This four-number vector is one full factor of the chain-rule product from earlier — P(x₄ | x₁,x₂,x₃) — computed, not assumed. Greedy decoding would commit to C, the argmax, append it as x₄, and feed the five-token (well, four) sequence back through the same stack to compute P(x₅ | x₁,…,x₄) next.

From logits to text: decoding strategies

The forward pass gives a full probability distribution over the vocabulary at every position; how that distribution becomes a chosen token is a separate decision, applied at generation time only. Greedy decoding always takes the argmax — deterministic, fast, but prone to repetitive, over-safe text because it never risks a slightly-lower-probability word that might lead somewhere more interesting. Temperature sampling divides the logits by a scalar T before the softmax: as T → 0 the distribution sharpens toward the same one-hot argmax as greedy decoding; as T → ∞ it flattens toward uniform, sampling becomes closer to random, and coherence degrades. Top-k and top-p (nucleus) sampling restrict sampling to a small, high-probability subset of the vocabulary before drawing a random choice, trading some of temperature sampling's diversity risk for a guarantee that only plausible tokens are ever candidates.

Whichever strategy is used, the generation loop is the direct, literal implementation of the chain rule: run the whole stack, look only at the last position's logits (everything before it was needed only to compute that final vector, not consumed on its own), pick a token, append it, repeat.

import torch
import torch.nn.functional as F

def generate(model, prompt_ids, max_new_tokens):
    ids = prompt_ids
    for _ in range(max_new_tokens):
        logits = model(ids)                       # forward pass, all positions
        next_logits = logits[:, -1, :]             # only the last position matters
        probs = F.softmax(next_logits, dim=-1)
        next_id = torch.argmax(probs, dim=-1, keepdim=True)  # greedy
        ids = torch.cat([ids, next_id], dim=1)     # feed back in
    return ids

Notice the model is re-run on the entire growing sequence at every iteration — this is why naive autoregressive generation is expensive: producing a 500-token reply costs roughly 500 forward passes, each one slightly longer than the last (production systems cache the key/value vectors from earlier positions instead of recomputing them, but the token-by-token dependency itself is unavoidable — it's what "autoregressive" means).

Evaluating the model: perplexity

During training, GPT is scored by how much probability it assigned to the tokens that actually occurred (teacher forcing: the ground-truth prefix is fed in, not the model's own guesses, so every position's loss can be computed in one parallel pass under the causal mask — the mismatch between this always-correct training regime and inference's self-fed, potentially-erroneous prefix is called exposure bias, and is one reason generation quality can degrade over very long outputs). The standard aggregate metric is perplexity: given the probabilities the model assigned to each actual next token in a sequence, perplexity is the geometric-mean inverse probability, PPL = (∏t 1/P(x_t|x_{<t}))^(1/T).

Suppose a model, generating a 4-token sequence, assigned probabilities 0.4, 0.3, 0.5, and 0.6 to the tokens that were actually chosen. The joint probability is 0.4 × 0.3 × 0.5 × 0.6 = 0.036. Perplexity is (1/0.036)^(1/4) = 27.78^0.25 ≈ 2.30. Read this as: on average, across these four decisions, the model was about as uncertain as if it had to choose uniformly at random among 2.3 equally likely options — far better than guessing among the whole vocabulary, but not perfectly confident either. Lower perplexity is better; a model that always assigned probability 1 to the correct token would score perplexity exactly 1.

Common misconception: "GPT reads the whole sentence before writing the first word"

Because finished GPT output often reads as if it were planned end to end — a setup in the first line pays off in the last — it's tempting to assume the model looked ahead, understood where the sentence was going, and then wrote it in order. It did not, and cannot. The causal mask worked through above is not a training-time convenience that gets relaxed at inference time; it is a permanent structural property of every forward pass. When GPT produces token 40, positions 41 onward do not exist in the computation at all — there is no "future" vector to attend to, masked or otherwise. Every token is the single best next guess given only what precedes it, computed independently of what has not yet been generated.

This is precisely the axis on which GPT differs from BERT, which many students meet first in an NLP introduction: BERT is an encoder, trained to fill in masked tokens using context from both directions simultaneously — for BERT, seeing the whole sentence including what comes after the blank is the entire point. GPT is a decoder, and bidirectional context would break the chain-rule factorization that makes autoregressive generation well defined in the first place: if position 3 could see position 5, then generating position 5 later would require already knowing position 5. The apparent foresight in good GPT output is not lookahead; it is the residual signature of patterns learned from enormous training data, expressed one irreversible, causally-blind step at a time.

Active recall

Attempt each question before reading its answer.

1. Write the chain-rule decomposition of P(x₁, …, x_T) that autoregressive language models rely on, and state in one sentence what each factor corresponds to inside GPT.
2. A model assigns probabilities 0.25, 0.4, and 0.5 to the three tokens actually generated in a sentence. Compute the joint probability of the sentence and the model's perplexity on it.
3. Using the causal-mask code in this chapter, explain why teacher forcing lets GPT be trained with a single parallel forward pass per batch, rather than one sequential step per token.
4. A next-token distribution is {rain: 0.5, sun: 0.3, cloud: 0.2}. What does greedy decoding output? Qualitatively, what happens to this distribution as temperature T → 0 and as T → ∞?
5. Self-attention computes weighted sums over token vectors regardless of their order. Explain why GPT would fail without positional encoding, using a concrete pair of sentences.
6. Redo the chapter's masked-attention calculation for position 2 of "Kohli hits six" independently — with e₁=[1,0], e₂=[0,1], e₃=[1,1] and W_q=W_k=W_v=I — and confirm you get the same output vector as the worked example.

Answers.

1. P(x₁,…,x_T) = ∏t=1T P(x_t | x₁,…,x_{t−1}). Each factor is exactly the probability distribution GPT's final softmax layer outputs at step t, conditioned only on tokens generated up to that point.

2. Joint probability = 0.25 × 0.4 × 0.5 = 0.05. Perplexity = (1/0.05)^(1/3) = 20^(1/3) ≈ 2.71.

3. The causal mask guarantees position i's output depends only on positions ≤ i, which is exactly the dependency structure teacher forcing needs: feeding the entire ground-truth sequence in at once and masking future positions lets the network compute correct, independent predictions for every position in one matrix-multiply pass, instead of needing to wait for position t−1's actual output before starting position t (which is only required at inference time, when the true continuation isn't known in advance).

4. Greedy decoding outputs "rain" (the argmax, 0.5). As T → 0, the softmax sharpens and the distribution collapses toward a one-hot vector on "rain" — sampling becomes equivalent to greedy decoding. As T → ∞, the distribution flattens toward uniform (roughly 1/3 each), increasing randomness and diversity of output at the cost of coherence.

5. Consider "the train delayed the passenger" versus "the passenger delayed the train" — identical multiset of tokens, opposite meaning. Self-attention's dot-product scores depend only on which vectors are present, not on the sequence position they occupy, so without a positional signal added into each token's vector, these two sentences would produce the same set of attention computations per token identity, and the model would have no basis for distinguishing them.

6. Scores: e₂·e₁=0 (scaled 0), e₂·e₂=1 (scaled 0.7071), e₂·e₃ masked to −∞. Softmax over [0, 0.7071, −∞] gives weights 0.3302, 0.6698, 0. Output = 0.3302·[1,0] + 0.6698·[0,1] + 0 = [0.3302, 0.6698] — matching the chapter's worked value.

Think About It

Think about this: How would you explain gpt architecture: autoregressive language modeling to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where gpt architecture: autoregressive language modeling is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting gpt architecture: autoregressive language modeling to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind gpt architecture: autoregressive language modeling, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.