GPT Architecture: Autoregressive Generation

The commentary bot that can never take back a word

Picture a live text-generation system that drafts short cricket match updates automatically from a live score feed — the kind of tool a sports app might use to auto-post a one-line update after every over. The system has decided the update will begin "India". The very next thing it must do is commit to one more word — "won", "need", "trail", "collapse" — before it has any idea what word will come after that. It cannot draft the whole sentence in its head and then type it out; it produces exactly one word, glues that word onto what it already wrote, looks at the new, slightly longer sentence, and asks the same question again: given everything written so far, what is the single most probable next word? Repeat until a full stop appears.

That loop — commit to one token, feed your own output back in as input, repeat — is the entire mechanical essence of a GPT model at generation time. Every property that makes GPT-style generation behave the way it does (why it can occasionally paint itself into a grammatical corner, why the same prompt can produce different completions on different runs, why longer generations get slower per token) falls directly out of this loop and the one structural constraint that makes it well-defined: the model, at the moment it is choosing token t, is mathematically forbidden from looking at token t+1 or anything beyond it, because those tokens do not exist yet. This chapter makes that loop precise: the probability model behind it, the architectural device that enforces it, and the arithmetic of turning a raw score into an actual chosen word.

From the chain rule to a causal mask

A GPT model defines a probability distribution over an entire sequence of tokens x₁, x₂, …, x_T (a token here is a sub-word unit from a fixed vocabulary, produced by byte-pair encoding — a detail covered in the NLP tokenization chapter). Directly modelling the joint distribution P(x₁, x₂, …, x_T) over every possible sequence in one shot is intractable — the number of possible sequences grows exponentially with vocabulary size and length. GPT sidesteps this using the chain rule of probability, which is exact for any joint distribution, not an approximation:

P(x₁, x₂, …, x_T) = P(x₁) · P(x₂ | x₁) · P(x₃ | x₁, x₂) · … · P(x_T | x₁, …, x_{T-1})

Each factor on the right is a distribution over a vocabulary of a few tens of thousands of tokens, conditioned on everything before it — a problem a neural network can be trained to approximate directly. A model that factors a sequence this way, predicting each element from the elements that came before it, is called autoregressive — literally "regressing on itself", the same naming convention as an AR(p) model in classical time-series statistics, except GPT's "regression" target is a categorical distribution over discrete tokens produced by a softmax, not a continuous value produced by a weighted sum. The word "regression" in the name is inherited from statistics; GPT is doing per-step classification, not curve-fitting.

The chain rule tells you what to compute at each step: P(x_t | x₁, …, x_{t-1}). It does not yet tell you how a transformer computes it. Recall from the transformer architecture chapter that self-attention lets every position in a sequence build a representation by taking a weighted combination of every other position's value vectors, where the weights come from a softmax over query–key dot products. Used exactly as described there, that mechanism is bidirectional: position 3 freely attends to position 7, even in a sequence where position 7 lies in the "future" relative to position 3. That is precisely what BERT-style encoders do, and it is exactly what GPT's decoder must not do. If position 3's representation were allowed to peek at position 7 during training, the model would learn to predict token 3 by cheating — reading the answer straight off a token that, at generation time, will not exist yet. GPT's decoder block reuses the identical scaled dot-product attention mechanism with exactly one structural change: before the attention scores are passed through softmax, every score between a query at position i and a key at position j where j > i is overwritten with −∞. Since softmax(−∞) = 0, this guarantees position i's output vector is a weighted combination of value vectors from positions 1 through i only. This is the causal mask, and it is the single architectural fact that turns generic self-attention into an autoregressive language model.

Inside one decoding step

The diagram below shows both halves of the mechanism together: on the left, the causal mask for a five-token sequence, showing exactly which query–key pairs survive and which get zeroed out; on the right, the full loop a GPT model runs once per generated token — encode the sequence so far under that mask, project the final layer's output to a score over the vocabulary, turn those scores into a probability distribution, sample one token, and feed the lengthened sequence back in as the new input.

The left grid is not a generic illustration — it is the exact mask that was in force at every one of the five steps used to generate the sentence "India won the match ." in the worked example below. Row "the" (query position 3) shows ticks under Ind, won, the and crosses under mat and the full stop: at the instant the model chose the word "the", the tokens "match" and "." had not been generated yet, so there was structurally nothing there to attend to. The right-hand loop is what actually executes, once per output token, until an end-of-sequence token is produced or a length limit is hit.

Worked example: generating "India won the match." one token at a time

Take a toy vocabulary of six tokens: {India, won, lost, the, match, .}. Suppose the prompt is just "India", and — after passing "India" through the full decoder stack under the causal mask, and multiplying the final hidden vector by the unembedding matrix — the model outputs these six raw scores (logits), one per vocabulary token:

India: -2.0   won: 2.0   lost: 0.5   the: 1.0   match: -1.0   .: -3.0

These numbers are illustrative — chosen to make the arithmetic land on clean, checkable figures — not the output of an actual trained network, but the softmax-and-sample procedure that turns them into a chosen word is exactly what a real GPT model runs at every step. Softmax converts logits into a probability distribution:

softmax(z_i) = exp(z_i) / Σ_j exp(z_j)

import math

def softmax(logits):
    m = max(logits)                       # subtract max for numerical stability
    exps = [math.exp(x - m) for x in logits]
    total = sum(exps)
    return [e / total for e in exps]

vocab = ["India", "won", "lost", "the", "match", "."]
logits_after_india = [-2.0, 2.0, 0.5, 1.0, -1.0, -3.0]

probs = softmax(logits_after_india)
for token, p in zip(vocab, probs):
    print(f"{token:8s} {p:.4f}")

Tracing it by hand: subtracting the max logit (2.0) gives scaled values [-4.0, 0.0, -1.5, -1.0, -3.0, -5.0], whose exponentials are approximately [0.0183, 1.0000, 0.2231, 0.3679, 0.0498, 0.0067], summing to about 1.6658. Dividing each by that sum gives the printed output:

India    0.0110
won      0.6003
lost     0.1339
the      0.2208
match    0.0299
.        0.0040

Under greedy decoding — always taking the argmax — the model outputs "won" (probability 0.6003), which is also the single most likely continuation under any sampling scheme, since sampling schemes only reweight the distribution; they never change which token has the highest logit. Two other common decoding controls act on this same distribution before a token is drawn:

Temperature divides every logit by a constant T before the softmax, which rescales how sharply the distribution favours the top choice without changing the ranking. At T = 0.5, the scaled logits become [-4.0, 4.0, 1.0, 2.0, -2.0, -6.0]; the exponentials are [0.0003, 1.0000, 0.0498, 0.1353, 0.0025, 0.00005], summing to about 1.1879, giving P(won) ≈ 0.8418 — sharper, more deterministic. At T = 2, the scaled logits are [-1.0, 1.0, 0.25, 0.5, -0.5, -1.5]; the exponentials sum to about 6.849, giving P(won) ≈ 0.3969 and P(the) ≈ 0.2408 — flatter, more room for the model to wander off the single most likely path. T = 1 recovers the original distribution exactly, since dividing by 1 changes nothing.

Top-k filtering keeps only the k highest-probability tokens and renormalises just those. With k = 3 on the T = 1 distribution, the surviving tokens are won (0.6003), the (0.2208) and lost (0.1339), summing to 0.9551; dividing each by that sum gives won ≈ 0.6285, the ≈ 0.2312, lost ≈ 0.1402 — a token like "." (0.0040) is now structurally impossible to sample, no matter how the dice fall.

def top_k_filter(vocab, probs, k):
    ranked = sorted(zip(vocab, probs), key=lambda vp: vp[1], reverse=True)
    top = ranked[:k]
    total = sum(p for _, p in top)
    return [(tok, p / total) for tok, p in top]

for tok, p in top_k_filter(vocab, probs, k=3):
    print(f"{tok:8s} {p:.4f}")
# won      0.6285
# the      0.2312
# lost     0.1402

Top-p (nucleus) sampling instead keeps the smallest set of highest-probability tokens whose cumulative probability first reaches a threshold p. With p = 0.9: won alone gives a cumulative 0.6003 (below 0.9), adding the reaches 0.8211 (still below 0.9), adding lost reaches 0.9551 (crosses 0.9) — so the nucleus is {won, the, lost}, coincidentally the same set top-k(3) produced here. That coincidence is an artifact of this tiny six-token vocabulary; in a real 50,000-token vocabulary the two methods routinely disagree, because top-p's nucleus size adapts to how peaked or flat the distribution is at each step, while top-k's size never does.

Suppose the sampler (any of the schemes above) selects "won". The loop in the diagram now runs again with the lengthened sequence "India won" as input. A fresh forward pass under the causal mask — recomputed over the new, longer sequence — might produce logits strongly favouring "the" (say the = 3.0, match = 1.0, . = -0.5, and the remaining three tokens each at -6.0); softmax gives P(the) ≈ 0.8578. Continuing this same procedure: given "India won the", the model favours "match" (P ≈ 0.9702); given "India won the match", it favours "." (P ≈ 0.9994), completing the sentence. Every one of these four steps runs the identical machinery from the diagram — decoder pass, logits, softmax, sample, append — differing only in what the current sequence is and therefore what logits come out.

One number worth sitting with: multiplying the four chosen-token probabilities together (0.6003 × 0.8578 × 0.9702 × 0.9994 ≈ 0.499) gives the joint probability the model assigned to this exact four-word completion under greedy decoding — essentially a coin flip, even though every individual step looked confident (three of the four steps exceeded 85%). This is a direct, numeric consequence of the chain rule: probabilities multiply, so per-token confidence does not translate into sentence-level confidence, and small per-step uncertainty compounds fast over longer generations.

Training in parallel, generating in sequence

The causal mask has a second job beyond correctness at inference time: it makes training efficient. During training, the full target sequence is already known, so the entire sequence can be pushed through the decoder in a single forward pass — the mask ensures position t's output still only depends on positions 1..t, so all T positions can be scored against their true next tokens simultaneously and in parallel on a GPU, rather than one painstaking step at a time. Feeding the true previous tokens as input at every position, regardless of what the model itself would have predicted, is called teacher forcing.

Generation cannot use teacher forcing, because there is no ground-truth future to feed in — the model must consume its own sampled output as the next input, exactly as the diagram's loop shows. This creates a mismatch called exposure bias: the model was trained conditioned on always-correct history but must generate conditioned on its own, possibly imperfect, history. It also creates a real computational cost. Without any optimisation, producing token t+1 means re-running the decoder over all t previous tokens, and self-attention's cost for a length-t sequence is roughly O(t²) per layer (every position attends to every earlier position). Summed naively across generating T tokens one at a time, total cost grows like O(T³). Production systems avoid most of this using a KV cache: the key and value vectors computed for tokens 1..t are stored, so step t+1 only computes query/key/value for the one new token and attends it against the cached keys and values — an O(t) operation rather than O(t²) — bringing the total cost of generating T tokens down to O(T²). This is why GPT-style generation is described as sequential and comparatively slow per token compared to the fully parallel forward pass used during training, and why the length of the growing context is the dominant cost driver in real deployments. For scale, GPT-2 (small) uses 12 decoder layers, a 768-dimensional hidden state, and a 50,257-token vocabulary; GPT-3 scales this to 96 layers and a 12,288-dimensional hidden state over a 2,048-token context window — the mechanism in the diagram above is identical at every one of those sizes, only the numbers N, d and |V| change.

Misconception: "self-attention sees the whole sentence, so GPT must be planning ahead"

Students who have just learned self-attention from an encoder (BERT-style) chapter often carry over the assumption that any transformer's attention layer lets a token see the entire sequence, future included — so surely a fluent, grammatically coherent GPT completion means the model looked ahead and planned the sentence before typing the first word. It did not, and structurally cannot. The causal mask in the diagram above is applied inside every layer, at every step, which means position i's representation is mathematically a function of positions 1..i only — never of any position beyond it, no matter how many layers are stacked or how large the model is. Re-running the model after generating more tokens does not retroactively let an earlier position "learn" about later ones either: each forward pass reapplies the same mask, so the representation computed for position 3 in a 10-token sequence is identical in form to the representation computed for position 3 back when the sequence was only 3 tokens long, plus whatever information genuinely flowed from positions 1 and 2. Grammatical fluency across a full sentence is not evidence of lookahead; it is evidence that P(x_t | x₁, …, x_{t-1}) alone, learned well enough, is often enough to keep committing to locally sensible continuations that happen to cohere globally. This is also precisely why greedy decoding can go wrong in ways lookahead would have prevented: having committed to a word that seemed best given only the past, the model has no mechanism to reconsider that choice once a better global sentence becomes visible two words later.

Active recall

Attempt these before reading the worked answers below.

1. Why can't a text-generation model simply reuse BERT's fully bidirectional self-attention unchanged?
2. Using the logits table from the worked example, what is P(the) at temperature T = 1? Show the steps.
3. For the same example, is there any temperature T > 0 at which greedy decoding would choose "lost" instead of "won" as the first token? Justify your answer.
4. A deployment generates T = 200 tokens per response without a KV cache, then switches on caching. Describe, in order-of-growth terms, how the total decoding compute changes.
5. True or false: once the model has generated token 5, token 3's internal representation is retroactively updated to include information from token 5. Justify your answer using the causal mask.
6. Using the T = 1 probabilities from the worked example, compute the renormalised top-k distribution for k = 2.

Worked answers

1. Bidirectional attention lets a position attend to tokens after it. During training this would let the model predict token t by directly reading token t's own value out of a later position that attends back to it — the loss would collapse without the model learning anything useful. At inference, tokens after position t do not exist yet, so there is nothing for bidirectional attention to attend to even if it were allowed. The causal mask enforces P(x_t | x_<t) structurally, which is the exact quantity autoregressive generation needs.
2. Logits are [-2.0, 2.0, 0.5, 1.0, -1.0, -3.0] for [India, won, lost, the, match, .]. Subtracting the max (2.0) gives [-4.0, 0.0, -1.5, -1.0, -3.0, -5.0]; exponentials are approximately [0.0183, 1.0000, 0.2231, 0.3679, 0.0498, 0.0067], summing to 1.6658. P(the) = 0.3679 / 1.6658 ≈ 0.2208.
3. No. Temperature divides every logit by the same constant T > 0 before the softmax. Dividing by a positive constant is a monotonic (order-preserving) transformation, so whichever logit was largest before scaling (won, at 2.0) remains the largest after scaling for any T > 0. Temperature changes how peaked or flat the resulting distribution is; it never changes which token is the argmax, so greedy decoding's choice is temperature-invariant.
4. Without caching, each of the T steps re-runs full self-attention over its growing prefix, costing roughly O(t²) at step t; summed over t = 1..T this totals roughly O(T³). With a KV cache, step t only computes attention for the one new token against t cached key/value pairs, an O(t) cost per step, totalling roughly O(T²) across the full generation — asymptotically cheaper, though still growing with the square of sequence length because the cached context itself keeps growing.
5. False. The causal mask is applied identically inside every forward pass: at the moment token 3's representation is computed, only positions 1 through 3 exist in the mask's allowed set, regardless of how many tokens are eventually appended afterward. There is no mechanism in a decoder-only GPT that reaches backward to edit an earlier position's representation once later tokens are produced; each generation step is a fresh forward pass that reapplies the same forward-only mask.
6. The T = 1 probabilities in descending order are won (0.6003), the (0.2208), lost (0.1339), match (0.0299), India (0.0110), . (0.0040). Top-2 keeps won and the, summing to 0.6003 + 0.2208 = 0.8211. Renormalising: P(won) = 0.6003 / 0.8211 ≈ 0.7311, P(the) = 0.2208 / 0.8211 ≈ 0.2689.

Think About It

Think about this: How would you explain gpt architecture: autoregressive generation to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where gpt architecture: autoregressive generation is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting gpt architecture: autoregressive generation to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind gpt architecture: autoregressive generation, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.