Picture the engineering team behind a food-delivery platform operating at Swiggy or Zomato scale. Every minute, thousands of support chats arrive: "please cancel my order the driver is not moving," "food was cold but not the driver's fault," "refund kab aayega bhai." The system needs to read each message once and output a verdict: cancel-request, refund-complaint, compliment, spam. It never needs to write a reply, a summary, or a single new sentence. It needs one thing done extremely well: absorb the whole message, left context and right context simultaneously, and compress it into a decision.
That requirement is an architecture decision, not just a modeling one. A model that generates text token-by-token, deciding word n+1 before it has ever seen word n+2, is solving a different problem than a model that gets to read the entire input before committing to an answer. BERT (Devlin, Chang, Lee, and Toutanova, 2019, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," NAACL-HLT) was built for exactly the second problem: full-sentence understanding, not generation. This chapter assumes you already know how scaled dot-product and multi-head attention compute their weighted sums (that mechanism is covered in this course's attention-mechanism chapters); the question here is what BERT does with that mechanism — which half of the Transformer it keeps, how it is pretrained, and what the resulting representations are actually used for once training is done.
The architectural fork: encoder, decoder, or both
The original Transformer (Vaswani et al., 2017) is an encoder-decoder pair built for machine translation: an encoder reads the source sentence with unrestricted self-attention, and a decoder generates the target sentence one token at a time, attending back to the encoder's output through cross-attention. Three families of pretrained models later split this pair apart, and the split is a genuine engineering fork, not a naming convention.
An encoder-only model, BERT's family, keeps only the encoder stack and uses unmasked self-attention: every token's query can attend to every key in the sequence, left and right, in a single joint attention computation per layer. There is no causal restriction. This is the right shape when the task is "read everything, output a judgment": classification, tagging, extraction.
A decoder-only model, GPT's family, keeps only the decoder stack, minus the cross-attention sublayer (there is no separate source to attend to), and applies a causal mask: position i's query can only attend to keys at positions ≤ i. This is the right shape for "write the next token given everything so far," which is what open-ended generation and chat actually are.
An encoder-decoder model, T5's and BART's family, keeps both: a bidirectional encoder for the input and a causal decoder for the output, connected by cross-attention. This is the right shape when the task is a genuine transformation from one sequence to a structurally different one — translation, abstractive summarization — where you need full understanding of the input before generating a differently-shaped output.
There is a real complexity argument underneath this choice, not just a taxonomy. Self-attention's dominant cost per layer is computing the n × n attention score matrix, which is Θ(n²·d) for sequence length n and hidden size d, plus Θ(n·d²) for the query/key/value and output projections. Causal masking does not change this: the score matrix is still computed at full size during training, and the mask simply zeroes out (via −∞ before softmax) the disallowed upper-triangular entries. Masking buys nothing in training-time FLOPs. Its payoff shows up only at inference for a decoder generating autoregressively: because position i never depended on positions i+1..n, the keys and values already computed for positions 1..i can be cached, so generating token i+1 costs Θ(i·d) instead of recomputing the whole Θ(n²·d) matrix — the well-known KV-cache optimization. BERT gets no such benefit and needs none: it is never used to generate a sequence one token at a time. It processes a fixed input once and produces a fixed output once. Choosing encoder-decoder for the support-chat classifier would mean paying for a full causal decoder stack, and the cross-attention machinery connecting it to the encoder, to solve a problem that never needed decoding in the first place: added parameters, added latency, zero benefit.
Building the input: WordPiece, segments, and learned positions
BERT does not tokenize on whole words. It uses WordPiece, a subword vocabulary of 30,522 tokens for the original bert-base-uncased model, built so that common words stay whole and rare or unseen words decompose into known fragments rather than becoming a single opaque [UNK]. The algorithm used at inference time is a greedy longest-match-first segmentation: for each word, find the longest prefix that exists in the vocabulary, consume it, then repeat on the remainder, marking every fragment after the first with a "##" prefix to signal "this continues the previous piece, do not treat it as a new word start."
Here is that algorithm, fully defined and traceable, run over a toy 5-entry vocabulary and one out-of-vocabulary word:
vocab = {"un", "##afford", "##able", "afford", "##ing"} # toy WordPiece vocabulary
def wordpiece_tokenize(word, vocab):
tokens = []
start = 0
while start < len(word):
end = len(word)
matched = None
while start < end:
piece = word[start:end]
candidate = piece if start == 0 else "##" + piece
if candidate in vocab:
matched = candidate
break
end -= 1
if matched is None:
return ["[UNK]"]
tokens.append(matched)
start = end
return tokens
print(wordpiece_tokenize("unaffordable", vocab))
Trace it by hand. The word is "unaffordable", 12 characters, indices 0–11. The outer loop starts with start = 0. The inner loop tries the longest possible slice first: word[0:12] = "unaffordable", not in the vocabulary; it shrinks end one character at a time until word[0:2] = "un", which is in the vocabulary (matched as-is, since start == 0). First token: "un". Now start = 2. The inner loop again starts from the longest remaining slice: word[2:12] = "affordable", tested as the continuation candidate "##affordable", not in the vocabulary; it shrinks down to word[2:8] = "afford", tested as "##afford", which is in the vocabulary. Second token: "##afford". Now start = 8. Remaining slice word[8:12] = "able", tested as "##able", in the vocabulary. Third token: "##able". start reaches 12, equal to the length, and the outer loop ends. The call prints exactly:
['un', '##afford', '##able']Three WordPiece tokens reconstruct one out-of-vocabulary word, and the "##" marks exactly which fragments are continuations. This vocabulary also contains "afford" (a valid standalone first piece) and "##ing", which you'll need in the active-recall section.
Each WordPiece token id is looked up in a token embedding table, but that alone doesn't reach the encoder. BERT sums three learned vectors per position: the token embedding, a segment embedding (one of exactly two learned vectors, E_A or E_B, marking which of the two input sentences this position belongs to — for a single-sentence input, every position gets E_A), and a position embedding. This last point is a genuine departure from the original Transformer paper, which used fixed sinusoidal position encodings computed from a formula. BERT instead learns a distinct embedding vector for each absolute position up to a maximum sequence length of 512, trained by gradient descent exactly like any other parameter. There is no closed-form generalization beyond position 512 — it is a lookup table, not a function — which is precisely why BERT cannot process sequences longer than 512 tokens without architectural surgery.
Masked language modeling: what actually gets predicted, and why 80/10/10
Because BERT's self-attention is unmasked, "predict the next word" is not a well-posed pretraining task — every position can already see every other position, including the one it would be predicting, making the task trivially solvable by copying. BERT instead uses masked language modeling (MLM): 15% of WordPiece positions in each training sequence are selected, and the model must predict the original token at each selected position using bidirectional context from all the surrounding tokens.
The selected 15% is not simply replaced with a [MASK] token uniformly. The original paper specifies a three-way split applied to the selected positions: 80% of the time, replace the token with the literal [MASK] symbol; 10% of the time, replace it with a random token from the vocabulary; 10% of the time, leave the original token unchanged. This looks unnecessarily fussy until you see the failure mode it prevents. [MASK] is a training-time artifact — it never appears in the text the model sees during fine-tuning or real use. If every masked position were literally replaced with [MASK], the model could learn to only build a rich contextual representation when it detects the [MASK] token, and produce a lazier representation everywhere else, since only [MASK] positions ever get graded. Since the model cannot know in advance which of the selected 15% will be masked, randomized, or left alone, it is forced to maintain a genuinely predictive distributional representation at every token position, not just the visibly masked ones. The random-token 10% additionally forces the model to actually use context to notice something is wrong, rather than assuming every input token is trustworthy.
The loss itself, at each selected position, is ordinary softmax cross-entropy over the 30,522-entry vocabulary. Take a sentence "the reserve bank [MASK] rates," where the masked word was originally "raised." Suppose the model's final linear layer produces these illustrative logits over five candidate tokens (a full 30,522-way softmax is impractical to hand-trace, so this is a reduced but faithful stand-in with the same arithmetic):
import math
logits = {"raised": 2.0, "cut": 1.0, "held": 0.5, "discussed": 0.3, "announced": -1.0}
exp_vals = {w: math.exp(v) for w, v in logits.items()}
Z = sum(exp_vals.values())
probs = {w: e / Z for w, e in exp_vals.items()}
loss = -math.log(probs["raised"])
print(round(Z, 4), round(probs["raised"], 4), round(loss, 4))
Trace the ledger by hand: e2.0 = 7.3891, e1.0 = 2.7183, e0.5 = 1.6487, e0.3 = 1.3499, e−1.0 = 0.3679. Summed, Z = 13.4738. The probability assigned to the correct token, "raised," is 7.3891 / 13.4738 = 0.5484. Cross-entropy loss is −ln(0.5484) = 0.6007 nats. The code above prints exactly 13.4738 0.5484 0.6007. That loss, backpropagated only through the masked position's output vector and the shared parameters that produced it, is what pushes the encoder's weights toward representations that make "raised" the highest-scoring completion given "the reserve bank ___ rates" from both directions at once.
Next-sentence prediction, and why it didn't survive
BERT pairs MLM with a second pretraining objective, next-sentence prediction (NSP): given two sentences A and B packed into one input as [CLS] A [SEP] B [SEP], with segment embeddings marking which sentence each token belongs to, the model predicts a single binary label from the final hidden vector at the [CLS] position — IsNext if B genuinely followed A in the source corpus, NotNext if B was a sentence sampled at random from elsewhere. Devlin et al. constructed training pairs 50/50 between the two cases, motivated by the idea that many downstream tasks (natural language inference, question-answer pair ranking, duplicate-question detection) hinge on relationships between two sentences, and a pretraining signal for sentence-pair coherence should help.
It turned out to be the weaker half of the recipe. Liu et al. (2019, "RoBERTa: A Robustly Optimized BERT Pretraining Approach") ran controlled ablations and found that removing NSP entirely, and instead training on contiguous blocks of full sentences that could cross document boundaries, matched or improved downstream performance compared to keeping it, once training used dynamic masking (the 80/10/10 pattern recomputed fresh each epoch instead of fixed once during preprocessing), larger batches, and more data. The lesson generalizes past this one paper: not every auxiliary pretraining objective that sounds well-motivated actually earns its keep, and the only way to know is the controlled ablation, not the intuition. The [CLS]/[SEP]/segment-embedding machinery NSP relies on didn't disappear — it is exactly the machinery sentence-pair fine-tuning reuses — but the specific IsNext/NotNext classification loss during pretraining is now widely considered dead weight.
From pretrained weights to a working model: fine-tuning heads
Neither the MLM head nor the NSP head survives past pretraining. Both are thrown away, and the only thing that gets reused is the encoder stack itself: 12 layers and 110 million parameters (bert-base), or 24 layers and 340 million parameters (bert-large), producing a contextual 768- or 1024-dimensional vector for every input token. Everything downstream is a small task-specific head bolted onto that stack, trained with a small learning rate on labeled data.
For sentence-level classification (sentiment on a product review, intent on a support chat), the head is the simplest possible: take the final hidden vector at the [CLS] position, run it through one linear layer projecting to the number of classes, softmax, cross-entropy loss. The [CLS] vector is used here specifically because, through self-attention, it has aggregated information from every other position in the sequence across 12 layers — by the final layer it is not "the embedding of an empty token," it is a learned pooled summary of the whole input.
For token-level tagging (named-entity recognition over addresses in a logistics pipeline, part-of-speech tagging), the head instead applies its own linear-plus-softmax classifier independently to every token's final hidden vector, predicting a BIO-scheme tag per position, with no [CLS] pooling involved.
For extractive question answering (an IRCTC-style FAQ bot locating the answer span inside a policy document), the head is more specific still: two new vectors S and E are learned, and for every token position i in the passage, a start-logit is computed as the dot product S·hi and an end-logit as E·hi. Softmax over start-logits gives the most likely answer start position, softmax over end-logits gives the most likely end position, and the span between them is returned as the answer. No new sequence is generated; the model only points at existing tokens.
For sentence-pair tasks (natural language inference, or flagging duplicate support tickets against each other), the input goes back to the two-segment format NSP originally used — [CLS] A [SEP] B [SEP] with segment embeddings distinguishing A from B — and the [CLS] vector again feeds a linear classifier, exactly as in single-sentence classification, just with a differently structured input.
In every case, fine-tuning updates the entire encoder's weights end-to-end (not just the new head), typically for only 2–4 epochs at a learning rate in the range 2×10⁻⁵ to 5×10⁻⁵, per the original paper's recommendations — small compared to the pretraining run, because the encoder already encodes broad linguistic structure and only needs to be nudged toward the specific task. A cheaper alternative, feature-based use, freezes the encoder entirely and trains only a lightweight classifier (logistic regression, a shallow network) on top of its fixed output vectors; it is faster and needs less labeled data, at some cost in downstream accuracy compared to full fine-tuning.
Misconception: "bidirectional" does not mean two separate one-directional passes
Before BERT, ELMo (Peters, Neumann, Iyyer, Gardner, Clark, Lee, and Zettlemoyer, 2018, "Deep contextualized word representations," NAACL-HLT) produced contextual word vectors by training a left-to-right LSTM language model and a separate right-to-left LSTM language model, then concatenating their hidden states at each position. It is easy to describe BERT the same way in your head: "it looks left, it looks right, then it combines the two" — but that is precisely the architecture BERT's authors built against and explicitly distinguish themselves from in the paper's introduction. In ELMo, the forward LSTM at position i never sees anything past position i during its own computation, and the backward LSTM never sees anything before position i during its own computation; they are two independently-optimized models whose outputs are stapled together only at the very end. Each direction's internal representation was built blind to the other direction the whole way through.
BERT's self-attention does not work like that. Within a single layer, a query at position i computes attention weights against every key in the sequence in one joint operation — there is no "forward pass" and "backward pass" to concatenate, because there was never a directional restriction to begin with. Every one of the 12 (or 24) layers lets every position condition on every other position simultaneously, and that joint conditioning compounds across layers, so by the final layer a token's representation has been shaped by deeply interleaved left-and-right context, not by merging two representations that were each computed in isolation from the other side. This is the specific technical content behind "deep bidirectional" in the paper's title, and it is why BERT needed a masking objective at all: true joint bidirectional self-attention makes naive next-token prediction trivial (the answer is already visible), which is a problem ELMo's two separate directional LSTMs never had.
Active recall
Q1. Architecturally, why can a decoder-only model like GPT not simply adopt BERT's masked-language-modeling objective during pretraining?
Q2. Using the vocabulary {"un", "##afford", "##order", "afford", "##ing"} — note "##order" has replaced "##able" — trace wordpiece_tokenize("affording", vocab) by hand. What does it print?
Q3. In the MLM worked example, the logits were {"raised": 2.0, "cut": 1.0, "held": 0.5, "discussed": 0.3, "announced": -1.0}. Suppose training nudges the "announced" logit up to 3.5 while every other logit stays fixed. Recompute the loss for the correct label "raised," and describe every quantity in the softmax computation that changes as a result, not only the loss.
Q4. A team fine-tunes a RoBERTa-style encoder (pretrained without NSP) for duplicate-support-ticket detection, a sentence-pair task. Does the [CLS]/[SEP]/segment-embedding fine-tuning setup still work, given that NSP was never part of pretraining?
Q5. If BERT-base's input sequence length doubles from 256 to 512 tokens, how does the self-attention compute cost per layer scale, and why does a causal decoder's KV-cache trick not apply to BERT regardless of sequence length?
Answers
A1. GPT's self-attention uses a causal mask: at position i, the query can only attend to keys at positions ≤ i. MLM requires predicting a masked token using context from both sides — positions before and after it. A causally masked model at a masked position could never attend to anything after it, so it would have strictly less information than an MLM objective is designed to require; the task and the architecture's attention pattern are structurally incompatible. This is exactly why GPT-family models are pretrained with plain next-token prediction instead, which only ever needs leftward context, matching the causal mask exactly.
A2. Word = "affording", 9 characters, indices 0–8. start=0: longest first-piece match — try word[0:9]="affording" down through decreasing lengths; word[0:6]="afford" matches the vocabulary entry "afford" (allowed as a first piece since start==0). Token 1: "afford", start=6. Remaining word[6:9]="ing", tested as continuation "##ing", which is in the vocabulary. Token 2: "##ing", start=9=len(word), loop ends. Output: ['afford', '##ing']. Note the removal of "##able" from the vocabulary for this question has no effect on this particular trace, since the word never needs it.
A3. New logits: raised=2.0, cut=1.0, held=0.5, discussed=0.3, announced=3.5. e3.5 = 33.1155. New Z = 13.4738 − 0.3679 + 33.1155 = 46.2214. New p(raised) = 7.3891 / 46.2214 = 0.1599. New loss = −ln(0.1599) = 1.8334 nats — roughly triple the original 0.6007, even though the "raised" logit itself never moved. The ripple is not confined to "raised" and "announced": every probability in the softmax is coupled through the shared denominator Z. p(cut) falls from 2.7183/13.4738=0.2018 to 2.7183/46.2214=0.0588; p(held) falls from 0.1224 to 0.0357; p(discussed) falls from 0.1002 to 0.0292. Since the cross-entropy gradient with respect to each logit is (pi − yi), the gradient pressure pushing down on "cut," "held," and "discussed" all shrink even though their own logits were untouched — raising one wrong class's score doesn't just compete with the target, it relaxes the gradient signal on every other non-target class simultaneously, because probability mass is a shared, fixed-sum resource.
A4. Yes. The [CLS] pooling position, the [SEP] separator, and the two segment embeddings (E_A, E_B) are part of BERT's fixed input architecture, independent of which auxiliary losses were used during pretraining. NSP was only ever a pretraining-time classification head sitting on top of that architecture; removing NSP removes the loss term, not the structural ability to pack two segments into one input and read a pooled [CLS] vector out afterward. Fine-tuning for a new sentence-pair task attaches a fresh classification head to [CLS] and trains it from labeled data directly — it never depends on what the pretraining objective happened to be.
A5. Self-attention's score-matrix cost per layer is Θ(n²·d). Doubling n from 256 to 512 roughly quadruples that term (512²/256² = 4), while the Θ(n·d²) projection cost only doubles; for typical BERT hidden sizes the quadratic term dominates at these lengths, so overall per-layer cost roughly quadruples. This is a training-time (and single-pass inference) cost, and it applies to BERT exactly the same way regardless of length, because BERT always processes the entire sequence in one shot — it is never asked to emit tokens one at a time. A causal decoder's KV-cache benefit is specifically about incremental, autoregressive generation, where positions 1..i are frozen and reused as new tokens are appended one by one; BERT has no such mode, since it produces its full output in a single forward pass rather than incrementally, so there is nothing to cache across steps.
Think About It
Think about this: How would you explain bert and transformer encoders: masked language modeling to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where bert and transformer encoders: masked language modeling is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting bert and transformer encoders: masked language modeling to at least 3 other topics you have studied.