BERT Pre-training: Masked Language Modeling

The exam trick you already know

Every CBSE English paper has a fill-in-the-blanks section. "ISRO successfully _____ Chandrayaan-3 on 14 July 2023." You do not guess this word by scanning left to right and stopping the moment you hit the gap. You read the whole sentence — the subject "ISRO," the object "Chandrayaan-3," the date — and only then commit to "launched." The word after the blank is doing as much work as the words before it. This exercise has a name in reading-pedagogy research: the cloze procedure, introduced by Wilson Taylor in 1953 to measure reading comprehension by deleting words from a passage and asking a reader to restore them using context on both sides.

BERT's pre-training objective, masked language modeling (MLM), is this exact task, industrialized. Take a sentence, hide roughly 15% of its tokens, and train a neural network to recover them using every surrounding token — left and right — simultaneously. That single design decision is the reason BERT exists as a separate architecture from GPT rather than being "the same transformer, different name." Everything in this chapter follows from taking the cloze test seriously as a training signal for a hundred-million-parameter network.

Why a left-to-right predictor cannot give you this

A standard language model — the objective GPT is pre-trained with — predicts token t from tokens 1...t-1 only. This is enforced with a causal attention mask: position i in every transformer layer is architecturally forbidden from attending to any position j > i. It has to be this way, because otherwise the training task collapses. If you removed the causal mask and asked an encoder to predict token t while letting it attend to token t itself, the network would not learn language — it would learn to copy. Even one layer of unrestricted self-attention lets position t route its own identity straight into its own output representation, so "predicting" it becomes trivial lookup, not comprehension. A bidirectional encoder therefore cannot be pre-trained with a next-token objective at all; the objective itself has to change, not just the attention pattern.

MLM sidesteps this by attacking the input rather than the attention mask. Instead of hiding future positions from the computation, it deletes the identity of certain positions from the input entirely, replacing the token with a special [MASK] symbol (or, as you will see, something else) before any layer runs. Now full bidirectional self-attention is safe: position t can look at every other position in the sentence, because it no longer contains a leaked answer. It has to reconstruct the missing word from context alone — the way you did with "launched." This is the core trade Devlin, Chang, Lee, and Toutanova made in the original 2018 BERT paper: give up next-token generation as a training signal, in exchange for a representation that has genuinely seen both directions of context at every layer.

The masked language modeling objective, formally

Given an input sequence of tokens x = (x_1, ..., x_n), choose a random subset of positions M ⊂ {1, ..., n}, corrupt the tokens at those positions to produce x̃, and train the network to maximize the log-likelihood of the original tokens at exactly those positions, conditioned on the corrupted sequence:

L_MLM = − Σ_{i ∈ M} log P(x_i | x̃; θ)

Two details matter and are easy to miss. First, the sum runs only over the masked positions M — roughly 15% of the sequence — not over every token. Unmasked positions contribute nothing to the loss; they exist purely as context the model is allowed to read. This is the opposite of a standard language model, where every position (after the first) contributes a loss term. Second, P(x_i | x̃; θ) is a full softmax distribution over the entire WordPiece vocabulary (30,000 subword types in the original BERT), computed by feeding the final-layer hidden state at position i through a linear layer tied to the input embedding matrix. The network is not choosing between a handful of plausible words; at training time it is scored against all 30,000.

Constructing the input: WordPiece, [CLS]/[SEP], and the embedding sum

Before any masking happens, the raw sentence is broken into WordPiece subword units — "chandrayaan-3" might split into pieces like chandra, ##ya, ##an, -, 3, so that rare proper nouns never force an out-of-vocabulary token. A [CLS] token is prepended (its final hidden state becomes the summary vector used later for classification and for Next Sentence Prediction), and a [SEP] token marks the boundary between the two segments that BERT actually trains on — more on why there are two segments in a moment. Every token's representation entering layer 1 is the elementwise sum of three learned embeddings: the token embedding (what word/subword is this), the segment embedding (does this position belong to sentence A or sentence B), and the position embedding (where in the sequence does this sit, since self-attention itself carries no notion of order). Special tokens and this three-way sum are what the masking procedure operates on top of.

The 80/10/10 rule — and the mismatch it prevents

Of the 15% of WordPiece tokens selected for prediction in a given training sequence, BERT does not always replace them with [MASK]. It applies a three-way split, decided independently for each selected position:

• 80% of the time — replace the token with [MASK].
• 10% of the time — replace the token with a random token drawn from the vocabulary.
• 10% of the time — leave the token exactly as it is.

The naive design — always use [MASK] — has a specific, fixable failure mode. [MASK] is an artifact of pre-training. It never appears in a downstream fine-tuning dataset, and it never appears at inference time when the model is, say, classifying a movie review or tagging named entities in a news article. If the model only ever learned to build rich contextual representations for the 15% of positions that carry the fake [MASK] token, its behavior on the other 85% of positions — the ones that matter at deployment — would be undertrained and untested. Keeping 10% of the selected positions completely unmodified forces the model to keep building a strong contextual representation for every real token, since it can never be sure, from the input alone, whether a given token is "trustworthy" or "one of the ones I need to be suspicious of and double-check against context." The random-word 10% pushes this further: it forces the model to use context to detect an implausible token and correct toward the right answer rather than passively copying whatever surface form it sees — a much closer proxy for what happens when a model has to understand messy, natural fine-tuning data.

Worked example: masking a real sentence and computing the loss

Start from the WordPiece-tokenized sentence "ISRO launched Chandrayaan-3 to the moon in 2023." (subwords simplified to whole words here for readability):

tokens = ["[CLS]", "isro", "launched", "chandrayaan-3", "to",
          "the", "moon", "in", "2023", ".", "[SEP]"]

The 15% sampler is applied only to the nine real content/function tokens (indices 1–9); [CLS] and [SEP] are never selected. Suppose it selects index 6 ("moon") and index 7 ("in") — two out of nine is close to the expected 15% rate for a short sentence. Each selected position independently draws a number r uniformly from [0, 1) to decide which of the three branches applies. Say index 6 draws r = 0.42 (below 0.80 → replace with [MASK]) and index 7 draws r = 0.87 (between 0.80 and 0.90 → replace with a random vocabulary token, which happens to land on "near"):

tokens[6] = "[MASK]"   # r = 0.42, below 0.80
tokens[7] = "near"     # r = 0.87, in [0.80, 0.90)

labels = {6: "moon", 7: "in"}   # every other index is ignored (label = -100)

print(tokens)

Tracing this line by line: index 6 originally held "moon" and is overwritten with "[MASK]"; index 7 originally held "in" and is overwritten with "near"; every other index is untouched. So the statement print(tokens) outputs exactly:

['[CLS]', 'isro', 'launched', 'chandrayaan-3', 'to', 'the', '[MASK]', 'near', '2023', '.', '[SEP]']

The label dictionary records that the network must be scored, at index 6, against the true token "moon," and at index 7, against the true token "in" — using -100 as the ignore-index convention (standard in PyTorch's cross-entropy loss) for every other position, including the token "near" itself, which is now sitting in the input but is not the thing being predicted at its own position.

Now compute the loss contribution from index 7. In a real model the softmax runs over all 30,000 WordPiece types; for hand-computation, restrict attention to six plausible candidates the final layer might assign non-trivial logits to: "in," "2023," "on," "moon," "at," "near," with hypothetical output logits (before softmax) of 4.2, 2.0, 1.1, 0.9, 0.6, and −0.3 respectively. Exponentiate each:

e^4.2 = 66.686
e^2.0 =  7.389
e^1.1 =  3.004
e^0.9 =  2.460
e^0.6 =  1.822
e^-0.3 = 0.741
sum   = 82.102

Dividing each exponential by the sum 82.102 gives the softmax probabilities: P(in) = 0.812, P(2023) = 0.090, P(on) = 0.037, P(moon) = 0.030, P(at) = 0.022, P(near) = 0.009 — these six sum to 1.000, as they must. The true label at this position is "in," so the cross-entropy loss contributed by this single masked position is −ln(0.812) ≈ 0.208 nats. A companion loss term would be computed the same way at index 6 (true label "moon"), and the two are averaged (or summed, then normalized by batch size) into the training signal that actually flows backward through all twelve encoder layers. Notice what does not happen: no loss is computed at index 1 ("isro"), index 4 ("to"), or any of the other seven positions — MLM's loss surface touches only the ~15% of the sequence that was corrupted, which is precisely why BERT needs orders of magnitude more raw text than a next-token model to see the same number of "real" prediction events per token of text.

Architecture at a glance

The pipeline below traces the same sentence end to end: masking, embedding, twelve layers of bidirectional self-attention, and a softmax restricted to the corrupted positions.

Next Sentence Prediction: the objective BERT later shed

The two-segment structure (sentence A, [SEP], sentence B) exists because the original BERT paper trained on a second, simultaneous objective: Next Sentence Prediction (NSP). Fifty percent of the time, segment B genuinely is the sentence that followed segment A in the source document (label IsNext); the other fifty percent, segment B is a random sentence sampled from elsewhere in the corpus (label NotNext). The final hidden state of the [CLS] token is fed through a small classifier to predict which case it is, and this loss is added directly to the MLM loss during pre-training. The stated motivation was to give the model some signal about inter-sentence coherence, useful for downstream tasks like question answering and natural language inference, which operate on sentence pairs.

It is worth knowing — and this is the kind of detail that separates a syllabus-level answer from a research-grounded one — that NSP did not survive scrutiny. Liu et al.'s 2019 RoBERTa paper ran controlled ablations and found that removing NSP entirely, while keeping MLM and training on longer contiguous spans of text, matched or beat the original BERT on downstream benchmarks. The two objectives BERT trains jointly are not equally load-bearing: MLM is the one doing the representational heavy lifting; NSP was, at best, neutral, and later architectures (RoBERTa, ALBERT's replacement Sentence-Order Prediction) either dropped it or redesigned it. The lesson generalizes beyond BERT: a pre-training objective earns its place in the loss function only if ablating it actually hurts downstream performance, and that has to be measured, not assumed from the design intuition that motivated it.

Common misconception: "BERT writes text like GPT does"

Because both are "transformer language models pre-trained on huge text corpora," students conflate BERT and GPT as the same kind of model wearing different names, and expect that you could prompt BERT the way you prompt a chatbot and get a written continuation back. This is wrong, and the wrongness traces directly to everything above. GPT's pre-training loss scores a prediction at every position in the sequence, conditioned only on what came before it — that is precisely the setup that supports autoregressive generation: sample a token, append it, repeat, always attending only backward. BERT's pre-training loss scores predictions at roughly 15% of positions, conditioned on the entire sequence including tokens that come after — the very thing that makes it a strong encoder for classification, tagging, and retrieval makes it structurally unsuited to left-to-right generation, because at inference time on a fresh sentence, there is no "future context" to condition on; the mechanism that makes BERT good at understanding depends on information that generation cannot supply. BERT is an encoder that produces contextual representations for the tokens you feed it; you fine-tune a classification head, a span-extraction head, or a similarity function on top of those representations. It is not, and was never trained to be, a text generator.

Active recall

Attempt each question before reading its answer.

1. Why does removing the causal mask from a standard next-token language model break the training objective, and how does MLM avoid that problem instead of just removing the mask and hoping for the best?
2. A training sequence has 220 WordPiece tokens (ignore [CLS]/[SEP]). How many tokens are selected for the MLM loss, and — in expectation — how many of those are replaced with [MASK], how many with a random token, and how many left unchanged?
3. Why does keeping 10% of selected tokens completely unchanged still force useful learning, given that the model could "cheat" by just copying the input at that position?
4. A masked position has three candidate logits before softmax: "delhi" = 3.0, "mumbai" = 1.5, "chennai" = 0.5. The true label is "mumbai." Compute the softmax probabilities and the cross-entropy loss at this position.
5. What does RoBERTa's ablation of Next Sentence Prediction tell you about how to evaluate whether an auxiliary pre-training objective is actually earning its place in the loss?
6. A classmate says "BERT is just GPT trained on more data." Give the one-sentence architectural reason this is false.

Answers.

1. A next-token objective without a causal mask is self-defeating: with unrestricted bidirectional attention, position t can route its own token identity into its own output through the attention mechanism, so "predicting" token t from a representation that already contains token t is trivial copying, not language modeling — the loss goes to zero without the model learning anything about context. MLM avoids this not by changing the attention pattern but by changing the input: the token at a masked position is deleted from the input itself (replaced by [MASK] or noise) before any layer runs, so there is no leaked identity for full bidirectional attention to exploit.

2. 15% of 220 = 33 tokens selected for the loss. Of those 33, in expectation 80% × 33 = 26.4 (≈26) are replaced with [MASK], 10% × 33 = 3.3 (≈3) are replaced with a random token, and 10% × 33 = 3.3 (≈3) are left unchanged. (Exact counts vary run to run since each of the 33 positions draws its own random branch independently — these are expected values, not guarantees.)

3. The model cannot tell, from the input alone, whether the token sitting at a given position is one of the 10%-unchanged selected tokens (which it must still learn to represent contextually, since it will be scored on reconstructing it) or an ordinary 85%-not-selected token that contributes nothing to the loss. Because it can never distinguish these cases at inference over the input surface form, it is forced to keep building a genuinely contextual, non-lazy representation at every position, not just the visibly-masked ones — which is exactly the property needed for downstream fine-tuning, where [MASK] never appears at all.

4. Exponentiate: e^3.0 = 20.086, e^1.5 = 4.482, e^0.5 = 1.649; sum = 26.217. Probabilities: P(delhi) = 20.086/26.217 = 0.766, P(mumbai) = 4.482/26.217 = 0.171, P(chennai) = 1.649/26.217 = 0.063 (sums to 1.000). True label is "mumbai," so loss = −ln(0.171) ≈ 1.766 nats. Note this is a much larger loss than the 0.208 worked earlier — because here the model assigned most of its probability mass to the wrong word ("delhi"), which is exactly the training signal that pushes its weights to fix this on the next update.

5. An auxiliary objective's design rationale ("sentence coherence should help downstream QA") is a hypothesis, not evidence. The only way to know if it earns its place in the loss function is a controlled ablation: train an otherwise-identical model with and without it, and compare downstream performance. RoBERTa did this and found NSP was not load-bearing — the intuition that motivated it did not survive measurement, which is the standard every component of a pre-training recipe should be held to.

6. BERT is pre-trained with a masked language modeling objective using unrestricted bidirectional self-attention, so it is architecturally an encoder that builds a representation from full-sentence context and cannot generate text autoregressively; GPT is pre-trained with a causal next-token objective, restricting each position to leftward context, which is exactly what autoregressive generation requires — the difference is the attention/objective design, not the amount of training data.

Think About It

Think about this: How would you explain bert pre-training: masked language modeling to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.