BERT: Bidirectional Encoder Representations

On 25 October 2019, Google announced that a model called BERT had gone live inside Search and would touch roughly one in every ten queries typed into the box, in every language Search operated in, including Hindi and other Indian languages within two months. The example Google published to explain why was the query "2019 brazil traveler to usa need a visa." A keyword-matching search engine sees "brazil," "usa," "traveler," "visa" and returns pages about US citizens who need a Brazilian visa — because it treats "to" as a throwaway stopword and matches on nouns. The actual intent runs the other direction: a Brazilian citizen traveling to the USA. Getting this right requires the system to hold "traveler," "to," and "usa" in a single joint context and let each word's meaning bend around the others, in both directions at once. That is precisely the computation BERT performs, and it is why the "B" in its name is doing all the work.

This chapter builds BERT from the ground up: what "bidirectional" costs to achieve, why a masked-word prediction task was the only way to buy it, how the encoder stack and input representation are built, and how a single pretrained network becomes a classifier, a named-entity tagger, or a question-answering system with almost no new architecture. You already know self-attention and the Transformer block from the deep learning and NLP chapters; here you see what BERT specifically does with that machinery that GPT-style models cannot.

The directionality problem language models had before BERT

Before 2018, the dominant way to get contextual word representations was a language model: predict the next word given the words so far, and use the hidden states it builds along the way as embeddings. This objective is naturally left-to-right — you can only condition on words you have already seen, because the training signal is "guess the next one." ELMo (2018) tried to add right-context by training a second LSTM that reads the sentence backwards and predicting each word from its future context, then concatenating the forward and backward hidden states. This is genuinely useful, but it is shallow bidirectionality: two independent unidirectional models are trained separately and glued together at the output. Neither the forward LSTM nor the backward LSTM ever, at any layer, computes a representation that has simultaneously seen both the left and right context of a word.

BERT's insight was to drop the autoregressive objective entirely and use a Transformer encoder stack, where self-attention lets every token attend to every other token in the sequence, left and right, at every one of its layers — not just at the end. The catch is exactly why nobody had done this for language modeling before: if you try to train "predict word 5" using a model where word 5 can attend to itself, the task becomes trivial. Every layer of self-attention would let the target word's own identity leak straight through into its own prediction, and the network learns nothing except to copy its input. Full bidirectional self-attention and a left-to-right prediction objective are fundamentally incompatible. BERT's entire pretraining design exists to route around this contradiction.

Masked Language Modeling: buying bidirectionality with a fill-in-the-blank task

BERT's fix is to stop predicting the next word and instead predict missing words scattered throughout the sentence — a task where seeing both directions is not cheating, because the word you're asked to recover is never available to look at. This objective is Masked Language Modeling (MLM), and the recipe from the original paper (Devlin, Chang, Lee & Toutanova, Google AI Language, 2018) is precise:

• Randomly select 15% of the WordPiece tokens in each training sequence as prediction targets.
• Of those selected tokens: 80% are replaced with a special [MASK] token, 10% are replaced with a random token from the vocabulary, and 10% are left unchanged.
• The model must predict the original token at every selected position, regardless of which of the three things it saw there.

The 80/10/10 split is not decoration — it solves a second, subtler leakage problem. If every masked position always showed the literal string [MASK], the model would only ever need to learn "what word goes here" for inputs containing that artificial token, and it would never learn to produce good representations for real, unmasked words — a mismatch, since [MASK] never appears during fine-tuning or real use. Occasionally substituting a random wrong word forces the model to stay skeptical of every token's surface identity, not just the masked ones, which is what makes the resulting embeddings useful as general-purpose representations rather than a mask-detector.

Consider a 20-token input sentence during pretraining. 15% of 20 is 3 tokens selected. In expectation, of those 3: 2.4 become [MASK], 0.3 become a random substituted token, and 0.3 are left as-is (these are expected counts — the actual draw for any one sentence is a small integer outcome of independent per-token coin flips, so a specific 20-token sentence might see exactly 2 masked, 1 substituted, 0 unchanged, or any other combination near this ratio). The loss is computed only at those 3 positions, using the true original token as the label at each of them — a cross-entropy over the ~30,000-token WordPiece vocabulary.

Input representation: WordPiece, segments, and position, summed

Before any of this reaches the Transformer layers, BERT builds each input vector as the elementwise sum of three learned embeddings:

• Token embedding, from a WordPiece vocabulary of about 30,000 subword units. WordPiece splits rare words into known pieces — "hyperparameter" might tokenize as hyper + ##parameter, where ## marks a piece that continues the previous token rather than starting a new word. This keeps the vocabulary bounded while still covering arbitrary words, including transliterated Indian-language terms or code identifiers the pretraining corpus never saw as whole words.
• Segment embedding, one of exactly two learned vectors (E_A or E_B) marking whether a token belongs to the first or second sentence in a pair — needed because BERT is often given two sentences at once, separated by a [SEP] token.
• Position embedding, a learned vector per absolute position (0 to 511, since BERT's maximum sequence length is 512 tokens), since self-attention itself has no built-in notion of order and would otherwise treat the input as an unordered set.

Every sequence is also prefixed with a special [CLS] token. [CLS] carries no meaning of its own at input time; it exists purely so that after 12 (BERT-base) or 24 (BERT-large) layers of self-attention have let it attend to every other token, its final hidden state can serve as a single pooled, whole-sequence representation — the vector that classification heads are attached to. BERT-base has 12 layers, 12 attention heads, and a hidden size of 768, for roughly 110 million parameters; BERT-large has 24 layers, 16 heads, hidden size 1024, and roughly 340 million parameters. Both were pretrained on BooksCorpus (about 800 million words) plus English Wikipedia (about 2.5 billion words, text passages only, no tables or lists).

Next Sentence Prediction: giving [CLS] something to learn from sentence pairs

MLM alone teaches BERT to model relationships between nearby words, but many downstream tasks — answering a question about a passage, checking whether one sentence follows logically from another — require modeling relationships between whole sentences. BERT's second pretraining objective, Next Sentence Prediction (NSP), targets exactly this. Each training example is a pair of sentences A and B, packed as [CLS] A [SEP] B [SEP]. 50% of the time B is the sentence that actually follows A in the source text (label IsNext); the other 50% of the time B is a random sentence pulled from elsewhere in the corpus (label NotNext). A binary classifier sitting on top of the final [CLS] hidden state is trained to tell these apart, jointly with the MLM loss on the same batch. Later work (RoBERTa, Liu et al., 2019) found that removing NSP and simply training MLM longer, on more data, with larger batches, matched or beat the original BERT on most benchmarks — evidence that NSP's contribution was smaller than the original paper's ablations suggested, though the [CLS]-plus-[SEP] input format it motivated is still exactly what BERT's fine-tuning setups rely on today.

Worked example: what "bidirectional" computes, by hand

Self-attention is what actually delivers bidirectionality, so it is worth tracing one query through it with real numbers rather than taking "attends to everything" on faith. Take the four-token sentence RBI cuts repo rate (word-level for arithmetic simplicity; a real BERT tokenizer would further split some of these into WordPieces). Suppose that after the learned query/key/value projections, one attention head produces these vectors — invented here to keep the arithmetic small, but structurally exactly what a trained W_Q, W_K, W_V would output:

token   query (for "cuts")   key         value
RBI     [1, 0]               [1, 1]      [2, 0]
cuts    —                    [0, 1]      [0, 2]
repo    —                    [1, -1]     [1, 1]
rate    —                    [-1, 1]     [3, -1]

We are computing the new, contextual representation of the token "cuts". Step 1: dot the query for "cuts" against every key in the sequence, including keys that come after "cuts" in the sentence — this is the step a causal, left-to-right model is forbidden from taking, since it would have "repo" and "rate" masked out of view entirely.

score(cuts, RBI)  = (1)(1) + (0)(1)  =  1
score(cuts, cuts) = (1)(0) + (0)(1)  =  0
score(cuts, repo) = (1)(1) + (0)(-1) =  1
score(cuts, rate) = (1)(-1)+ (0)(1)  = -1

Step 2: scale by 1/√d, with head dimension d = 2, so √2 ≈ 1.41421:

RBI: 1 / 1.41421  =  0.70711
cuts: 0 / 1.41421 =  0
repo: 1 / 1.41421 =  0.70711
rate: -1 / 1.41421 = -0.70711

Step 3: softmax across these four scaled scores. Exponentiating gives e^0.70711 ≈ 2.02811 (twice, for RBI and repo), e^0 = 1, and e^-0.70711 ≈ 0.49307. These sum to Z ≈ 5.54930, giving attention weights:

weight(RBI)  = 2.02811 / 5.54930 ≈ 0.36547
weight(cuts) = 1.00000 / 5.54930 ≈ 0.18020
weight(repo) = 2.02811 / 5.54930 ≈ 0.36547
weight(rate) = 0.49307 / 5.54930 ≈ 0.08885

(These sum to 1.00000, as any softmax output must.) Step 4: the new representation of "cuts" is the weighted sum of every value vector, using these weights:

out_x = 0.36547(2) + 0.18020(0) + 0.36547(1) + 0.08885(3) = 1.36297
out_y = 0.36547(0) + 0.18020(2) + 0.36547(1) + 0.08885(-1) = 0.63703

new representation of "cuts" ≈ [1.363, 0.637]

Read the weights, not just the final vector: "RBI" (left context) and "repo" (right context) each pull 36.5% of the new "cuts" vector — almost identical influence, despite sitting on opposite sides of the word. "rate," two positions to the right, still contributes 8.9%. A causal decoder computing this same query could only ever have used the RBI and cuts columns; the repo and rate columns would be masked to -∞ before the softmax, forcing weight(repo) = weight(rate) = 0 and reallocating everything to the left context alone. That masked-out 45% of the probability mass — the part coming from "repo rate," which is exactly what tells the model this "cuts" means a rate cut and not, say, a budget cut or a production cut — is what bidirectional attention buys and a left-to-right model cannot have. In a real BERT layer this happens for every token, in parallel, across 12 or 16 attention heads at once, then stacks 12 or 24 times so that by the final layer "cuts" has folded in second- and third-order context (what "repo" itself attended to, and so on).

Fine-tuning: one pretrained encoder, many downstream heads

Pretraining produces a network that is good at nothing directly useful — filling in blanks and guessing whether two sentences are adjacent are not real tasks anyone wants solved. What pretraining buys is a set of encoder weights that already understand English (or Hindi, or code, depending on the pretraining corpus) well enough that very little additional machinery turns them into a task solver. Fine-tuning takes the pretrained weights, adds one small task-specific layer, and trains the whole network end-to-end on labeled data for the target task, typically for just a few epochs:

• Sentence classification (sentiment, spam detection): feed the final [CLS] vector into one new linear layer plus softmax over the class labels.
• Named entity recognition: feed every token's final hidden state into a shared linear layer plus softmax over entity tags (PERSON, ORG, LOCATION, O), predicting one tag per token.
• Extractive question answering (SQuAD-style — find the answer span inside a given passage): pack [CLS] question [SEP] passage [SEP] as input, and add two new vectors that dot-product against every passage token's hidden state to score it as the start and end of the answer span, independently.

In every case the pretrained encoder itself is architecturally untouched — the same stack of self-attention layers, the same 768 or 1024 hidden dimensions — and only a few thousand to a few hundred thousand new parameters are added on top, versus the 110 million-plus already trained. This is the paradigm BERT established that the field still runs on: pretrain once on unlabeled text at massive scale, fine-tune cheaply on small labeled datasets per task, rather than training a fresh network from random weights for every new problem.

The misconception to correct

The most common mistake at this point is to think BERT is "bidirectional" the same way ELMo is bidirectional — that somewhere inside it, a left-to-right pass and a right-to-left pass are computed separately and stitched together. They are not. ELMo genuinely does run two independent LSTMs and concatenate their final hidden states; that is shallow, late-fusion bidirectionality. BERT has no forward pass and no backward pass at all — there is exactly one pass, and every self-attention layer, from the first to the last, lets every token see every other token in the same computation, simultaneously. The worked "cuts" example above is not a fusion of two separate directional views; the 0.36547 weight on "RBI" and the 0.36547 weight on "repo" were computed in the very same softmax, at the very same layer. This is precisely what the original paper's title is asserting when it calls BERT "deep" bidirectional rather than just bidirectional: depth here means the bidirectional conditioning happens at every layer of the stack, not only once at the output, which is why BERT representations at every intermediate layer already blend both directions — a property ELMo's architecture cannot produce at any layer.

How bidirectional attention differs from left-to-right

Active recall

Attempt these before reading the answers below.

1. Why can't you train a deep bidirectional Transformer with a "predict the next word" objective, the way GPT is trained?
2. A 40-token sequence goes through BERT's masking procedure. How many tokens are selected for masking, and in expectation how many of those become [MASK], how many become a random token, and how many stay unchanged?
3. What specific problem does the 10% "leave unchanged" and 10% "random token" split solve, that always using [MASK] would not?
4. In the worked "RBI cuts repo rate" example, why is it significant that weight(repo) ≈ weight(RBI) ≈ 0.365, rather than weight(repo) being much smaller?
5. Name the specific way BERT's bidirectionality differs from ELMo's, beyond just "BERT is deep and ELMo is shallow."
6. You want to fine-tune BERT to tag every word in a Hindi-English code-switched tweet as PERSON, LOCATION, ORG, or O. Which token's hidden state(s) do you attach the classification head to, and why not just [CLS]?

Answers.

1. Because self-attention has no directional restriction by default — every token can see every position in the sequence. If you kept that and asked the model to predict word i from the sequence, word i's own vector would flow straight into its own prediction through every layer, so the model could get zero training loss by simply copying its input instead of learning language structure. A next-word objective only works if the network is prevented from seeing the target, which for a bidirectional network means masking out all future positions — at which point it is no longer bidirectional, it's causal.

2. 15% of 40 = 6 tokens are selected. In expectation: 0.8 × 6 = 4.8 become [MASK], 0.1 × 6 = 0.6 become a random token, and 0.1 × 6 = 0.6 stay unchanged. These are expected values from independent per-token draws — any specific sentence's actual split will be nearby integers, not exactly these fractions.

3. If every masked position always literally contained the token [MASK], the model would only need to learn good representations for inputs containing that artificial token — a token that never appears in real text during fine-tuning or deployment. This train/inference mismatch would leave the model undertrained on ordinary, unmasked words. Occasionally showing the correct-looking-but-wrong random token, and occasionally showing the true token itself, forces the model to build genuinely context-sensitive representations for every position, not just the artificially marked ones.

4. Because it demonstrates that "cuts" is drawing nearly equal information from its left context (RBI) and its right context (repo), inside the exact same attention computation. A model restricted to left context alone could never assign any weight to "repo" at all — that weight would be forced to exactly 0 by causal masking, and the 0.365 currently going to "repo" would have to be redistributed entirely to RBI and cuts. The near-equal split is the numerical signature of true bidirectional conditioning, not an artifact of this particular toy example.

5. ELMo runs two entirely separate LSTMs — one trained left-to-right, one trained right-to-left — and only combines their outputs by concatenation at the very end, so no internal computation of either LSTM ever has access to both directions simultaneously. BERT has one network with one self-attention computation per layer in which every token attends to every other token — left and right context are combined inside the same softmax, at every layer, not fused after the fact from two independently-trained models.

6. Every token's own final hidden state, not [CLS]. [CLS] is built to summarize the whole sequence into one vector, which is right for a single sequence-level decision (is this tweet spam, positive, negative), but tagging is a per-token decision — you need one PERSON/LOCATION/ORG/O prediction for each individual word, so the shared classification head is applied independently to each token's own hidden state coming out of the final encoder layer.

Think About It

Think about this: How would you explain bert: bidirectional encoder representations to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where bert: bidirectional encoder representations is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting bert: bidirectional encoder representations to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind bert: bidirectional encoder representations, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.