A food-delivery platform operating at national scale receives thousands of support messages a minute, most of them free text: "Order 4521 for Paneer Tikka from Truptii's Kitchen hasn't arrived, I'm at Koramangala, call me on 9876543210." No human reads every ticket before it is routed. A system reads it first, and it has to pull out exactly four facts from that one sentence: an order ID, a dish name, a restaurant name, and a phone number, so the ticket can be handed automatically to the right rider-support queue with the right fields pre-filled. That extraction task — find the spans of text in a sentence that name a specific thing, and label each span with what kind of thing it is — is Named Entity Recognition, NER. This chapter builds one from the ground up: how the labels are represented, how a model scores a whole sentence at once instead of guessing word by word, how the best label sequence is actually computed, and how you measure whether the system is any good.
What NER Actually Asks a Model to Do
NER is not a classification problem over whole sentences ("is this sentence about a restaurant?"); it is a sequence labeling problem over tokens. Every token in the sentence gets exactly one label, and the labels for a sentence together carve it into named spans plus everything else. The label set depends entirely on the task: a newswire system typically tags PERSON, ORGANIZATION, LOCATION, DATE, and MONEY; the food-delivery ticket triage system above would define its own schema — ORDER_ID, DISH, RESTAURANT, PHONE — because the "entities" that matter are whatever the downstream system needs to act on. NER is not one fixed vocabulary of labels; it is a labeling framework you instantiate per task.
What makes it harder than a dictionary lookup is that the same surface word can be a different entity type — or no entity at all — depending on where it sits in the sentence. "Washington" is a PERSON in "Washington crossed the Delaware," a LOCATION in "the flight lands in Washington," and part of an ORGANIZATION in "Washington announced new sanctions" (referring to the U.S. government). A gazetteer — a static list of known person names, city names, company names — cannot resolve this, because the word "Washington" appears verbatim on all three lists. Only the surrounding context, the words before and after, disambiguates it. This is the central design constraint every NER system has to satisfy: the label assigned to a token must be a function of its context, not just its identity.
The BIO Tagging Scheme
A single entity often spans more than one token, so a plain "which category" label per word is not enough — you also need to know where one entity ends and the next begins. The standard solution is BIO tagging: every token gets a label of the form B-TYPE (beginning of an entity of that type), I-TYPE (inside/continuation of that entity), or O (outside any entity, i.e., not part of a named entity at all).
Take the sentence "Reserve Bank of India cut repo rate today." The organization "Reserve Bank of India" spans four tokens, and BIO tagging encodes that as one continuous unit:
Reserve B-ORG
Bank I-ORG
of I-ORG
India I-ORG
cut O
repo O
rate O
today OThe B/I distinction earns its keep exactly when two entities of the same type sit back to back with no O tag to separate them. Compare "Reserve Bank of India" (one four-word ORG) against a hypothetical "Google Microsoft announced a joint venture," where "Google" and "Microsoft" are two separate one-word ORGs with no gap between them. Tagged as I-ORG I-ORG for both, a decoder cannot tell whether it is looking at one two-word organization or two one-word ones — the sequence "I-ORG I-ORG" is genuinely ambiguous under an IO-only scheme (no B). Tagged as B-ORG B-ORG, the second B forces a new-entity boundary and the ambiguity disappears. Every extra B in a run of the same tag type is a promise from the model that a new entity has started.
The Misconception: Token Accuracy Is Not the Right Number
A natural first instinct is to evaluate an NER tagger the way you would evaluate a classifier: count the fraction of tokens whose predicted tag matches the gold tag. This number is misleading, and the reason is arithmetic, not philosophical. Take the sentence "Rahul Gandhi met officials from Google and ISRO yesterday in Bengaluru to discuss AI policy." It has 15 tokens. The true entities are Rahul Gandhi (PER, 2 tokens), Google (ORG, 1 token), ISRO (ORG, 1 token), and Bengaluru (LOC, 1 token) — 5 entity tokens out of 15, and 10 tokens that are correctly O.
A baseline model that predicts O for every single token, without looking at the input at all, gets 10 out of 15 tokens correct: 66.7% token accuracy. It has identified zero entities. It would be useless in production, yet it clears two-thirds "accuracy" — a number that sounds respectable to anyone reading a dashboard. Real corpora skew the O tag even more heavily than this toy example, which makes the trap worse the more realistic the data gets. This is precisely why the standard practice — used in every serious NER benchmark — is entity-level exact-match evaluation: a predicted entity only counts as correct if both its span (the exact set of tokens) and its type match the gold entity exactly. Getting the type right but the boundary wrong ("Rahul" instead of "Rahul Gandhi") counts as a full miss, not partial credit. Precision, recall, and F1 are computed over whole entities, never over individual tokens. The next section works this out numerically.
From Gazetteers to Neural Sequence Labelers
Four generations of approach, each fixing a limitation of the one before:
Rule-based / gazetteer. Regular expressions for structured entities (phone numbers, PAN numbers, dates) plus dictionary lookups for names. Fast, interpretable, and completely blind to context — it is exactly the approach that fails on "Washington" above, and it cannot recognize a name it has never seen (out-of-vocabulary, OOV).
Hidden Markov Models (HMM). An early statistical fix: model the tag sequence as a Markov chain and the words as emissions from hidden tag states. It captures local tag-to-tag dependencies (a generative P(word|tag) and P(tag|previous tag)) but the feature set is thin — the model can only condition on the current word, not on rich, overlapping context.
Conditional Random Fields (CRF). A discriminative model over the whole tag sequence: instead of generating words from tags, it directly scores how well a candidate tag sequence fits the sentence, using arbitrary hand-designed features per position — word shape (does "ISRO" look like "XXXX", all-caps?), capitalization, prefix/suffix, part-of-speech tag, gazetteer membership, and a window of surrounding words. Crucially, a CRF still keeps a learned tag-to-tag transition score, so it never independently guesses each tag; it scores full sequences. This is the piece that fixes the boundary problem: a plain per-token classifier with no transition model can output nonsense like O, I-PER, I-PER (an "I" tag appearing with no preceding "B"), because it decides each token in isolation. A CRF's transition scores can be trained to make that jump costly, steering the decoder away from illegal sequences.
BiLSTM-CRF and transformer token classifiers. Hand-designed features are replaced by learned ones. A bidirectional LSTM reads the sentence left-to-right and right-to-left, producing for each token a representation that has seen both its left and right context; that representation is fed through a dense layer to get per-token, per-tag emission scores, and a CRF layer sits on top to add the transition scores and decode the best-scoring full sequence — this is the BiLSTM-CRF architecture, and it is the one this chapter builds by hand below. Fine-tuning a transformer (BERT-style) for token classification follows the same shape, with self-attention replacing the LSTM as the context encoder; because self-attention already looks at the whole sentence at once, many transformer NER systems skip the CRF layer entirely, though production systems often keep it anyway as a cheap guarantee against illegal tag sequences.
Architecture: Inside a BiLSTM-CRF Tagger
The diagram below traces one three-token sentence — "Aryan joined ISRO" — through every stage of a BiLSTM-CRF tagger, using the exact emission and transition scores worked out numerically in the next section, so the diagram and the arithmetic describe the same computation.
Read the red path: the CRF layer does not just look at each column's emission scores in isolation — it considers all nine possible tag-to-tag transitions between every pair of adjacent columns (the light gray mesh) and keeps only the highest-scoring complete path through the whole sentence (the bold red edges). That "keep only the best path so far" rule is exactly the dynamic-programming idea behind the Viterbi algorithm, worked out numerically next.
Worked Example: Decoding "Aryan joined ISRO" with Viterbi
A CRF assigns every candidate tag sequence a score equal to the sum of emission scores (how well each tag fits its token, from the BiLSTM) plus transition scores (how well each tag fits the tag before it, learned separately). Finding the best sequence by trying every combination is exponential: with 3 possible tags and 3 tokens there are 3³ = 27 sequences, and for a K-tag scheme over an N-token sentence it is K^N — intractable for realistic sentences. The Viterbi algorithm avoids this by dynamic programming: for each position, it only needs to remember the best score achievable for ending in each tag, not every path that could have led there. That is the same principle behind edit distance or longest-common-subsequence — an optimal path to any point must itself be built from an optimal path to some earlier point.
Use the tag set {O, PER, ORG} and this sentence's scores (same numbers as the diagram, in log-space so scores simply add):
Emission scores
O PER ORG
Aryan 0.5 2.5 0.2
joined 2.0 0.1 0.1
ISRO 0.3 0.2 2.8
Transition scores (row = from, column = to)
O PER ORG
O 1.0 0.3 0.3
PER 1.2 -1.0 0.4
ORG 1.0 0.2 -1.0
START scores: O=0.5 PER=1.0 ORG=0.2
END scores: O=0.5 PER=0.2 ORG=0.5Let score(i, t) be the best total score of any path that ends at position i in tag t. The recurrence is score(i, t) = emission(word_i, t) + max over previous tag p of [score(i−1, p) + transition(p, t)], and the base case folds in the START score instead of a previous position.
Position 1 (Aryan). No previous tag, so score(1, t) = START(t) + emission(Aryan, t):
score(1,O) = 0.5 + 0.5 = 1.0
score(1,PER) = 1.0 + 2.5 = 3.5
score(1,ORG) = 0.2 + 0.2 = 0.4Position 2 (joined). For each tag, try all three previous tags and keep the best:
tag O: from O: 1.0+1.0=2.0 | from PER: 3.5+1.2=4.7 | from ORG: 0.4+1.0=1.4
best = 4.7 (from PER) -> score(2,O) = 2.0 + 4.7 = 6.7
tag PER: from O: 1.0+0.3=1.3 | from PER: 3.5-1.0=2.5 | from ORG: 0.4+0.2=0.6
best = 2.5 (from PER) -> score(2,PER) = 0.1 + 2.5 = 2.6
tag ORG: from O: 1.0+0.3=1.3 | from PER: 3.5+0.4=3.9 | from ORG: 0.4-1.0=-0.6
best = 3.9 (from PER) -> score(2,ORG) = 0.1 + 3.9 = 4.0Every best-previous-tag at position 2 is PER — unsurprising, since PER had the highest score at position 1 by a wide margin, and none of the transition penalties are steep enough to overturn that lead yet.
Position 3 (ISRO).
tag O: from O: 6.7+1.0=7.7 | from PER: 2.6+1.2=3.8 | from ORG: 4.0+1.0=5.0
best = 7.7 (from O) -> score(3,O) = 0.3 + 7.7 = 8.0
tag PER: from O: 6.7+0.3=7.0 | from PER: 2.6-1.0=1.6 | from ORG: 4.0+0.2=4.2
best = 7.0 (from O) -> score(3,PER) = 0.2 + 7.0 = 7.2
tag ORG: from O: 6.7+0.3=7.0 | from PER: 2.6+0.4=3.0 | from ORG: 4.0-1.0=3.0
best = 7.0 (from O) -> score(3,ORG) = 2.8 + 7.0 = 9.8Finish. Add the END score to each position-3 total: O gives 8.0+0.5=8.5, PER gives 7.2+0.2=7.4, ORG gives 9.8+0.5=10.3. ORG wins with 10.3. Backtracking through the stored best-previous-tag pointers: position 3 = ORG (arrived from O), position 2 = O (arrived from PER), position 1 = PER. Reading forward: PER, O, ORG — "Aryan" is a person, "joined" is outside any entity, "ISRO" is an organization, which is exactly the correct tagging, and it fell out of pure arithmetic, never an explicit rule about capitalized words following "joined."
Note the win margin: the greedy, tag-each-token-by-its-own-best-emission choice would also have picked PER, O, ORG here (each token's individually highest emission score already matches the winning tag), so this particular sentence does not showcase transition scores overturning a greedy mistake — but position 2's calculation shows the mechanism that would: every tag's best score at position 2 came from a PER predecessor, not because PER-to-X transitions are universally favored, but because PER's lead from position 1 was large enough to survive the transition costs. A sentence where the top two emission scores at some position are close together is exactly where transition scores can flip the outcome, and no greedy per-token decision could reproduce that.
The same computation as code — the search order and the recurrence are identical to the trace above, so the output is fully determined by the tables just used:
def viterbi(tokens, tags, emission, transition, start_score, end_score):
n = len(tokens)
score = [{} for _ in range(n)]
back = [{} for _ in range(n)]
for t in tags:
score[0][t] = start_score[t] + emission[tokens[0]][t]
back[0][t] = None
for i in range(1, n):
for t in tags:
best_prev, best_val = None, float("-inf")
for p in tags:
val = score[i - 1][p] + transition[p][t]
if val > best_val:
best_val, best_prev = val, p
score[i][t] = best_val + emission[tokens[i]][t]
back[i][t] = best_prev
best_tag, best_val = None, float("-inf")
for t in tags:
val = score[n - 1][t] + end_score[t]
if val > best_val:
best_val, best_tag = val, t
path = [best_tag]
for i in range(n - 1, 0, -1):
path.append(back[i][path[-1]])
path.reverse()
return path
tags = ["O", "PER", "ORG"]
tokens = ["Aryan", "joined", "ISRO"]
emission = {
"Aryan": {"O": 0.5, "PER": 2.5, "ORG": 0.2},
"joined": {"O": 2.0, "PER": 0.1, "ORG": 0.1},
"ISRO": {"O": 0.3, "PER": 0.2, "ORG": 2.8},
}
transition = {
"O": {"O": 1.0, "PER": 0.3, "ORG": 0.3},
"PER": {"O": 1.2, "PER": -1.0, "ORG": 0.4},
"ORG": {"O": 1.0, "PER": 0.2, "ORG": -1.0},
}
start_score = {"O": 0.5, "PER": 1.0, "ORG": 0.2}
end_score = {"O": 0.5, "PER": 0.2, "ORG": 0.5}
print(viterbi(tokens, tags, emission, transition, start_score, end_score))
# ['PER', 'O', 'ORG']The cost of this computation is what makes it practical: at each of the N positions, the inner loop checks K previous tags for each of K current tags, so the whole decode is O(N·K²) work, against O(K^N) for trying every full sequence. With N = 8 tokens and a realistic K = 9 tags (O plus B/I for four entity types), brute force needs 9^8 = 43,046,721 sequence evaluations; Viterbi needs 8×9² = 648. That is roughly 66,400 times fewer operations for a sentence you would still call short.
Evaluating an NER System: Precision, Recall, F1
Return to the food-delivery ticket: "Order 4521 for Paneer Tikka from Truptii's Kitchen delayed, call 9876543210." Four gold entities: ORDER_ID = "4521", DISH = "Paneer Tikka", RESTAURANT = "Truptii's Kitchen", PHONE = "9876543210". Suppose the trained system predicts ORDER_ID = "4521" (correct), DISH = "Tikka" (wrong boundary — the gold span is "Paneer Tikka," so this does not match), RESTAURANT = "Truptii's Kitchen" (correct), PHONE = "9876543210" (correct).
Count at the entity level, exact span and type: true positives (TP) are predictions that exactly match a gold entity — 3 of them (ORDER_ID, RESTAURANT, PHONE). False negatives (FN) are gold entities with no matching prediction — 1 (the correct DISH span "Paneer Tikka" was never produced). False positives (FP) are predictions with no matching gold entity — 1 (the spurious span "Tikka" tagged DISH does not match any gold entity, since "Paneer Tikka" ≠ "Tikka").
def prf1(tp, fp, fn):
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
return round(precision, 3), round(recall, 3), round(f1, 3)
print(prf1(3, 1, 1))
# (0.75, 0.75, 0.75)Precision 0.75 says three-quarters of what the system flagged as an entity was actually correct; recall 0.75 says three-quarters of the entities that were really there got found; F1, the harmonic mean, summarizes both in one number. Note that the single boundary mistake on "Paneer Tikka" cost the system twice — once as a missed gold entity (hurting recall) and once as a wrong prediction (hurting precision) — which is exactly why entity-level exact-match scoring is stricter, and more honest, than counting correct tokens.
Active Recall
Attempt each question before reading its answer.
- Why does the word "the" in "the Reserve Bank of India cut rates" get the tag O rather than I-ORG?
- A per-token classifier with no CRF layer outputs the tag sequence O, I-PER, I-PER, O for a four-word sentence. What is wrong with this sequence, and what mechanism in a CRF is specifically designed to make sequences like it unlikely?
- A system achieves precision 0.6 and recall 0.9 on entity-level evaluation. Compute its F1.
- For the sentence "Rahul Gandhi met officials from Google and ISRO yesterday in Bengaluru to discuss AI policy," a system correctly tags Google and ISRO as ORG, tags only "Rahul" (not "Rahul Gandhi") as PER, and produces no tag at all for Bengaluru. Compute entity-level precision, recall, and F1.
- Why can a static gazetteer (a fixed list of known organization names) never correctly resolve "Washington" across "Washington crossed the Delaware" and "Washington announced sanctions today," no matter how large the list gets?
- For an 8-token sentence with a 9-tag BIO scheme, roughly how many times more sequence evaluations does brute-force search require compared to Viterbi decoding?
Answers.
1. "the" is not part of any named entity — it is not "Reserve," "Bank," "of," or "India" themselves and it precedes the ORG span rather than sitting inside it, so it correctly falls outside the four-token span and takes O.
2. The sequence starts an I-PER tag with no preceding B-PER — under BIO, I-TYPE is only valid as a continuation of a B-TYPE (or another I-TYPE) of the same type; here it appears right after an O, which is illegal. A CRF's learned transition score for O→I-PER can be trained low (even strongly negative), so during Viterbi decoding that transition is disfavored against every competing path, steering the decoder toward valid sequences like O, B-PER, I-PER, O instead. A plain per-token classifier has no transition term at all, so it has no mechanism to penalize this jump.
3. F1 = 2 × 0.6 × 0.9 / (0.6 + 0.9) = 1.08 / 1.5 = 0.72.
4. Gold entities: Rahul Gandhi (PER), Google (ORG), ISRO (ORG), Bengaluru (LOC) = 4. Predictions: "Rahul" (PER, wrong — doesn't match the gold span exactly), Google (ORG, correct), ISRO (ORG, correct). TP = 2, FP = 1 ("Rahul" matches no gold entity), FN = 2 (Rahul Gandhi and Bengaluru both go unmatched). Precision = 2/3 ≈ 0.667, recall = 2/4 = 0.5, F1 = 2 × 0.667 × 0.5 / (0.667 + 0.5) ≈ 0.667 / 1.167 ≈ 0.571.
5. Because a gazetteer only stores the word's identity, and identity alone doesn't determine entity type — "Washington" appears verbatim on a person-name list, a city list, and could plausibly stand in for a government/organization reference, so a lookup returns all matches (or an arbitrary tie-break) regardless of the sentence it appears in. Disambiguation requires context: the verb "crossed," the preposition "in," or the verb "announced" around it — signals no static list can encode.
6. Brute force: 9^8 = 43,046,721. Viterbi: 8 × 9² = 648. Ratio ≈ 43,046,721 / 648 ≈ 66,400 — roughly 66,000 times more work for brute force on a sentence this short, and the gap widens exponentially as sentence length grows.
Think About It
Think about this: How would you explain named entity recognition: building an ner system to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind named entity recognition: building an ner system, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.