Named Entity Recognition and Information Extraction

A quantitative trading desk subscribes to a feed of exchange filings and business-news headlines — several thousand a day, arriving faster than any analyst can read them. Somewhere in a stream that includes "Tata Motors will invest ₹18,000 crore in its EV plant in Sanand, Gujarat by 2027" and "Infosys posts an 18% rise in quarterly profit, announces ₹9,300 crore buyback," a trading algorithm needs to know, within milliseconds: which company, what financial action, how much money, by when. The sentence itself is useless to the algorithm. What it needs is a row in a table — company="Tata Motors", action="capex", amount=180000000000, location="Sanand, Gujarat", year=2027 — built automatically from raw text, thousands of times a second, without a human reading a single headline. That conversion, from free-flowing sentences to structured records a program can query and reason over, is the subject of this chapter. It happens in two stages. First you have to find and label the meaningful chunks of text — Named Entity Recognition (NER). Then you have to connect those chunks into a structured fact — Information Extraction (IE), of which NER is the first and most fundamental step.

What counts as a named entity

A named entity is a span of text that refers to a specific, uniquely identifiable thing in the world: a person ("Sundar Pichai"), an organisation ("Indian Space Research Organisation"), a location ("Bengaluru"), a date ("last Tuesday"), a monetary amount ("₹9,300 crore"), or a percentage ("18%"). It is not a general noun. "Company," "profit," and "plant" are not entities — they are categories. "Tata Motors" is an entity because it picks out one specific referent from the set of all companies.

NER treats this as a two-part decision made for every token in a sentence: (1) is this token part of a named entity at all, and if so (2) which type, and (3) does it continue an entity that started at the previous token or begin a new one? That third question matters because entities are frequently more than one word long — "Indian Space Research Organisation" is four tokens naming a single organisation, not four separate entities.

NER as sequence labeling: the BIO scheme

The standard way to represent this is the BIO scheme (Beginning, Inside, Outside), the same sequence-labeling framing used for part-of-speech tagging, but with a different label set and purpose — POS tagging asks "what grammatical role does this token play," NER asks "is this token part of a real-world entity, and which kind." Every token gets exactly one tag: O if it is outside any entity, B-TYPE if it is the first token of an entity of that type, and I-TYPE if it continues an entity of that type from the previous token. Consider:

Notice "Indian" gets B-ORG, not B-LOC — even though "Indian" alone often signals a location or nationality, here it is the first word of an organisation's proper name. The tag depends on what the whole span turns out to be, not on the word in isolation. This is the central difficulty of NER, and we return to it below.

Once a sequence is tagged, converting BIO tags into a clean list of entities is a single linear pass: walk the tokens, and whenever a B- tag appears, close off whatever entity was open and start a new one; whenever an I- tag appears that matches the type currently open, extend it; whenever O appears, close off any open entity.

def extract_entities(tagged_tokens):
    entities = []
    current = None
    for token, tag in tagged_tokens:
        if tag == "O":
            if current:
                entities.append(current)
                current = None
        elif tag.startswith("B-"):
            if current:
                entities.append(current)
            current = {"type": tag[2:], "text": token}
        elif tag.startswith("I-"):
            if current and current["type"] == tag[2:]:
                current["text"] += " " + token
            else:
                current = {"type": tag[2:], "text": token}
    if current:
        entities.append(current)
    return entities

Tracing this on the tagged sentence above: Sundar/B-PER opens current={"type":"PER","text":"Sundar"}. Pichai/I-PER matches the open type, so current["text"] becomes "Sundar Pichai". visited/O closes it into entities. the/O does nothing. Indian/B-ORG opens a new entity; Space, Research, Organisation (all I-ORG) extend it to "Indian Space Research Organisation". in/O closes it. Bengaluru/B-LOC opens and immediately closes (the next tag is a fresh B-) as "Bengaluru". last/B-DATE opens, Tuesday/I-DATE extends it to "last Tuesday", and the final ./O closes it. The function returns:

[{"type": "PER", "text": "Sundar Pichai"},
 {"type": "ORG", "text": "Indian Space Research Organisation"},
 {"type": "LOC", "text": "Bengaluru"},
 {"type": "DATE", "text": "last Tuesday"}]

Decoding a tag sequence: the Viterbi algorithm

The harder question is how a model chooses the tags in the first place. A naive approach scores each token independently and picks the highest-scoring tag for that token alone — call this "greedy tagging." It fails for a structural reason: entity tags are not independent of their neighbours. An I-ORG almost never follows an O, and if "leads" scores slightly higher as the start of an organisation name than as a plain verb, a greedy tagger will happily mislabel it, ignoring the fact that no organisation called "Leads" makes sense between "Ravi" and "Infosys." What is needed is a way to score entire tag sequences, combining how well each tag fits its token (an emission score) with how well each tag follows the previous one (a transition score), and to search efficiently over all sequences for the best-scoring one. That search is the Viterbi algorithm, a dynamic-programming method, and the transition-plus-emission scoring it operates on is exactly what a Conditional Random Field (CRF) layer computes — the same CRF layer sitting on top of today's transformer-based NER models.

Work through a deliberately small example so every number can be checked by hand. Sentence: "Ravi leads Infosys," three tokens, and — to keep the trellis hand-traceable — a reduced tag set of just three tags: O, B-PER, B-ORG (no I- tags, since every entity here is a single token). The scores below are toy numbers standing in for what a trained model would output; they are not probabilities, just relative preferences.

Note that on emission scores alone, "leads" is scored slightly higher as B-ORG (3) than as O (2) — a plausible mistake a shallow model might make from surface features alone (it is capitalised-adjacent to "Infosys," a known organisation). Transition scores, read as "from row-tag to column-tag," and start scores (the score of each tag opening the sentence):

The transition table encodes a real linguistic regularity: after a person-entity, plain words (O) are strongly favoured (score 3) over immediately starting another named entity — a person's name is usually followed by a verb, not another proper noun. Viterbi's job is to find, for every (token, tag) pair, the best cumulative score of any path reaching that pair, by combining the best path to every possible previous tag with the transition into the current tag, then adding this token's emission score. Step by step:

Token 1 ("Ravi"): score(tag) = start(tag) + emission(tag, Ravi). score(O) = 1+1 = 2. score(B-PER) = 2+5 = 7. score(B-ORG) = 0+0 = 0. Best so far: B-PER at 7.

Token 2 ("leads"): for each tag j, take the max over the previous tag i of [score₁(i) + trans(i→j)], then add emission(j, leads). For j=O: max(2+1, 7+3, 0+1) = max(3, 10, 1) = 10, from B-PER. score(O) = 10+2 = 12. For j=B-PER: max(2+0, 7+0, 0+0) = 7, from B-PER. score(B-PER) = 7+0 = 7. For j=B-ORG: max(2+0, 7+0, 0+0) = 7, from B-PER. score(B-ORG) = 7+3 = 10. Every best-path-so-far at this step traces back through B-PER — expected, since B-PER was the only strong option after token 1.

Token 3 ("Infosys"): For k=O: max(12+1, 7+3, 10+1) = max(13, 10, 11) = 13, from O. score(O) = 13+1 = 14. For k=B-PER: max(12+0, 7+0, 10+0) = 12, from O. score(B-PER) = 12+0 = 12. For k=B-ORG: max(12+0, 7+0, 10+0) = 12, from O. score(B-ORG) = 12+6 = 18.

The highest final score is 18, for B-ORG at "Infosys," reached via O at "leads." Backtracking: token 3 = B-ORG ← token 2 = O ← token 1 = B-PER. The decoded sequence is B-PER, O, B-ORG — Ravi tagged as a person, leads as outside any entity, Infosys as an organisation — the linguistically correct answer, score 18.

Compare this to the greedy sequence a per-token argmax would have produced: B-PER (emission 5, correct), B-ORG (emission 3 beats O's 2 — wrong), B-ORG (emission 6, correct). That sequence's total score under the same model is start(B-PER)+e(B-PER,Ravi) + trans(B-PER→B-ORG)+e(B-ORG,leads) + trans(B-ORG→B-ORG)+e(B-ORG,Infosys) = (2+5) + (0+3) + (0+6) = 16. Viterbi's answer (18) beats greedy's answer (16) and is also the one a human would tag — the transition scores, which greedy ignores entirely, are what fix the mistake.

The diagram below is exactly this computation as a trellis: one column per token, one row per tag, every node holding the cumulative best score reaching it, and the red path the winning backtrace.

The full algorithm, generalised, confirms the hand trace exactly:

def viterbi(tokens, tags, start, trans, emit):
    n = len(tokens)
    dp = [{} for _ in range(n)]
    bp = [{} for _ in range(n)]
    for tag in tags:
        dp[0][tag] = start[tag] + emit[tag][tokens[0]]
        bp[0][tag] = None
    for i in range(1, n):
        for tag in tags:
            best_score, best_prev = max(
                (dp[i-1][prev] + trans[prev][tag], prev) for prev in tags
            )
            dp[i][tag] = best_score + emit[tag][tokens[i]]
            bp[i][tag] = best_prev
    last_tag = max(tags, key=lambda t: dp[n-1][t])
    path = [last_tag]
    for i in range(n-1, 0, -1):
        path.append(bp[i][path[-1]])
    path.reverse()
    return path, dp[n-1][last_tag]

trans = {"O": {"O":1,"B-PER":0,"B-ORG":0},
         "B-PER": {"O":3,"B-PER":0,"B-ORG":0},
         "B-ORG": {"O":1,"B-PER":0,"B-ORG":0}}
start = {"O":1, "B-PER":2, "B-ORG":0}
emit = {"O":     {"Ravi":1, "leads":2, "Infosys":1},
        "B-PER": {"Ravi":5, "leads":0, "Infosys":0},
        "B-ORG": {"Ravi":0, "leads":3, "Infosys":6}}

path, score = viterbi(["Ravi","leads","Infosys"],
                       ["O","B-PER","B-ORG"], start, trans, emit)
print(path, score)
# ['B-PER', 'O', 'B-ORG'] 18

Running the DP loop by substitution: dp[0] = {"O":2, "B-PER":7, "B-ORG":0}, exactly as computed. At i=1, for tag="O" the generator yields (2+1,"O")=(3,"O"), (7+3,"B-PER")=(10,"B-PER"), (0+1,"B-ORG")=(1,"B-ORG"); max picks (10,"B-PER"), so dp[1]["O"] = 10+2 = 12, bp[1]["O"]="B-PER" — matching the hand trace at every step through i=2. The final max(tags, key=...) over dp[2] = {"O":14, "B-PER":12, "B-ORG":18} selects "B-ORG", and backtracking through bp reconstructs ["B-PER","O","B-ORG"] with total score 18.

From spans to structure: information extraction

NER answers "what entities are in this text." Information Extraction goes further: it links entities into typed relations and fills them into a structured record — the task that traces back to the 1990s DARPA-funded Message Understanding Conferences (MUC), which pioneered "template filling" as an evaluation task. Take the headline: "Tata Motors will invest ₹18,000 crore in its EV plant in Sanand, Gujarat by 2027." NER first produces spans: ORG="Tata Motors", MONEY="₹18,000 crore", LOC="Sanand", LOC="Gujarat", DATE="2027". IE then does three more things NER never attempts on its own:

Relation extraction — deciding which entities are connected and how. "Tata Motors" is not merely co-present with "₹18,000 crore"; it is the investor of that specific amount, and "Sanand, Gujarat" is the location of that investment, not an unrelated place mentioned in the sentence. A relation extractor typically uses the dependency structure of the sentence (which entity is the subject of "invest," which is the object of "in") to attach the right role to each entity, rather than just listing them.

Normalization — converting a text span into a canonical, machine-usable value. "₹18,000 crore" is meaningless to a database column expecting a number of rupees. One crore is 10⁷, so ₹18,000 crore = 18,000 × 10⁷ = 1.8 × 10¹¹ rupees (₹180,000,000,000). "2027" normalizes to a year field; "next Tuesday" (if it appeared) would need to resolve against the article's publication date to become an absolute calendar date.

Slot filling — assembling the normalized, related entities into one structured record:

{
  "investor": "Tata Motors",
  "action": "capital_expenditure",
  "amount_inr": 180000000000,
  "location": "Sanand, Gujarat",
  "target_year": 2027
}

This record, not the sentence, is what a downstream system — a trading algorithm, a search index, a knowledge graph — actually consumes. NER is the first and most reliable stage of this pipeline; everything after it (relation extraction, normalization, entity linking to a canonical company ID) compounds whatever error rate NER started with, which is why NER quality is scrutinised so closely in production IE systems.

How NER systems are actually built

Four generations of technique, each fixing a limitation of the last. Rule-based / gazetteer systems match tokens against curated lists of known names (a gazetteer) plus hand-written regular expressions for dates, money, and percentages. High precision on entities the list already contains, zero recall on anything new — a gazetteer of NSE-listed companies from 2020 has no entry for a company that IPO'd in 2025. Statistical sequence models (Hidden Markov Models, and later Conditional Random Fields) replaced fixed lookup with learned emission and transition scores, driven by hand-engineered features per token: capitalisation pattern, prefix/suffix, part-of-speech tag, and — still useful, now as one signal among many rather than the whole answer — gazetteer membership. This is precisely the emission/transition machinery worked through above, just with scores learned from labelled data instead of chosen by hand. Neural sequence models (BiLSTM-CRF) replaced hand-engineered features with learned word and character embeddings feeding a bidirectional LSTM, whose output still passes through a CRF layer for the transition scoring and Viterbi decoding — the neural network supplies better emission scores, the CRF still enforces sequence-level consistency. Transformer fine-tuning (BERT-style token classification) replaced the LSTM with contextual embeddings from a pretrained transformer, so that "Washington" carries a different vector depending on its surrounding words before tagging even begins; state-of-the-art systems still frequently keep a CRF/Viterbi output layer on top, because raw per-token argmax over transformer outputs can still produce the same greedy mistakes worked through earlier.

Measuring NER quality: precision, recall, and F1 at the entity level

NER is scored per entity, not per token — getting 90% of tokens right is meaningless if every multi-word entity has its boundary or type wrong. The standard (strict) rule: a predicted entity counts as a true positive only if both its span (exact token boundaries) and its type match a gold entity exactly. Take the sentence "Virat Kohli plays for Royal Challengers Bengaluru at Chinnaswamy Stadium." Gold entities: {(PER,"Virat Kohli"), (ORG,"Royal Challengers Bengaluru"), (LOC,"Chinnaswamy Stadium")}. Suppose a model predicts: {(PER,"Virat Kohli"), (ORG,"Royal Challengers Bengaluru"), (ORG,"Chinnaswamy Stadium")} — right spans throughout, but the stadium tagged as an organisation instead of a location.

The first two predictions match exactly (span and type): 2 true positives. The third predicts the correct span but the wrong type, so under strict matching it counts as both a false positive (a wrong (ORG,"Chinnaswamy Stadium") was output) and a false negative (the gold (LOC,"Chinnaswamy Stadium") was never produced). TP=2, FP=1, FN=1.

Precision = TP/(TP+FP) = 2/3 ≈ 0.667. Recall = TP/(TP+FN) = 2/3 ≈ 0.667. F1 = 2·P·R/(P+R) = 2·(2/3·2/3)/(4/3) = (8/9)/(4/3) = 2/3 ≈ 0.667. A model that gets every boundary right but confuses ORG and LOC is penalised exactly as if it had missed the entity outright — strict matching gives no partial credit for "close."

The misconception: "a gazetteer of names is enough"

The most common wrong mental model of NER is that it is a lookup problem: keep a big enough list of known people, companies, and places, and match tokens against it. This fails for a structural reason, not just a coverage reason. The same string routinely takes different entity types depending purely on context. "Washington" is B-PER in "George Washington crossed the Delaware," B-LOC in "protests reached Washington," and the first token of a B-ORG span in "the Washington Post reported." A single dictionary entry for the string "Washington" cannot resolve this — there is no type to assign to the string in isolation, because the string does not determine the type; the surrounding words do. This is exactly what "leads" in the worked Viterbi example illustrated from the emission side: a word's surface form only weakly suggests its tag, and transition context is often what breaks the tie correctly. Gazetteers remain genuinely useful — as one feature feeding a context-aware model, boosting confidence when a span does match a known name — but a system that relies on lookup alone cannot recognise any entity it has not already memorised, and cannot disambiguate the entities it has.

Active recall

Attempt each question before reading its answer.

1. Tag "Narayana Murthy founded Infosys in Pune in 1981 ." with BIO tags using types PER, ORG, LOC, DATE.

2. In the Viterbi example, suppose trans(B-PER→B-ORG) were changed from 0 to 5, with every other score unchanged. Recompute the two candidate paths into B-ORG at token 3 (via O and via B-PER at token 2) and state what happens to the decoding.

3. Why can't NER be solved by a dictionary lookup of known names? Give a concrete example.

4. Gold entities: {(PER,"Rohit Sharma"), (ORG,"BCCI"), (LOC,"Wankhede"), (DATE,"March 2026")}. Predicted: {(PER,"Rohit Sharma"), (ORG,"BCCI"), (LOC,"Mumbai")}. Compute precision, recall, and F1 under strict matching.

5. For the sentence "Reliance Industries acquired a 40% stake in the retail chain for ₹8,500 crore," write the NER output (typed spans) and then the IE output (a structured record with a normalized amount).

6. Why do transformer-based NER systems commonly keep a CRF/Viterbi layer instead of just taking the highest-scoring tag independently at each token?

Answers.

1. Narayana/B-PER, Murthy/I-PER, founded/O, Infosys/B-ORG, in/O, Pune/B-LOC, in/O, 1981/B-DATE, ./O.

2. The two routes the question names do tie: via O, dp[1][O] + trans(O→B-ORG) = 12 + 0 = 12; via B-PER, dp[1][B-PER] + trans(B-PER→B-ORG) = 7 + 5 = 12. But trans(B-PER→B-ORG) is a single shared matrix entry, reused at every step it applies to — including the token1→token2 transition itself, since token 1 is tagged B-PER. That means dp[1][B-ORG] also changes: it was max(2+0, 7+0, 0+0)+emit(B-ORG,'leads') = 7+3 = 10, and becomes max(2+0, 7+5, 0+0)+3 = 12+3 = 15. A third route into token 3's B-ORG — via token 2 = B-ORG — now scores dp[1][B-ORG]+trans(B-ORG→B-ORG) = 15+0 = 15, which strictly beats both routes the question asked about (12 and 12). This is not a tie at all: dp[2][B-ORG] = 15+emit(B-ORG,'Infosys') = 15+6 = 21, reached via token 2 = B-ORG, beating dp[2][O] = 17 and dp[2][B-PER] = 15. The corrected decoding is B-PER, B-ORG, B-ORG with total score 21 — a decisive change in the best path and its score, not an implementation-dependent tie-break at an unchanged score of 18. The lesson: because CRF/Viterbi transition matrices are shared parameters reused at every applicable step, changing one entry can ripple into candidate paths the question doesn't explicitly ask about — always re-check the full trellis, not just the two routes that seem obviously affected.

3. The same string can be different entity types depending on context, which a fixed list cannot resolve: "Washington" is a person in "George Washington," a location in "reached Washington," and part of an organisation in "the Washington Post." A list also cannot recognise any entity — a newly listed company, a newly appointed official — that postdates its compilation.

4. TP = 2 (Rohit Sharma, BCCI both match exactly). Predicted "Mumbai" matches no gold entity → FP = 1. Gold "Wankhede" and "March 2026" are both unmatched → FN = 2. Precision = 2/(2+1) = 2/3 ≈ 0.667. Recall = 2/(2+2) = 1/2 = 0.5. F1 = 2·(2/3·1/2)/(2/3+1/2) = 2·(1/3)/(7/6) = (2/3)·(6/7) = 12/21 = 4/7 ≈ 0.571.

5. NER: ORG="Reliance Industries", PERCENT="40%", ORG="the retail chain" (or a generic MISC/target-company tag), MONEY="₹8,500 crore". IE: normalizing ₹8,500 crore = 8,500 × 10⁷ = 8.5 × 10¹⁰ rupees. Record: {"acquirer": "Reliance Industries", "action": "stake_acquisition", "stake_percent": 40, "amount_inr": 85000000000}.

6. Independent per-token argmax ignores transition plausibility and can produce sequences that are individually locally-confident but globally wrong or structurally illegal (an I-ORG with no preceding B-ORG, or — as in the worked example — tagging "leads" as B-ORG because its emission score edges out O). The worked example showed this concretely: the greedy sequence scored 16 under the model's own scoring function, while the CRF/Viterbi decode found the sequence scoring 18, which was also the linguistically correct one. The CRF layer enforces sequence-level consistency that per-token classification cannot see.

Think About It

Think about this: How would you explain named entity recognition and information extraction to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind named entity recognition and information extraction, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.