Bhashini, India's National Language Translation Mission, exists because of a routing problem. A citizen files a grievance in Hindi on a government helpline; the backend case-management system, the analytics dashboards, and half the support staff work in English. Someone — or something — has to convert "मेरा राशन कार्ड रद्द कर दिया गया है, कृपया कारण बताएं" into "My ration card has been cancelled, please state the reason" in milliseconds, without a human translator in the loop, at the scale of hundreds of millions of queries. That something is a neural machine translation (NMT) system, and Hindi-English is the pair it has to get right most often, because the two languages sit on opposite sides of India's digital-public-infrastructure stack. This chapter builds the NMT pipeline that does this translation from first principles, traces one attention computation by hand down to the last decimal, and pins down exactly why Hindi and English are a harder pair than their shared vocabulary of loanwords suggests.
Why word-for-word substitution cannot work
Start with the simplest possible idea for translation: look up each source word in a dictionary and emit the target word in the same position. Take the Hindi sentence "राम सेब खाता है" — Ram, apple, eats-is (a compound verb marking present tense). Word-for-word substitution gives "Ram apple eats-is." The grammatical English sentence is "Ram eats apple" (or, with the article Hindi does not have, "Ram eats an apple"). The verb has moved from the end of the sentence to the middle, and a determiner has appeared out of nowhere.
This is not a translation quirk, it is a structural fact about the two languages. Hindi is a subject-object-verb (SOV) language: the verb is base-generated at the end of the clause, and case relationships that English signals with word order and prepositions are signalled in Hindi with postpositions and verb agreement instead. English is subject-verb-object (SVO). Any sentence longer than a couple of words therefore requires the target-side generator to look at source words that are not adjacent to the current output position, and often to look at a source word that appears much later in the sentence than the current position in the source-order timeline. A translation system that reads Hindi left to right and writes English left to right, one token at a time, in lockstep, physically cannot produce correct output for the common case, because the information needed to write the second English word (the verb) sits at the far end of the Hindi sentence. This is the specific engineering problem that attention was invented to solve, and it is the problem the rest of this chapter is built around.
The pipeline: tokenize, encode, attend, decode
A modern Hindi-English NMT system (IndicTrans2 from AI4Bharat, which powers much of Bhashini's Hindi-English pair, and the earlier Google NMT and OpenNMT systems before it, all share this skeleton) has four stages.
1. Subword tokenization. Text is not split on whitespace, because Devanagari script makes whitespace splitting a poor unit of meaning on both accounting axes that matter for a neural network: vocabulary size and rare-word coverage. A single Devanagari syllable can be a conjunct of two or three consonants sharing a virama (क्ष, त्र, ज्ञ), so the character-level alphabet is deceptively small but the surface-form vocabulary of "words" is enormous — inflection for case, gender, number, and postposition attachment (लड़के, लड़के को, लड़कों के लिए) multiplies every noun into many surface forms. A whitespace-word vocabulary would need millions of entries and would still hit unknown words constantly at test time. The fix used by every production system is a data-driven subword vocabulary — byte-pair encoding (BPE) or SentencePiece's unigram language model — trained on the parallel corpus to find a fixed inventory (typically 32,000–64,000 units) of frequent character sequences shared across both scripts. "खाता" might stay whole because it is frequent; a rare proper noun gets split into three or four subword pieces that the model has seen before even though the whole word never appeared in training. This is the same mechanism you met for English tokenization in your earlier NLP unit, applied here to a script where the payoff is larger because Devanagari's morphology is richer than English's.
2. Encoding. Each subword token is embedded into a vector, and a stack of self-attention layers (the Transformer encoder from your deep learning unit) lets every Hindi token look at every other Hindi token and update its representation with context — so the token for "खाता" (root of "eats") ends up encoding not just "eat" in isolation but "eat, third person, present habitual, agreeing with a masculine singular subject," because self-attention pulled that information in from "राम" and "है" elsewhere in the sentence. The output is one contextual vector per source subword: h₁, h₂, h₃, ….
3. Cross-attention. This is the mechanism that solves the reordering problem from the previous section. At every step of generating the English output, the decoder is not restricted to looking at the source token in the "corresponding" position — it computes a similarity score between its current state and every single encoder vector h₁…hₙ, turns those scores into a probability distribution with softmax, and takes a weighted average of all the encoder vectors as its "context" for that step. When the decoder needs to produce the English verb "eats," it can assign high weight to the Hindi verb's encoder vector even though that vector sits at the end of the source sentence and the decoder is only on its second output word. This is exactly the non-monotonic lookup that word-for-word substitution could not do.
4. Autoregressive decoding. The decoder generates the target sentence one subword at a time, feeding each generated token back in as input for the next step (with a causal mask so it cannot see future tokens it hasn't generated yet), until it emits an end-of-sentence marker.
Worked example: computing one attention step by hand
Toy dimensions make this traceable without a GPU. Suppose the encoder has already processed "राम सेब खाता-है" (Ram / apple / eats, with the compound verb treated as one token for simplicity) and produced three 2-dimensional hidden states — invented here for illustration, but structurally the kind of vectors a trained encoder would output, where the model has learned to put "eats" far apart from the two nouns along the second dimension:
h1 (राम / "Ram") = [1.0, 0.1]
h2 (सेब / "apple") = [0.1, 1.0]
h3 (खाता-है / "eats") = [0.9, -0.8]The decoder has already emitted "Ram" and is now computing its second output word. Its current hidden state — the query — is s₂ = [0.85, −0.75]. Dot-product attention scores each encoder vector against this query:
score(s2, h1) = 0.85(1.0) + (-0.75)(0.1) = 0.850 - 0.075 = 0.775
score(s2, h2) = 0.85(0.1) + (-0.75)(1.0) = 0.085 - 0.750 = -0.665
score(s2, h3) = 0.85(0.9) + (-0.75)(-0.8) = 0.765 + 0.600 = 1.365Softmax turns these three real numbers into a probability distribution. Using e ≈ 2.71828:
e^0.775 ≈ 2.171
e^-0.665 ≈ 0.514
e^1.365 ≈ 3.916
sum ≈ 6.601
a1 = 2.171 / 6.601 ≈ 0.329
a2 = 0.514 / 6.601 ≈ 0.078
a3 = 3.916 / 6.601 ≈ 0.593 (check: 0.329 + 0.078 + 0.593 = 1.000)These weights are the numbers a Transformer paper would draw as line thickness in an attention-visualization heatmap: 59.3% of the decoder's attention at this step lands on h₃ — the vector encoding the Hindi verb, sitting at the far end of the source sentence — even though the decoder is only two steps into generation and h₃ is the last of three source tokens. Only 32.9% lands on h₁ ("Ram," already translated) and a mere 7.8% on h₂ ("apple," not yet needed). The context vector for this step is the attention-weighted average of the three encoder states:
c2 = a1*h1 + a2*h2 + a3*h3
= 0.329*[1.0, 0.1] + 0.078*[0.1, 1.0] + 0.593*[0.9, -0.8]
= [0.329, 0.033] + [0.008, 0.078] + [0.534, -0.474]
= [0.871, -0.364]This context vector — dominated by h₃'s contribution because a₃ is largest — is concatenated with the decoder's own state and passed through the output layer, which assigns the highest probability to the subword "eats." Verifying this by direct computation (not just claiming it):
import numpy as np
h = np.array([[1.0, 0.1], [0.1, 1.0], [0.9, -0.8]])
s2 = np.array([0.85, -0.75])
scores = h @ s2
weights = np.exp(scores) / np.exp(scores).sum()
context = weights @ h
print(np.round(scores, 3)) # [ 0.775 -0.665 1.365]
print(np.round(weights, 3)) # [0.329 0.078 0.593]
print(np.round(context, 3)) # [ 0.871 -0.364]Running this script prints exactly those three lines — the hand computation and the code agree to three decimal places, which is the standard you should hold every NMT arithmetic claim to before trusting it.
Diagram: cross-attention reordering राम सेब खाता है → Ram eats apple
Common misconception
Students who have used a phrasebook or a bilingual dictionary tend to assume neural MT is a smarter, statistical version of the same idea: match source words to target words, roughly in the order they occur, and smooth over the grammar. The worked example above shows precisely why this is wrong for Hindi-English. There is no "corresponding position" for the decoder to look at — the attention mechanism exists specifically because the correct source information for the 2nd English output word is the 3rd Hindi input token. A system constrained to monotonic (order-preserving) alignment — which is what pre-2014 phrase-based statistical MT effectively assumed, with a limited reordering window — degrades badly on exactly this SOV-to-SVO pair, which is one of the concrete historical reasons NMT with unrestricted attention replaced phrase-based SMT for Hindi rather than merely improving on it.
What is specifically hard about Hindi-English, beyond word order
Three further asymmetries between the languages matter for how the model has to be trained and where it fails, and none of them are generic "AI translation is hard" observations — they are properties of this specific language pair.
Gender agreement with no source signal. Hindi verbs and adjectives inflect for the grammatical gender of the subject: "राम खाता है" (Ram eats, masculine — खाता) versus "सीता खाती है" (Sita eats, feminine — खाती). English "eats" carries no gender. This means English-to-Hindi translation is the harder direction of the pair in a specific, measurable way: the model must correctly resolve the subject's gender from context (a proper noun's typical gender, an earlier pronoun, world knowledge) that the English source sentence does not encode at the point of the verb, and it will silently guess wrong on ambiguous or gender-neutral inputs like "the doctor eats lunch," producing a grammatically well-formed but potentially incorrect Hindi sentence.
Register collapse. Hindi distinguishes three second-person forms with different verb conjugations and social meaning — तू (intimate, or an insult if misused), तुम (familiar), आप (respectful/formal) — where English has only "you." Hindi-to-English translation loses this distinction for free (all three collapse to "you," which is usually fine). English-to-Hindi translation must invent a register the source sentence never specified; a model trained mostly on formal parallel text (news, government documents, court judgments) will default to आप even in a casual chat context, and a model trained on informal social-media parallel data will make the opposite error. This is a genuine, unresolvable-from-the-source ambiguity, not a training bug to be fixed by more data alone.
Case marking via postpositions. Where English uses word order and prepositions to mark grammatical roles ("to Ram," "from the station"), Hindi attaches postpositions after the noun and often changes the noun's oblique form: "राम" becomes "राम को" (dative, "to Ram"), "स्टेशन" becomes "स्टेशन से" (ablative, "from the station"). The encoder's subword tokenizer must learn to treat "को," "से," "में," "के लिए" as near-function-word units carrying grammatical rather than lexical weight — get this wrong and the model conflates "Ram gave [something] to Sita" with "Ram gave [something] from Sita," a genuinely different and sometimes legally or factually important sentence in a judgment-translation context like the one the SUVAS-class systems referenced earlier are built for.
Evaluating the output: BLEU, and why it under-counts Hindi-English errors
Translation quality is scored automatically with BLEU (bilingual evaluation understudy), which measures n-gram overlap between the system's candidate translation and one or more human reference translations, multiplied by a brevity penalty that punishes outputs shorter than the reference. Trace one BLEU-2 computation by hand. Reference: "the court found the accused guilty" (6 tokens). Candidate (system output, missing the second "the"): "the court found accused guilty" (5 tokens).
Unigram precision: every candidate token — the, court, found, accused, guilty — appears in the reference (clipped to the reference's own count), so matched = 5 out of 5 candidate tokens: p1 = 5/5 = 1.0. Bigram precision: candidate bigrams are (the,court), (court,found), (found,accused), (accused,guilty) — four bigrams. The reference's bigrams are (the,court), (court,found), (found,the), (the,accused), (accused,guilty). Three of the candidate's four bigrams match; (found,accused) does not appear in the reference, which instead has (found,the) and (the,accused). So p2 = 3/4 = 0.75. Brevity penalty, since candidate length c = 5 is shorter than reference length r = 6: BP = e^(1 − r/c) = e^(1 − 1.2) = e^(−0.2) ≈ 0.8187. Combined BLEU-2 = BP × √(p1 × p2) = 0.8187 × √0.75 ≈ 0.8187 × 0.8660 ≈ 0.709.
A single dropped article costs almost 30 points of BLEU-2 on an otherwise perfectly correct, perfectly fluent translation. This is not a corner case for Hindi-English: Hindi has no articles at all — no equivalent of "a" or "the" — so an English target sentence generated from Hindi source must invent definiteness from context every single time, and a reference translator and the model will systematically disagree on exactly these function words even when both translations are equally correct English. This is why Hindi-English BLEU scores are structurally depressed relative to, say, French-English scores, independent of how good the underlying model actually is, and why a SUVAS-class legal translation system cannot be evaluated by BLEU alone — a translation that flips को (to) for से (from) in a judgment can score nearly identically on BLEU to the correct version, because both differ from the reference by a single low-weight function word, while carrying completely different legal meaning. High n-gram overlap is not the same claim as "safe to publish as an official court translation," and a curriculum or a production pipeline that treats a BLEU number as a pass/fail gate for this language pair is trusting a metric past what it can support.
Active recall
Attempt each question before reading its answer.
Q1. Why does a translation system that reads Hindi left-to-right and writes English left-to-right, one token per source token, fail structurally on Hindi-English — not just occasionally, but as a rule?
Q2. Given encoder hidden states h1 = [0.6, 0.4], h2 = [−0.2, 0.9], h3 = [1.1, −0.3] and decoder query s = [0.9, −0.6], compute the three attention weights by hand (dot-product scores, then softmax).
Q3. Give one specific reason (not "Hindi is complex") that Hindi text is subword-tokenized rather than whitespace-tokenized before being fed to the encoder.
Q4. Why is English-to-Hindi translation of the sentence "the doctor eats lunch" a genuinely harder problem than Hindi-to-English translation of "डॉक्टर खाना खाती है," even though both describe the same fact?
Q5. Reference: "the police arrested the driver" (5 tokens). Candidate: "police arrested driver" (3 tokens, both instances of "the" dropped). Compute unigram BLEU precision and the brevity penalty.
Q6. A SUVAS-class system scores a translated judgment at BLEU 0.91 against a human reference, and a court clerk is asked whether it is safe to publish without review. What is wrong with answering "yes" from the BLEU score alone?
A1. Hindi is SOV (verb at the end of the clause) and English is SVO (verb after the subject). Producing the 2nd English word correctly routinely requires information from a source token that is not 2nd, and can be last, in the Hindi sentence. A strictly monotonic, one-token-per-step system has no mechanism to reach ahead in the source sentence, so it fails on the common case, not the rare one — this is exactly why cross-attention, which lets every decoder step look at every encoder position, was necessary rather than optional.
A2. Scores: s·h1 = 0.9(0.6) + (−0.6)(0.4) = 0.54 − 0.24 = 0.30. s·h2 = 0.9(−0.2) + (−0.6)(0.9) = −0.18 − 0.54 = −0.72. s·h3 = 0.9(1.1) + (−0.6)(−0.3) = 0.99 + 0.18 = 1.17. Exponentials: e^0.30 ≈ 1.350, e^−0.72 ≈ 0.487, e^1.17 ≈ 3.222; sum ≈ 5.059. Weights: a1 ≈ 1.350/5.059 ≈ 0.267, a2 ≈ 0.487/5.059 ≈ 0.096, a3 ≈ 3.222/5.059 ≈ 0.637 (sum ≈ 1.000). h3 again dominates the decoder's attention.
A3. Devanagari morphology multiplies a single noun or verb root into many inflected surface forms (case, gender, number, postposition attachment — लड़का, लड़के, लड़के को, लड़कों के लिए), so a whitespace/word-level vocabulary would need millions of entries and would still fail on unseen inflected forms at test time; a trained subword vocabulary (BPE/SentencePiece) reuses shared substrings across inflections and keeps rare or unseen words representable as a sequence of known pieces.
A4. Hindi verbs and adjectives agree in grammatical gender with the subject (खाता for masculine, खाती for feminine). "डॉक्टर खाना खाती है" already fixes the doctor's gender as feminine in the source sentence, so Hindi-to-English just drops that information (English "eats" is gender-neutral — an easy, information-discarding direction). Going the other way, "the doctor eats lunch" gives the model no gender signal at all, so it must guess खाता or खाती from world knowledge or context that may not exist, and can silently produce a wrong or biased translation with no way for the model to flag the ambiguity.
A5. Candidate tokens (police, arrested, driver) all appear in the reference, so unigram precision p1 = 3/3 = 1.0. Brevity penalty: c = 3, r = 5, BP = e^(1 − 5/3) = e^(−0.667) ≈ 0.513. Even with perfect precision, BLEU-1 = BP × p1 ≈ 0.513 — dropping both articles roughly halves the score, illustrating how harshly BLEU's brevity penalty treats the systematic Hindi-has-no-articles asymmetry.
A6. BLEU measures n-gram overlap with one reference translation; it weights every token roughly equally and cannot distinguish a dropped article from a flipped case-marking postposition (को vs से) that reverses who gave what to whom. A judgment translation can score 0.91 while containing exactly the kind of single-word legal-meaning error described in the postposition section — the metric would barely register it. BLEU is evidence of fluency and rough adequacy, not a certification of legal correctness; a high score narrows where a human reviewer needs to look, it does not remove the need for one.
Think About It
Think about this: How would you explain machine translation: hindi-english neural mt to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where machine translation: hindi-english neural mt is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting machine translation: hindi-english neural mt to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind machine translation: hindi-english neural mt, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.