Sentiment Analysis for Indian Languages

Picture the backend team at a food-delivery app that operates across India. Every night, a batch job pulls in the day's restaurant reviews and has to tag each one positive, negative, or neutral, so that operations can flag restaurants sliding in quality before ratings collapse. The team already has a sentiment classifier — it was trained on a large English movie-review dataset and gets high accuracy on English test sets. They plug it in. It quietly fails, not by crashing, but by returning confident, wrong answers on reviews like "khana bahut tasty tha but delivery bohot late ho gaya 😡" or "order thik tha, kuch khaas nahi". Neither sentence is in English. Neither is purely in Hindi either — both switch between Hindi words spelled in the Roman alphabet and English words, mid-sentence, sometimes mid-clause. This is not an edge case in the Indian market; for review, chat, and comment text from Indian users, this kind of code-mixing is closer to the default than the exception. Understanding why an English-tuned model breaks here, and what actually has to change to fix it, is the substance of this chapter.

What a sentiment classifier is actually computing

Strip away the branding around "sentiment analysis" and it is a text classification problem: map a sequence of tokens to one of a small set of labels — typically positive, negative, and neutral. Every approach, from a 1990s bag-of-words model to a 2020s transformer, does the same two things: turn text into numbers (features), then learn a function from those numbers to a label using labeled training examples. The oldest and most transparent version of that function is Naive Bayes, and it is worth working through by hand once, because its failure modes are exactly the failure modes that motivate everything modern Indian-language NLP systems do differently.

Naive Bayes classifies a document d into class c by picking the class that maximizes P(c) × ∏ P(word_i | c) — the prior probability of the class, times the product of the probability of each word given that class, treating words as conditionally independent (the "naive" assumption). Because a word that never appeared in a class's training data would otherwise force that whole product to zero, every implementation adds Laplace (add-one) smoothing: P(word|c) = (count(word,c) + 1) / (total_words_in_c + V), where V is the size of the full vocabulary across all classes.

Worked example: training and testing on a code-mixed review corpus

Take a toy training set of six food-delivery reviews, three per class, written the way Indian users actually write them — mixed Hindi and English, Roman script:

Every word in the positive set occurs exactly once, across nine total tokens: khana, tasty, tha, bahut, accha, service, amazing, food, quality. The negative set has twelve total tokens across three documents (four, five, and three words respectively): khana, bahut, bakwas, tha, delivery, late, aur, cold, food, worst, service, ever — also each occurring once. The union of both vocabularies has 16 distinct words (nine from the positive set, plus seven — bakwas, delivery, late, aur, cold, worst, ever — that appear only on the negative side), so V = 16. With three documents in each class, the prior is P(pos) = P(neg) = 0.5.

Now classify a new, unseen review: "khana accha tha but delivery late". Tokenizing on whitespace gives six tokens, but "but" never appeared in training, in either class — it is out-of-vocabulary (OOV) for this tiny model, so a whole-word Naive Bayes simply drops it and scores only the remaining five: khana, accha, tha, delivery, late. Applying Laplace smoothing with the class totals above (positive: 9 tokens, negative: 12 tokens, both over V = 16):

Multiplying across the row for positive: (2×2×2×1×1) / 25⁵ = 8 / 9,765,625 ≈ 8.192×10⁻⁷, times the 0.5 prior gives a positive score of 4.096×10⁻⁷. For negative: (2×1×2×2×2) / 28⁵ = 16 / 17,210,368 ≈ 9.297×10⁻⁷, times 0.5 gives 4.648×10⁻⁷. The negative score is larger — the model predicts negative, despite the sentence opening with "food was good." Here is the code that reproduces this exactly, so you can check the arithmetic by running it rather than trusting it:

from collections import Counter

pos_docs = ["khana tasty tha", "bahut accha service", "amazing food quality"]
neg_docs = ["khana bahut bakwas tha", "delivery late aur cold food", "worst service ever"]

pos_counts = Counter(" ".join(pos_docs).split())
neg_counts = Counter(" ".join(neg_docs).split())
vocab = set(pos_counts) | set(neg_counts)
V = len(vocab)
pos_total = sum(pos_counts.values())
neg_total = sum(neg_counts.values())

def word_prob(word, counts, total):
    return (counts.get(word, 0) + 1) / (total + V)

test = [w for w in "khana accha tha but delivery late".split() if w in vocab]

pos_score, neg_score = 0.5, 0.5
for w in test:
    pos_score *= word_prob(w, pos_counts, pos_total)
    neg_score *= word_prob(w, neg_counts, neg_total)

print(f"V={V}, pos_total={pos_total}, neg_total={neg_total}")
print(f"pos_score={pos_score:.4e}, neg_score={neg_score:.4e}")
print("negative" if neg_score > pos_score else "positive")

This prints V=16, pos_total=9, neg_total=12 and pos_score=4.0960e-07, neg_score=4.6484e-07, then negative — matching the hand derivation exactly. The point of tracing this is not that Naive Bayes is a bad algorithm; it is that the failure is structural, not accidental. The sentence has two clauses with opposite sentiment about two different aspects (food: good, delivery: late), and a bag-of-words model has no notion of "which clause a word belongs to" — it multiplies word probabilities as if they were independent evidence for one global label, so a strong negative word anywhere in the sentence can outvote a strong positive word elsewhere, with no mechanism to recognize they are talking about different things. This exact weakness is why aspect-mixed reviews are one of the hardest classes of input for any single-label sentiment system, English or Indian-language alike — but it compounds with several problems that are specific to Indian-language text, covered next.

Why English-tuned techniques specifically break on Indian-language text

Four distinct problems stack on top of the aspect-mixing issue above, and each demands a different fix:

Script and language mixing. A single sentence can carry Hindi words in Roman script, English words, and occasionally Devanagari script itself, switching mid-clause. A model whose vocabulary was built from English text alone treats every Hindi-origin token — accha, bakwas, thik, bilkul — as unknown, and an unknown token carries zero learned sentiment signal. In the worked example, if the training corpus had been purely English movie reviews, khana, accha, tha, bahut, bakwas would all be OOV; the model would only ever see whichever English words happened to survive in a given review, discarding what is frequently the most sentiment-bearing vocabulary in the sentence.

Transliteration variance. There is no single agreed Roman spelling for most Hindi words. "Good" can appear as accha, acha, achha, acchha — four surface forms of one word, each of which a whole-word vocabulary treats as a completely separate, unrelated token with its own (likely near-zero) training signal. A model needs to see enough examples of every spelling variant independently, which no realistic dataset provides.

Word order and negation scope. English sentiment heuristics that flip the polarity of the next word or two after "not" rely on English being Subject-Verb-Object with the negator placed right before the word it negates ("not good"). Hindi is Subject-Object-Verb, and its negator "nahi" typically sits near the verb, at the end of the clause — after the adjective it logically negates, not before it. "khana accha nahi tha" is literally "food good not was," meaning "the food was not good." A rule that scans backward from "nahi" looking for the next adjective, the way an English negation-handler scans forward from "not," looks in exactly the wrong direction and either misses the negation or attaches it to the wrong word.

Low resource availability. Large labeled sentiment datasets are abundant in English and scarce for most of India's other scheduled languages — a classifier for Odia or Assamese social-media sentiment cannot be trained the way an English one is, simply because the labeled examples do not exist at the same scale. This is the practical reason the field moved toward transfer learning rather than training a fresh model per language from scratch.

How production systems actually address this

Two changes, used together, solve most of the problems above without needing millions of labeled examples per language. The first is subword tokenization (byte-pair encoding, BPE, or similar). Instead of treating each whitespace-separated word as one atomic unit, a subword tokenizer learns a vocabulary of frequent character sequences from a large corpus and represents any word as a sequence of these pieces. Because "accha," "acha," and "achha" share almost all their characters, they decompose into overlapping subword pieces rather than becoming three unrelated out-of-vocabulary tokens — sentiment signal learned from one spelling partially transfers to the others through the shared pieces, and a word the tokenizer has genuinely never seen still gets broken into recognizable fragments instead of collapsing to a single unknown-token .

The second is pretraining a shared multilingual encoder before ever touching the sentiment task. Models such as Google's MuRIL and AI4Bharat's IndicBERT are pretrained with a masked-language-modeling objective — predict a randomly hidden word from its surrounding context — over large corpora that mix native-script text, Romanized (transliterated) text, and English, across more than a dozen Indian languages simultaneously. Because words that play the same role in similar contexts get pulled toward similar vectors during this pretraining, "अच्छा" (Devanagari), "accha" (Roman Hindi), "ভালো" (Bengali), and "good" (English) end up near each other in the model's embedding space — not because anyone told the model they are synonyms, but because they occur in statistically similar contexts across the pretraining corpus. Only after this shared space exists does fine-tuning happen: a small classification head is trained on whatever labeled sentiment data is available (often concentrated in one or two languages), and because the encoder underneath already aligns meaning across scripts, sentiment knowledge learned mostly from Hindi and English reviews transfers usefully to Bengali or Marathi reviews the classifier saw few or none of during fine-tuning. The diagram below shows both halves of this pipeline together.

The misconception: "just machine-translate to English first"

The most common instinct, on first meeting this problem, is: why build a multilingual model at all — why not run every review through a machine translation system into English, then use the excellent, well-tested English sentiment classifiers everyone already has? This sounds efficient and is wrong for reasons that are specific to this domain, not generic anti-translation caution. Machine translation systems are themselves trained mostly on relatively formal, well-punctuated parallel text. Casual, code-mixed, misspelled review text — exactly the input this pipeline receives — is precisely the distribution MT systems handle worst; a mistranslation of the sentiment-bearing word is now a compounding error baked into the input the downstream classifier never gets a chance to correct. Translation also has to resolve the word-order and negation-scope differences discussed above during the translation step, silently, with no visibility into whether it did so correctly — a translation that reorders "khana accha nahi tha" incorrectly hands the English classifier a sentence whose negation has already been mangled, and the classifier's output is now wrong for a reason invisible at the classification stage. And operationally, translate-then-classify means running two large models in sequence for every single review, doubling latency and infrastructure cost, when a single model fine-tuned directly on native-language and code-mixed labeled data does the same job in one pass, without a translation error sitting silently between the customer's words and the final label.

Active recall

Attempt these before reading the answers below.

1. An English-only Naive Bayes sentiment classifier is deployed, unmodified, on Hindi-English code-mixed reviews. What happens to most Hindi-origin words at inference time, and why does that matter?
2. Using the worked example's training corpus, a seventh negative-class document, "khana thanda tha" (food was cold), is added. Recompute P(khana | neg) with Laplace smoothing after this addition.
3. Why does the Hindi negator "nahi" break a negation-handling rule ported directly from English?
4. Why does subword (BPE) tokenization specifically help with transliteration spelling variants like "accha" / "acha" / "achha"?
5. In the embedding-space diagram, "अच्छा", "accha", and "ভালো" cluster together even though the sentiment fine-tuning stage never saw a dictionary mapping between them. At which stage of training does this alignment actually get learned?
6. A review reads: "service to bahut acchi thi, par khana bilkul bakwas tha" (the service was very good, but the food was absolutely terrible). Would one global positive/negative/neutral label capture the useful signal here? What would you use instead?

Answers.

1. Every Hindi-origin word — accha, bakwas, thik, bilkul, and so on — is out-of-vocabulary for a model trained only on English text, since it never appeared, in any spelling, in training. A whole-word model has no representation for it at all, so it is either dropped or mapped to a generic "unknown" that carries no learned sentiment. This matters because the words dropped are frequently the most sentiment-bearing words in the sentence — the classifier ends up making its decision based only on whatever English words happen to remain, which for a majority-Hindi review can be almost nothing.

2. The original negative class had 12 total tokens, and "khana" occurred once. Adding "khana thanda tha" contributes three new tokens, one of which — "khana" — repeats, bringing its count to 2; "thanda" is entirely new to the shared vocabulary, so V rises from 16 to 17, and the negative class total rises from 12 to 15. Applying Laplace smoothing: P(khana|neg) = (2 + 1) / (15 + 17) = 3/32 = 0.09375.

3. English places its negator directly before the word it negates ("not good"), so a heuristic that scans a short window backward from a sentiment-bearing word for "not" works. Hindi is Subject-Object-Verb, and "nahi" sits near the verb at the end of the clause — after the adjective it logically negates, not before it, as in "accha nahi tha" (good not was → "wasn't good"). A rule built for English's word order looks in the wrong direction and misses the negation entirely.

4. BPE builds its vocabulary from frequent character sequences observed in a large corpus, not from whole words. "accha," "acha," and "achha" share almost all of their characters, so they decompose into overlapping subword pieces rather than three unrelated whole-word tokens. Sentiment signal learned about one spelling's subword pieces is then automatically available when a different spelling of the same word appears, because the pieces — not the exact spelling — carry the learned representation.

5. This alignment emerges during pretraining, specifically from the masked-language-modeling objective run over a large multilingual and transliterated corpus — the model learns to predict words from context long before it ever sees a sentiment label, and words used in similar contexts across languages get pulled toward similar vectors purely from that statistical pressure. Fine-tuning only adjusts a small classification head (and lightly the encoder) on top of an embedding space that is already cross-lingually aligned; it does not create that alignment.

6. No — a single global label collapses two opposite polarities about two different things into one number, discarding exactly the information an operations team would act on (the food, not the service, is the problem). The right tool is aspect-based sentiment analysis (ABSA), which first identifies the aspect terms in the review (service, khana/food) and then predicts a separate polarity for each aspect rather than one polarity for the whole review.

Think About It

Think about this: How would you explain sentiment analysis for indian languages to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where sentiment analysis for indian languages is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting sentiment analysis for indian languages to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind sentiment analysis for indian languages, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.