Tokenizer Design: BPE and SentencePiece Explained

Why a Hindi sentence costs more tokens than its English translation

Take two sentences that say the same thing. English: "India launched a satellite." — 27 characters, and because every character is plain ASCII, also exactly 27 bytes in UTF-8. Now the Hindi equivalent: "भारत ने एक उपग्रह लॉन्च किया।" — 29 characters, but 77 bytes. Same meaning, almost the same character count, nearly three times the bytes. That gap is not an accident of this one sentence; it is baked into the Unicode standard itself. The Devanagari block (U+0900–U+097F, the range Hindi, Marathi, and Sanskrit are written in) sits above U+0800, and UTF-8's encoding rule requires three bytes for every code point from U+0800 to U+FFFF, against one byte for the ASCII range every English letter lives in. Run len('भ'.encode('utf-8')) and you get 3; run it on 'a' and you get 1. That is a measurable, checkable fact about the encoding, not a claim about any particular tokenizer — and dividing the two sentences' byte counts gives a ratio of 77/27 ≈ 2.85.

Why open a chapter on tokenizer design with byte counts instead of tokens? Because most production tokenizers used by large language models — including the byte-level BPE tokenizers behind the GPT family — operate directly on UTF-8 bytes, not on Unicode characters. Their merge rules are learned from a training corpus, and if that corpus is dominated by English and other Latin-script web text, the tokenizer accumulates thousands of merge rules that fold common English byte sequences — "ing", "tion", " the" — into single tokens, while Devanagari, Tamil, or Bengali byte sequences, being rarer in that same mix, get far fewer merges. English text ends up compressed to well below one token per byte; Hindi text, under a tokenizer with few Devanagari-specific merges, stays close to its raw byte count of three bytes per character. The 2.85x byte-length ratio measured above is therefore a floor on the real token-count penalty, not the actual figure — the true gap is typically wider once you account for how unevenly the merge budget was spent during training, a point the active-recall section returns to with the full argument. This is exactly the problem that motivates AI4Bharat's and Sarvam AI's Indic-first tokenizers: for a model serving a genuinely multilingual India — an IRCTC assistant, a Digital Public Infrastructure chatbot, a Bhashini translation pipeline — the same fixed context window and the same per-token price buy noticeably less usable Hindi, Tamil, or Kannada than English, purely as a function of how the tokenizer's vocabulary was built. Understanding BPE and SentencePiece is understanding exactly where that asymmetry comes from, and how to design a tokenizer that doesn't have it.

From words to characters to subwords: the design space

Before a transformer sees a sentence, something has to turn it into a fixed, finite set of integer ids. Three obvious choices sit across a spectrum, each with a distinct failure mode.

Word-level vocabularies — one id per distinct word — feel natural but break in two ways. First, natural language has an effectively unbounded set of word forms: English alone has hundreds of thousands of surface forms once inflections, compounds, and named entities are counted, and morphologically rich languages like Hindi or Tamil multiply this further through productive affixation. Any fixed vocabulary either grows enormous or leaves gaps. Second, any word absent from the vocabulary must be replaced by an out-of-vocabulary — the model sees an identical, uninformative token for "photosynthesis," "Aatmanirbhar," and a typo, with no way to recover what was actually written.

Character-level vocabularies solve the out-of-vocabulary problem completely — every string is just a sequence of a few hundred characters, or, at the byte level, exactly 256 values — but pay for it in sequence length. An eight-word sentence becomes roughly forty to fifty characters; self-attention costs quadratically in sequence length, and every character-level token carries far less information than a word-level one, forcing the model to spend capacity re-learning that "t-h-e" is a single unit before it can learn anything about what "the" means in context.

Subword tokenization is the compromise every modern LLM actually uses: a vocabulary of a few tens of thousands of variable-length character fragments, built automatically from a training corpus, chosen so common words stay whole while rare or unseen words fall back to smaller, still-reusable pieces. Byte Pair Encoding and SentencePiece are the two dominant algorithms for constructing that vocabulary, and they differ in exactly how they decide which fragments earn a slot.

Byte Pair Encoding: from data compression to neural vocabularies

BPE did not start as an NLP idea. Philip Gage described it in 1994 in The C Users Journal as a general-purpose data-compression trick: repeatedly find the most frequent adjacent pair of bytes in a file and replace every occurrence with one unused byte value, building a small substitution table you replay to decompress. Twenty-two years later, Rico Sennrich, Barry Haddow, and Alexandra Birch repurposed the same repeated-merge idea for an entirely different goal in their 2016 ACL paper Neural Machine Translation of Rare Words with Subword Units: instead of compressing a file, use the merge counts to build a subword vocabulary for a neural machine translation model, so rare and unseen words could be represented as sequences of frequent subword units instead of a single out-of-vocabulary token. That 2016 paper is the direct ancestor of every BPE tokenizer in production use today. OpenAI's GPT-2 (Radford et al., 2019) pushed one step further: instead of running BPE over Unicode characters, it runs over raw UTF-8 bytes, so the base vocabulary is fixed at exactly 256 symbols and no input string — any script, any emoji, any malformed text — can ever produce an unrepresentable character. This byte-level choice is why GPT-family tokenizers never emit an unknown-token .

The training algorithm has four steps, repeated until a target vocabulary size is reached:

1. Represent every word in the training corpus as a sequence of base symbols (characters, or bytes), with a word-boundary marker (this chapter uses "_") so the tokenizer can tell an "er" inside "server" from the "er" that ends "newer".
2. Count how often every adjacent pair of symbols occurs, summed across the whole corpus, weighted by each word's frequency.
3. Merge the single most frequent pair into one new symbol, and record that merge as a numbered rule.
4. Replace every occurrence of that pair throughout the corpus with the new merged symbol, and return to step 2.

Two details matter for correctness. Counting is corpus-wide and frequency-weighted — a pair inside a word occurring ten thousand times outweighs the same pair inside a word occurring twice, even though both are "the same pair." And the ordered list of merge rules produced during training is exactly what gets replayed at inference time: to tokenize a brand-new word, start from its base symbols and apply the learned rules in the order they were learned, merging wherever each rule's pattern appears, until no more rules apply. A word never seen during training can still tokenize cleanly, because it is built from pieces the training corpus did teach the tokenizer to recognize.

Worked example: training BPE on a four-word corpus, merge by merge

Take a training corpus of four words with their frequencies: low (5), lowest (2), newer (6), wider (3). Split each into characters plus an end-of-word marker "_":

low_     → l o w _            (freq 5)
lowest_  → l o w e s t _      (freq 2)
newer_   → n e w e r _        (freq 6)
wider_   → w i d e r _        (freq 3)

Step 1: count every adjacent pair across the corpus, weighted by frequency. The pair (e, r) occurs in "newer_" (weight 6) and "wider_" (weight 3), totalling 9. The pair (r, _) occurs in the same two words, also totalling 9 — a genuine tie. Real BPE implementations resolve ties with a fixed, deterministic rule; this worked example breaks ties by choosing the lexicographically smaller pair (exactly what the code below does), which picks (e, r). Merge rule #1: e + r → er, count 9. Every merge below was computed by running the training loop, not asserted by hand:

After exactly six merges, the four training words tokenize as:

low_     → [low, _]
lowest_  → [low, e, s, t, _]
newer_   → [n, ewer_]
wider_   → [w, i, d, er_]

Here is a minimal, runnable implementation of exactly this loop — executed to produce the merge list above, not just described:

from collections import Counter

def train_bpe(corpus, n_merges):
    vocab = {tuple(list(w) + ["_"]): f for w, f in corpus.items()}
    merges = []
    for _ in range(n_merges):
        pairs = Counter()
        for symbols, freq in vocab.items():
            for i in range(len(symbols) - 1):
                pairs[(symbols[i], symbols[i + 1])] += freq
        if not pairs:
            break
        top = max(pairs.values())
        a, b = sorted(p for p, c in pairs.items() if c == top)[0]
        merges.append(((a, b), top))
        new_vocab = {}
        for symbols, freq in vocab.items():
            merged, i = [], 0
            while i < len(symbols):
                if i < len(symbols) - 1 and symbols[i] == a and symbols[i + 1] == b:
                    merged.append(a + b); i += 2
                else:
                    merged.append(symbols[i]); i += 1
            new_vocab[tuple(merged)] = freq
        vocab = new_vocab
    return merges, vocab

corpus = {"low": 5, "lowest": 2, "newer": 6, "wider": 3}
merges, vocab = train_bpe(corpus, n_merges=6)
print(merges)
# [(('e', 'r'), 9), (('er', '_'), 9), (('l', 'o'), 7),
#  (('lo', 'w'), 7), (('e', 'w'), 6), (('ew', 'er_'), 6)]
print(vocab)
# {('low', '_'): 5, ('low', 'e', 's', 't', '_'): 2,
#  ('n', 'ewer_'): 6, ('w', 'i', 'd', 'er_'): 3}

To encode a word the tokenizer has never seen, apply the six learned rules, in order, to its character sequence. Take "lower" — not one of the four training words:

start:             l  o  w  e  r  _
rule 1  e+r→er:    l  o  w  er  _
rule 2  er+_→er_:  l  o  w  er_
rule 3  l+o→lo:    lo  w  er_
rule 4  lo+w→low:  low  er_
rule 5  e+w:        (no e,w adjacent — no change)
rule 6  ew+er_:      (no ew present — no change)
final tokens:      [low, er_]

"lower" was never in the training corpus, yet it tokenizes into two clean pieces, both reused from words the tokenizer did see. That reuse — recombining a small set of learned fragments instead of memorizing whole words — is the entire point of subword tokenization.

The misconception: "BPE merges are morphemes"

The most common wrong mental model of BPE is that it discovers linguistically meaningful units — prefixes, suffixes, roots — the way a linguist would segment a word. The worked example above is a direct counter-example. "lowest" has an obvious, linguistically real morpheme boundary: "low" + "est", the superlative suffix, the same one in "widest" or "newest". A tokenizer that "understood" morphology would merge "est" into a single unit. Ours does not. Trace the final tokenization again: lowest_ → [low, e, s, t, _] — four fragments after "low", not the two a linguist would produce. Why? "lowest" occurs only twice in this training corpus, so the pair (e, s) inside it never accumulates enough corpus-wide frequency to win a merge before the six-merge budget runs out — it is permanently outranked by pairs like (e, r) and (l, o) that happen to occur in the higher-frequency words "newer" and "low". Meanwhile "er_", which crosses what a linguist would consider a word-boundary-adjacent fragment plus the artificial "_" marker, does get merged early — purely because "newer" and "wider" together push that specific pattern's count to 9.

BPE, and SentencePiece's BPE mode, has no concept of morphology, syntax, or meaning. It has exactly one signal — how often a pair of adjacent symbols co-occurs in the training corpus — and it always spends its limited merge budget on whatever pair is currently most frequent, with no regard for whether the resulting unit corresponds to anything a human would call meaningful. This matters practically: a tokenizer's vocabulary is a direct, sometimes brittle function of the corpus it was trained on. Swap the training corpus — English news text versus Indian social-media code-mixed text versus legal documents — and the same target word can split completely differently, with no guarantee that "sensible" morpheme boundaries survive the swap.

SentencePiece: removing the whitespace assumption

Sennrich et al.'s original BPE implementation, and most classic subword tokenizers built on it, assume the input has already been split into words by a separate, language-specific pre-tokenizer — typically "split on whitespace and punctuation" — before BPE ever runs. That assumption is invisible and free for English, and actively broken for languages that mark no word boundaries with spaces at all, such as Chinese, Japanese, and Thai; those pipelines had to bolt on a separate segmenter (MeCab for Japanese, jieba for Chinese) per language before BPE could even start, and getting detokenization exactly right — remembering which side of a merged token used to have a space — turned into ad hoc, language-specific bookkeeping.

Taku Kudo and John Richardson's SentencePiece (Kudo & Richardson, EMNLP 2018) removes the assumption instead of patching around it. SentencePiece treats the raw input as a plain stream of Unicode characters, with no external pre-tokenizer at all — including no assumption that whitespace marks a word boundary. Space itself is replaced by a literal symbol, "▁" (U+2581), placed in front of the word it used to precede, and treated by the vocabulary exactly like any other character. "New Delhi" becomes the stream "▁New▁Delhi" before any merge rule runs, and BPE — or SentencePiece's other mode, described next — is trained and applied directly on that stream, with no separate word-boundary information required. Because ▁ is a normal vocabulary symbol rather than external metadata, detokenization becomes exact string concatenation of the output tokens followed by replacing ▁ with a real space: fully reversible, and identical code regardless of whether the underlying language uses spaces at all. This is what lets one SentencePiece model handle English, Chinese, and Devanagari-script Hindi with the same pipeline and the same code path.

SentencePiece is a framework, not a single algorithm — it implements two interchangeable ways to build the subword vocabulary. One is the BPE merge-loop described above, applied to the raw character stream instead of pre-split words. The other, and the one SentencePiece is best known for, is the Unigram Language Model tokenizer from Kudo's companion paper (Kudo, ACL 2018, Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates). Unigram works in the opposite direction from BPE: instead of starting from single characters and growing the vocabulary by merging, it starts from a large seed vocabulary of candidate substrings — harvested from frequent substrings in the corpus, for instance via a suffix array — and shrinks it. It assumes each vocabulary piece has an independent occurrence probability, estimates those probabilities with the EM algorithm so that the best (Viterbi) segmentation of the training corpus under this simple probability model is as likely as possible, computes how much each piece's removal would hurt that total likelihood, discards the least useful slice of the vocabulary (~25% per iteration — the library's default shrinking_factor is 0.75), and repeats until the vocabulary reaches its target size — always keeping every individual byte or character as a guaranteed fallback, which is what lets SentencePiece, like byte-level BPE, avoid an unknown-token symbol entirely.

The Unigram model's explicit probabilities give it a capability greedy BPE structurally cannot have: because a word can have more than one valid segmentation under the model, SentencePiece can sample among them during training instead of always producing the single canonical split — Kudo calls this subword regularization. Feeding the model different segmentations of the same word across training steps acts as data augmentation directly on the input representation, and Kudo's 2018 experiments showed it measurably improves translation quality and robustness, especially on low-resource and morphologically rich language pairs. Greedy BPE, by contrast, is fully deterministic — a given word always decomposes into exactly one fixed sequence of merges, with no room for that kind of variation.

Active recall

Attempt each question before reading its answer.

1. Why does a byte-level BPE tokenizer (GPT-2/GPT-4 style) never need an unknown-token , while a character-level BPE tokenizer trained only on a Latin-script corpus can?
2. Using the six merge rules learned in the worked example, tokenize "rower" from scratch, showing every rule that fires. Separately, explain why "tired" — which also contains both 'e' and 'r' — produces no merges at all under the same rule set.
3. Suppose "lowest" occurred 8 times in the training corpus instead of 2 (low = 5, newer = 6, wider = 3 unchanged). Recompute the first six merges. Do "newer" and "wider" — whose own frequencies did not change — end up more or less fragmented than in the original run?
4. Why can SentencePiece tokenize Chinese, Japanese, or Thai text correctly with no extra setup, while a classic whitespace-pretokenized BPE pipeline cannot?
5. The chapter measured a 2.85x byte-length ratio between an English sentence and its Hindi translation, from UTF-8 encoding alone. Explain why the real token-count penalty a byte-level BPE tokenizer imposes on Hindi is typically larger than this ratio, not equal to it.
6. What can SentencePiece's Unigram mode do that greedy BPE structurally cannot, and why does it help model training?

Answers

1. Byte-level BPE's base vocabulary is fixed at all 256 possible byte values before any merge rule is even learned, so every UTF-8-encoded input — any script, any emoji, malformed text — decomposes into some sequence of those 256 bytes even in the worst case where no merge rule matches at all; it falls back to one token per byte, never to an unrepresentable symbol. A character-level BPE tokenizer's base vocabulary is instead built only from characters that actually appeared in its training corpus; a character from a script the corpus never contained has no vocabulary entry at all and must be replaced by a designated unknown-token , permanently destroying that information.

2. "rower" starts as r o w e r _. Rule 1 (e+r→er) fires on the fourth and fifth symbols: r o w er _. Rule 2 (er+_→er_) fires next: r o w er_. Rules 3 and 4 need an (l,o) or (lo,w) pair, and rule 5 needs (e,w) — none present — so the word stops here: [r, o, w, er_], four tokens. "tired" is t i r e d _: it does contain both 'e' and 'r', but rule 1 requires 'e' immediately followed by 'r' — the pair (e, r) — while "tired" has 'r' followed by 'e', the pair (r, e), a different, unlearned pair. BPE merge rules are directional and order-sensitive; no rule matches "tired" anywhere, so it stays at six separate symbols. Which merges apply depends on exact adjacency and direction, not on which characters a word happens to contain.

3. Recomputing with lowest = 8 (low = 5, newer = 6, wider = 3 unchanged) gives a completely different first merge: (w, e) now totals 14 — 8 from "lowest" plus 6 from "newer" — beating the old winner (e, r), which drops out of contention entirely. The next five merges are (l, o)→lo [13, from low's 5 plus lowest's 8], (r, _)→r_ [9], then three merges that build "lowest" up piece by piece: (lo, we)→lowe [8], (lowe, s)→lowes [8], (lowes, t)→lowest [8] — so by merge 6, "lowest" itself has collapsed into a single token, and "lowest_" tokenizes as [lowest, _] — two tokens, the same pattern as "low_" → [low, _] earlier in the chapter, rather than the four fragments [e, s, t, _] from the original run. "newer" and "wider" — whose frequencies never changed — end up more fragmented than before: newer_ is now [n, e, we, r_], four tokens versus two in the original run, and wider_ is [w, i, d, e, r_], five tokens versus four. Even "low" — also unchanged in frequency — is not spared: merge 2's (l, o)→lo rule now fires on "low_" as well, before any "lowest"-specific merge can consume it, leaving it as [lo, w, _], three tokens versus the two, [low, _], it had in the original run. Raising one word's frequency didn't just help that word; it changed which pairs won the competition for the shared, fixed merge budget, leaving every other word in the corpus — even "low," which shares no morpheme with "lowest" beyond the letters themselves — worse off purely as a side effect.

4. Classic BPE pipelines require an external, language-specific pre-tokenizer to mark word boundaries with whitespace before BPE ever runs; Chinese, Japanese, and Thai text has no whitespace between words at all, so that pre-tokenization step either fails outright or requires bolting on a separate, per-language segmenter. SentencePiece never assumes word boundaries exist — it treats raw text as one continuous Unicode stream, with space itself demoted to an ordinary symbol (▁) rather than a structural signal — so the identical training and encoding pipeline runs unmodified on any script, spaced or unspaced.

5. The measured 2.85x is a floor set purely by UTF-8's fixed encoding rule (3 bytes per Devanagari code point versus 1 byte per ASCII character) — it says nothing yet about how well either language actually compresses under the tokenizer's learned merge rules. Those merge rules come from corpus frequency: an English-dominated pretraining corpus produces long chains of high-priority merges for common English byte sequences, folding whole common words into a single token, while Devanagari byte trigrams — though present — occur far less often in that same corpus mix and accumulate far fewer merges. English text compresses to well below one token per byte; Hindi text stays close to its raw byte count. The token-count ratio compounds the encoding-length ratio rather than merely inheriting it, making the real gap at least 2.85x, typically by a further, corpus-dependent margin.

6. Unigram keeps an explicit, EM-estimated probability for every vocabulary piece, so a given word can have more than one valid segmentation under the model, each with a computable likelihood — SentencePiece can sample among these instead of always emitting the single canonical split, a technique Kudo (2018) calls subword regularization. Exposing the model to varied segmentations of the same word across training steps acts as data augmentation on the input representation itself, improving robustness to any one segmentation being suboptimal at inference time. Greedy BPE has no probability model at all: a given word decomposes into exactly one fixed merge sequence, deterministically, every time, so it cannot supply this kind of variation.

Think About It

Think about this: How would you explain tokenizer design: bpe and sentencepiece explained to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where tokenizer design: bpe and sentencepiece explained is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting tokenizer design: bpe and sentencepiece explained to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind tokenizer design: bpe and sentencepiece explained, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.