A team building a Hindi-language customer-support layer on top of a commercial LLM API notices something odd in its billing dashboard. The English version of the product answers the same class of query for a fraction of the token cost of the Hindi version, and the Hindi conversation "forgets" earlier turns far sooner than the English one, even though both are capped at the same context-window limit advertised by the provider. Nobody changed the prompt logic. Nobody changed the model. The only thing that differs is the script the customer is typing in, and that alone is enough to make the Hindi deployment several times more expensive and several times more context-starved than the English one, before the model has generated a single token of output. This is not a bug in the application. It is a structural property of how the tokenizer's base alphabet interacts with UTF-8, and it is worth understanding precisely, because it is the single most consequential design decision in a tokenizer that a curriculum built around the byte-pair-merge algorithm and SentencePiece's interface rarely gets to.
You already know, from the companion chapter on tokenizer design, how the BPE merge loop works: start from an alphabet, count adjacent pair frequencies across a training corpus, merge the most frequent pair into a new symbol, repeat until you hit a target vocabulary size. You also know that SentencePiece treats text as a raw stream of Unicode, marks whitespace as an explicit symbol (▁) so that detokenization is lossless, and can run either the BPE merge loop or a probabilistic unigram language-model algorithm (Kudo, 2018) to build its vocabulary. What that leaves open is the question this chapter answers: what is "an alphabet," concretely, for a tokenizer that has to handle every human language, and why does the answer to that question decide, before a single merge rule is even learned, how many tokens a given script costs.
Two families, one starting alphabet
Production tokenizers split into two families based on what symbols they start merging from. The first, used by OpenAI's tokenizer lineage from GPT-2 onward (Radford et al., 2019) and shipped today as the open-source tiktoken library, starts from the 256 possible raw byte values of UTF-8-encoded text. Every input string, in any language, is first encoded to bytes, and the initial "alphabet" is simply the integers 0 through 255. Merges are then learned over sequences of these bytes. Because every one of the 256 byte values is already a vocabulary entry, byte-level BPE has a guarantee that plain character-level or word-level vocabularies cannot make cheaply: it never needs an <unk> token. Feed it an emoji, a Klingon transliteration, or a Devanagari word it has never merged before, and worst case it falls back to emitting several single-byte tokens: ugly, but never . This was explicitly GPT-2's motivation for choosing bytes over Unicode codepoints as the base alphabet (Radford et al., 2019): a codepoint-level vocabulary still needs an escape hatch for the roughly 150,000 assigned Unicode codepoints it did not see enough of during training, while a byte-level one needs none, since 256 codepoints is a small enough space to cover exhaustively.
The second family, SentencePiece (Kudo & Richardson, 2018), by default starts from raw Unicode codepoints, not bytes, and only drops to byte-level fallback for a codepoint its trained vocabulary genuinely never saw (a training-time option called byte_fallback). LLaMA's tokenizer is a concrete example: a 32,000-token SentencePiece BPE vocabulary trained on a corpus that was overwhelmingly English and code, with byte fallback enabled so that any script outside that training distribution still encodes without crashing (Touvron et al., 2023). The two families solve the "never emit unknown" problem differently, one by starting at the byte level everywhere, the other by falling back to bytes only when necessary, but neither family's design, by itself, prevents the cost blowup that opened this chapter. That blowup comes from a fact about UTF-8 encoding, and it applies identically whether you got to the byte level by starting there or by falling back into it.
The structural cost: UTF-8 is not script-neutral
UTF-8 encodes Unicode codepoints U+0000 to U+007F (essentially the ASCII range: the Latin alphabet, digits, common punctuation) in a single byte. Codepoints from U+0080 up to U+07FF take two bytes. Codepoints from U+0800 up to U+FFFF, which is where every Brahmic script used in India lives (Devanagari, Bengali, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada, Malayalam all sit in the U+0900 to U+0DFF band), take three bytes per codepoint. So before any tokenizer, any merge rule, or any training corpus enters the picture, a word written in Devanagari already needs roughly three times as many raw bytes as the same number of Latin characters. A byte-level BPE tokenizer's very first alphabet is already tilted against Indic scripts. What the learned merges do afterward either compounds that tilt or partially corrects it, and which of those happens depends entirely on how much Devanagari byte-sequence traffic the tokenizer's training corpus contained.
Fertility and compression ratio: the two numbers that matter
Two metrics quantify this. Fertility is the number of tokens a tokenizer needs to represent a fixed unit of content, typically measured per word, or per a fixed reference sentence translated across languages. Fertility is what your API bill, your decoding latency, and your usable context window are all directly proportional to: every extra token is an extra step of autoregressive decoding and an extra line item on the invoice. Compression ratio is UTF-8 byte length divided by token count: how many raw bytes each token, on average, is standing in for. It answers a different question: how much redundancy the learned merges managed to remove from this specific span of bytes. The two numbers are easy to conflate, and conflating them is the misconception this chapter corrects below, because they can point in opposite directions for the same piece of text.
Worked example: measuring the gap directly
Rather than estimate, measure. The following was run against the real, public tokenizers OpenAI ships in tiktoken: cl100k_base (GPT-3.5-turbo and GPT-4, 100,277-token vocabulary) and o200k_base (GPT-4o, 200,019-token vocabulary). The word compared is "water" in English against its Hindi equivalent "पानी" (pānī).
import tiktoken
cl100k = tiktoken.get_encoding("cl100k_base") # GPT-3.5-turbo, GPT-4
o200k = tiktoken.get_encoding("o200k_base") # GPT-4o
for word in ["water", "पानी"]:
ids_cl = cl100k.encode(word)
ids_o2 = o200k.encode(word)
print(word, "| cl100k:", len(ids_cl), "tok", ids_cl)
print(word, "| o200k :", len(ids_o2), "tok", ids_o2)
water | cl100k: 1 tok [13284]
water | o200k : 1 tok [15373]
पानी | cl100k: 4 tok [87262, 32511, 101, 44747]
पानी | o200k : 2 tok [2033, 16783]
"water" is UTF-8-encoded in 5 bytes (one byte per ASCII character) and is common enough in both tokenizers' training data to have earned its own single learned merge: 1 token either way. "पानी" is 4 Unicode codepoints, each 3 bytes under UTF-8, for 12 bytes total. Under cl100k_base it costs 4 tokens; under o200k_base, 2. That is already a fertility ratio of 4.0 and 2.0 respectively for a single word, against English's 1.0. Pulling the exact byte spans each token covers, using decode_single_token_bytes, shows why the two tokenizers differ so sharply:
cl100k_base tokens for "पानी" (12 bytes total):
token 87262 -> b'\xe0\xa4\xaa' 3 bytes = प (clean codepoint)
token 32511 -> b'\xe0\xa4\xbe\xe0\xa4' 5 bytes = ा + first 2 bytes of न (crosses a codepoint boundary!)
token 101 -> b'\xa8' 1 byte = last byte of न (a lone continuation byte)
token 44747 -> b'\xe0\xa5\x80' 3 bytes = ी (clean codepoint)
o200k_base tokens for "पानी" (12 bytes total):
token 2033 -> b'\xe0\xa4\xaa' 3 bytes = प (clean codepoint)
token 16783 -> b'\xe0\xa4\xbe\xe0\xa4\xa8\xe0\xa5\x80' 9 bytes = ानी (three clean codepoints merged as one unit)
Both tokenizers agree on splitting off प as its own token. Where they diverge is instructive: cl100k_base's second token, id 32511, is five raw bytes that begin partway through one Devanagari character and end partway through the next. It is not a valid standalone UTF-8 sequence, and decoding it in isolation produces the Unicode replacement character in place of the missing bytes. o200k_base, by contrast, learned a single 9-byte merge that exactly spans three whole codepoints (ा, न, ी together, "ānī"), so every token it emits still lines up with real character boundaries. o200k_base needed a bigger vocabulary and more Devanagari byte-sequence exposure during training to learn that merge; cl100k_base never had the frequency signal to learn it and fell back to whatever partial byte n-grams happened to be frequent enough to merge.
The same pattern holds at sentence scale. "The bank approved the loan." (27 bytes, 5 words) tokenizes to 6 tokens under both tokenizers. Its Hindi equivalent, "बैंक ने ऋण को मंज़ूरी दे दी।", literally "bank [agent] loan [dative] approval gave", i.e. "the bank approved the loan", is 72 UTF-8 bytes across 7 words, and tokenizes to 33 tokens under cl100k_base and 12 under o200k_base. That is a fertility ratio, Hindi tokens divided by English tokens for the same content, of 33/6 = 5.5 under cl100k_base and 12/6 = 2.0 under o200k_base, matching the word-level result closely and confirming it is not a one-word fluke. This exact pattern, a 3-to-15-fold token-count gap between languages on commercial tokenizers, is what Petrov et al. (2023) measured systematically across the FLORES-200 benchmark's 200 languages, and Ahia et al. (2023) traced through to real dollar cost and latency differences on commercial LLM APIs across 22 languages.
Compression ratio tells a genuinely different, almost opposite-sounding story. For "water": 5 bytes / 1 token = 5.0 bytes/token. For "पानी" under cl100k_base: 12 / 4 = 3.0 bytes/token, worse than English, consistent with the visible fragmentation above. But under o200k_base: 12 / 2 = 6.0 bytes/token, better compression than English's 5.0, because Devanagari packs more raw bytes into fewer, larger tokens once the tokenizer has learned to merge whole characters together. Sentence-level compression matches: English 27/6 = 4.5 bytes/token; Hindi o200k_base 72/12 = 6.0 bytes/token. Hindi is the more byte-efficient encoding under o200k_base and still costs the user twice as many tokens, twice the API bill, and half the effective context window for the same content. Compression ratio and fertility are not the same axis, and only one of them determines what you pay and how much conversation history fits in the window.
Correcting a common misconception
The natural conclusion to draw from "vocabulary doubled and the fertility gap dropped from 5.5x to 2.0x" is that a bigger vocabulary automatically buys fairness across languages, so the fix is simply to keep growing the vocabulary until the gap closes. That conclusion is wrong, and the byte-span evidence above shows why. Vocabulary size only sets how many merge slots exist; which byte sequences actually win those slots is decided entirely by their frequency in the training corpus. Doubling the vocabulary from 100,277 to 200,019 entries helped Hindi here specifically because o200k_base was also trained with substantially more non-English text feeding the frequency counts that the merge algorithm optimizes against: the extra slots had Devanagari byte n-grams to compete for them. If you instead doubled the vocabulary again while holding the training corpus's language mixture fixed, the additional merge slots would predominantly still be won by increasingly rare English and code sequences, since those remain the majority class in an unchanged corpus. A larger vocabulary is necessary but nowhere near sufficient for closing a fertility gap; the corpus mixture the tokenizer is trained on is the variable that actually does the work, and it is easy to forget that when a headline vocabulary number is the only thing being compared.
The efficiency cost of vocabulary growth is not free either
Growing the vocabulary to fix fertility has its own price, and it is paid in parameters and compute regardless of language. A transformer's token-embedding table and (if untied) its output projection are both matrices of shape vocabulary_size × d_model. Going from a 200,019-token vocabulary to a hypothetical 400,000-token one, holding the hidden size at a representative d_model = 4096, adds 200,000 × 4,096 ≈ 819.2 million parameters to the embedding table alone, and roughly 1.64 billion if the output projection is untied from the embedding and grows too. The final projection-plus-softmax step of every single decoding pass is a matrix multiply against that same vocabulary_size × d_model matrix, so its FLOP cost per generated token scales linearly with vocabulary size as well: doubling the vocabulary roughly doubles the cost of that one layer, on every token, for the lifetime of the model. And a larger vocabulary spreads the same training corpus across more distinct token IDs, so the long tail of newly-added, rarely-merged tokens gets a thinner training signal and a correspondingly weaker embedding, a second, independent cost of scaling the vocabulary without also scaling the amount of training data behind each language. Vocabulary growth is a real engineering lever, not a free fix, which is why production tokenizer redesigns (like the cl100k to o200k transition) pair a bigger vocabulary with deliberately rebalanced training data rather than vocabulary growth alone, and why organizations building Indian-language models from scratch, such as AI4Bharat, train dedicated SentencePiece vocabularies on Indic-heavy corpora rather than simply asking an English-centric tokenizer for a bigger vocabulary.
What this costs in a running system
Three consequences follow directly from a fertility ratio greater than 1, and they compound rather than being independent. First, API billing: nearly every commercial LLM API charges per token for both input and output, so a 2x to 5.5x fertility ratio is a 2x to 5.5x price increase for delivering the same information, precisely the "tokenization tax" Ahia et al. (2023) quantified across commercial APIs and 22 languages. Second, effective context window: a provider's advertised context limit is a token count, not a word count, so it has to be divided by the language's actual tokens-per-word to get a usable capacity. Taking the measured sentence-level fertility here (English 1.2 tokens/word; Hindi 4.71 tokens/word under cl100k_base, 1.71 under o200k_base) and extrapolating linearly across a 128,000-token context budget: roughly 106,667 words of English fit, against roughly 27,152 words of Hindi under cl100k_base and roughly 74,667 words under o200k_base, the same advertised window holding under a third, or up to about two-thirds, as much actual conversation, depending purely on which tokenizer generation is behind the API. Third, decoding latency: autoregressive generation produces one token per forward pass, so at a fixed tokens-per-second serving rate, saying the same thing in the higher-fertility language takes proportionally longer wall-clock time and proportionally more GPU-seconds of serving capacity per response. None of these three costs are visible in the model's parameter count or its benchmark accuracy; all three live entirely inside the tokenizer, and all three are measurable before a single forward pass runs, exactly as demonstrated in the worked example above.
Active recall
Attempt each question before reading the answer beneath it.
Q1. Define fertility and compression ratio precisely, in terms of tokens and bytes. Can a text span have a compression ratio higher than another language's while still having a worse (higher) fertility for the same content? Use the numbers from this chapter to support your answer.
Q2. Why does byte-level BPE (the family used by tiktoken) never need an <unk> token, regardless of input? Does SentencePiece's default codepoint-level mode share that guarantee, and if not, what feature does SentencePiece add to close the gap?
Q3. A teammate says: "o200k_base is strictly better for Hindi than cl100k_base, it just uses fewer tokens for the same text." Using only the byte-span data in this chapter (not the token counts), give the more precise, mechanism-level reason o200k_base is better, beyond "fewer tokens."
Q4 (ripple effect). Suppose OpenAI trained a hypothetical o400k_base: vocabulary doubled again, from 200,019 to 400,000 tokens, but the training corpus's language mixture is held exactly the same as the one used for o200k_base, no additional Hindi (or other non-English) data is added. Trace the ripple effects on: (a) the Hindi/English fertility ratio measured in this chapter; (b) the size of the embedding table, given a hidden size of 4,096; (c) the compute cost of the final output-projection-plus-softmax step per decoded token; (d) how the extra vocabulary slots get allocated across languages; (e) the effective Hindi context window at a fixed 128,000-token budget.
Q5. Using the sentence-level numbers (English: 6 tokens for a 5-word, 27-byte sentence; Hindi: 33 tokens under cl100k_base and 12 under o200k_base for a 7-word, 72-byte equivalent sentence), compute the compression ratio (bytes/token) for all three cases. Which has the best compression ratio, and does that tokenizer also have the best fertility? Explain the apparent contradiction.
Q6. A byte-level BPE tokenizer emits a token whose underlying bytes, decoded alone, produce the Unicode replacement character (as token 32511 does in the cl100k_base example). Does this mean the tokenizer has made an error, or lost information? What actually happens to those bytes downstream, and why does the model not need the token to be valid UTF-8 on its own to use it correctly?
Worked answers
A1. Fertility = tokens divided by content-unit (words, or a fixed reference sentence): it measures how many tokens a fixed amount of meaning costs. Compression ratio = UTF-8 bytes divided by tokens: it measures how much raw byte-length each token, on average, absorbs. Yes: the chapter's own sentence-level numbers show it directly. Under o200k_base, Hindi's compression ratio is 72/12 = 6.0 bytes/token, better (higher) than English's 27/6 = 4.5 bytes/token, yet Hindi still needs 12 tokens against English's 6 for the same content, i.e., strictly worse (higher) fertility. The reason both are true at once is that Hindi's raw byte footprint (72 bytes) is already 2.67x English's (27 bytes) purely from UTF-8 structure, before either tokenizer runs; excellent compression on an inflated starting point can still leave you with more tokens than a mediocre compression ratio on a compact starting point.
A2. Byte-level BPE's base alphabet is the 256 possible byte values, so every conceivable byte is already a valid single-token fallback; there is no byte value the vocabulary can fail to represent, hence no need for an escape symbol. SentencePiece's default mode starts from Unicode codepoints, not bytes, and a training corpus cannot possibly contain every one of the roughly 150,000 assigned codepoints with useful frequency, so without help it would need <unk> for anything unseen. SentencePiece closes the gap with an explicit byte_fallback option: any codepoint missing from the trained vocabulary is decomposed into its raw UTF-8 bytes and encoded using byte-level tokens instead, exactly as LLaMA's tokenizer does (Touvron et al., 2023).
A3. The mechanism-level reason: o200k_base's tokens for "पानी" respect character boundaries, its two tokens are प (a clean 3-byte codepoint) and ानी (a clean 9-byte, three-codepoint merge), while cl100k_base's second and third tokens are raw byte fragments that straddle the boundary between ा and न and are not valid standalone UTF-8. Fewer tokens is the visible symptom; the underlying cause is that o200k_base's training data gave it enough frequency signal on Devanagari byte sequences to learn character-respecting merges, where cl100k_base's did not and fell back to whatever partial byte n-grams were locally frequent.
A4. (a) Fertility ratio: with the language mixture unchanged, the new merge slots are won predominantly by the majority class in that mixture, still overwhelmingly English/code sequences, so the Hindi/English fertility ratio would improve only marginally from 2.0x, nowhere near proportionally to the vocabulary doubling; this is the direct consequence of the misconception correction above. (b) Embedding table: extra parameters = delta_vocab × d_model = 200,000 × 4,096 = 819,200,000, approximately 820 million additional parameters in the embedding matrix alone (roughly 1.64 billion if the output projection is untied and also grows). (c) The output-projection-plus-softmax matmul is [batch times seq, d_model] times [d_model, vocab_size]; its FLOP cost per decoded token scales linearly with vocabulary size, so this step's cost roughly doubles for every token generated, for the lifetime of the model. (d) Allocation: with the corpus mixture unchanged, most new slots go to increasingly rare English/code n-grams (pushing English fertility even closer to its floor of about 1 token/word), and only a small residual fraction go to non-English scripts. (e) Effective Hindi context window: since the fertility ratio barely moves from (a), the 128,000-token budget still yields roughly the same approximately 74,000-word Hindi capacity computed earlier (128,000 divided by 1.71 tokens/word) rather than climbing toward English's approximately 106,700-word capacity; the vocabulary doubling was expensive in parameters and compute but bought almost none of the fairness gain that a real corpus rebalancing (as evidently happened between cl100k_base and the actual o200k_base) delivered.
A5. English: 27/6 = 4.5 bytes/token. Hindi, cl100k_base: 72/33 ≈ 2.18 bytes/token. Hindi, o200k_base: 72/12 = 6.0 bytes/token. Best compression ratio: Hindi under o200k_base (6.0). Best fertility: English, at 6 tokens for its content versus Hindi's 12 (o200k_base) or 33 (cl100k_base) for equivalent content; English wins fertility even though it loses compression ratio to Hindi/o200k_base. The apparent contradiction resolves once you separate the two questions compression ratio and fertility actually answer: compression ratio only tells you how efficiently a tokenizer packed the bytes it was given, and Hindi's UTF-8 encoding hands it nearly three times as many raw bytes to work with for the same sentence, so even a superior packing job on that inflated byte count still lands on more tokens than English's leaner starting point got with mediocre packing.
A6. No error and no lost information. Byte-level BPE tokens are opaque integer IDs to the model; nothing requires that decoding a single token in isolation produce valid, displayable UTF-8, validity only has to hold once the full sequence of tokens for that span is decoded back to bytes and concatenated, at which point every byte is present and in order, and the original text reconstructs exactly. The replacement character only appears when you try to interpret one token's bytes on their own, which is a debugging/visualization artifact, not something the model or the detokenizer ever does in normal operation; the model's embedding table simply gives that token ID a learned vector like any other, and the detokenizer always operates on the whole token sequence together.
Think About It
Think about this: How would you explain tokenization deep dive: bpe, sentencepiece, multilingual tokenizers, and efficiency to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.
Practice Exercises
Now it is time to practice! Complete these challenges to solidify your understanding:
- Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
- Exercise 2: Find a real-world example where tokenization deep dive: bpe, sentencepiece, multilingual tokenizers, and efficiency is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
- Exercise 3: Create a mind-map connecting tokenization deep dive: bpe, sentencepiece, multilingual tokenizers, and efficiency to at least 3 other topics you have studied.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind tokenization deep dive: bpe, sentencepiece, multilingual tokenizers, and efficiency, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.