Tokenization: Breaking Text into Pieces

Open a food-delivery app's help chat and type this complaint: "Mera order 20 minute late hai, refund chahiye", a sentence any Indian teenager reads instantly, mixing Hindi and English the way millions of WhatsApp messages do every day. You understand it in a fraction of a second: late order, annoyed customer, wants money back. But the AI model sitting behind that chat window cannot "read" a sentence at all. Strip away the demos and the marketing, and a language model is a stack of matrix multiplications: arithmetic on numbers. It has never seen a letter, a word, or a Hindi verb ending in its life. Before your sentence can go anywhere near that arithmetic, something has to cut it into small pieces and turn each piece into a number. That cutting step is called tokenization, and it is the very first thing that happens to every prompt, every chat message, and every document that any AI text model ever processes. It happens before embeddings, before attention, before anything else you may have read about in other chapters.

It looks like a solved problem — "just split on spaces, right?" — until you try it on real text, especially the messy, code-mixed, multi-script text that Indian applications handle every single day. This chapter builds tokenization from first principles: why splitting on spaces fails, why splitting into single letters also fails, and how almost every modern language model actually finds that middle path, using an algorithm called Byte Pair Encoding. By the end, you will hand-trace the exact algorithm that trains GPT-style tokenizers, on a corpus small enough to compute yourself.

Tokens, Tokenizers, and Vocabularies

A token is the basic unit of text that a language model reads or generates. Depending on the system, a token can be a whole word, a piece of a word, a single character, or a punctuation mark. The program that converts raw text into a sequence of tokens is called a tokenizer. Every tokenizer is built around a fixed list of every token it is allowed to produce, called its vocabulary; each entry in that list has a unique whole-number address called a token ID. Tokenization, then, is a two-step translation: cut the text into pieces, then look up each piece's ID in the vocabulary. Only after that lookup does a sequence of numbers reach the neural network.

Take a plain sentence and a toy word-level vocabulary to see the idea before the complications arrive:

Vocabulary: {"I": 1, "love": 2, "cricket": 3, "watching": 4, "the": 5}

Text:   "I love cricket"
Tokens: ["I", "love", "cricket"]
IDs:    [1, 2, 3]

Three words became three tokens became three numbers. The neural network only ever sees [1, 2, 3], a list of integers it will convert into vectors on the very next step. Everything a model appears to "understand" about language starts from this list of numbers, which makes the quality of tokenization a real ceiling on what a model can do, not a minor implementation detail.

Why Not Just Split on Spaces?

Splitting on whitespace is the obvious first idea, and it fails within one sentence of real text. Try it in Python on our opening complaint:

text = "Mera order 20 minute late hai, refund chahiye!"
tokens = text.split()
print(tokens)
# ['Mera', 'order', '20', 'minute', 'late', 'hai,', 'refund', 'chahiye!']

Look closely at the last two tokens: "hai," and "chahiye!" still carry their punctuation glued on. A word-level vocabulary would now need separate, unrelated entries for "hai" and "hai,", and for "chahiye" and "chahiye!", with no way to know they share almost all their meaning. Push a little further and whitespace splitting only gets worse:

• Explosion of word forms. A word-level tokenizer treats "run", "runs", "running", and "ran" as four completely unrelated vocabulary entries, with no shared structure between them. Languages with rich suffix systems (Hindi, Marathi, Tamil, Telugu, and most other Indian languages inflect nouns and verbs far more than English does) make this worse, since a single verb root can legitimately appear in dozens of surface forms.
• The out-of-vocabulary problem. No matter how large a fixed word list you build, someone will eventually type a brand-new word, a misspelling, a username, or a technical term that is not in it. A pure word-level tokenizer has only one option for anything missing: a generic <UNK> ("unknown") token, which throws away all information about what that word actually was.
• Ambiguous boundaries. Should "₹499.99" be one token, or split into a currency symbol, digits, and a decimal point? Should "don't" split into "do" and "n't"? There is no single universally correct answer, and code-mixed text like our Hinglish example makes it worse: a whitespace-based tokenizer cannot tell it is looking at two languages stitched into one sentence, because no space marks where "English" ends and "Hindi" begins.
• Some languages don't use spaces at all. Chinese and Japanese text is not space-separated, and traditional Sanskrit text can fuse words together through a process called sandhi. Any tokenizer that assumes "a word is whatever sits between two spaces" simply breaks on this kind of text.

A fixed word-level vocabulary large enough to cover every inflected form of every word in even one language would run into hundreds of thousands, if not millions, of entries, and it would still fail the moment it met a word it had never seen before.

The Other Extreme: One Token per Character

If splitting into whole words creates too many vocabulary entries, the opposite idea is to split into individual characters. A character-level tokenizer needs only a tiny vocabulary: just the alphabet, digits, and punctuation, perhaps a few hundred symbols in total. It can represent absolutely any string, including words it has never encountered, because every word is simply a sequence of already-known characters. The out-of-vocabulary problem disappears completely.

The cost is sequence length. The word "cricket" becomes seven separate tokens: c, r, i, c, k, e, t. A short WhatsApp message can turn into a sequence of several hundred tokens instead of a few dozen. This matters because of how the Transformer architecture that powers modern language models actually computes: its self-attention mechanism compares every token in a sequence against every other token, so both the computation and the memory it needs grow roughly with the square of the sequence length. Quadrupling the token count does not just quadruple the cost: it multiplies it by roughly sixteen. Character-level tokenization also gives the model more work to recover meaning: instead of receiving "cricket" as one ready-made unit, it has to learn, purely from data, that the pattern c-r-i-c-k-e-t consistently points to one concept, every single time that word appears anywhere in training.

The Middle Path: Subword Tokenization

Subword tokenization is the compromise that essentially every production language model now uses: common whole words get their own single token, while rare or unfamiliar words get broken down into smaller, still-meaningful pieces (prefixes, suffixes, word roots) and, in the worst case, all the way down to individual characters or bytes. That fallback guarantees any input string can always be tokenized, with no <UNK> token ever required. "running" might become "run" plus "ning"; a rare technical word might become four or five small pieces; a common word like "the" stays whole. The vocabulary stays a manageable size (tens of thousands of entries, not millions), sequences stay reasonably short, and nothing is ever truly unrepresentable.

The algorithm that made this practical is called Byte Pair Encoding (BPE). It was originally invented in 1994 by Philip Gage as a data-compression technique with nothing to do with language at all. It simply found repeated byte patterns in a file and replaced them with shorter codes. In 2016, researchers Rico Sennrich, Barry Haddow, and Alexandra Birch repurposed the same idea to build subword vocabularies for machine translation, and it has powered the tokenizers behind GPT-2, GPT-3, and many of their successors ever since.

Byte Pair Encoding, Traced by Hand

BPE learns its vocabulary from a training corpus by repeating one simple move: find the pair of adjacent symbols that occurs most often, and merge that pair into a single new symbol. Repeat this thousands of times, and common patterns (first letters, then syllables, then whole common words) gradually fuse into single tokens, while rare patterns stay broken apart.

Let's run it by hand on a miniature corpus: four related words, each split into individual characters, with a special end-of-word marker _ attached so the algorithm can tell where one word stops and the next begins.

Word       Count   Starting symbols
"play"       5     p l a y _
"plays"      2     p l a y s _
"player"     4     p l a y e r _
"playing"    3     p l a y i n g _

Step 1: count every adjacent pair. A pair's total count is the sum of the counts of every word it appears in. The pair (p, l) appears once in every one of the four words, so its count is 5 + 2 + 4 + 3 = 14:

(p, l): 14      (y, e): 4
(l, a): 14      (e, r): 4
(a, y): 14      (r, _): 4
(y, _): 5       (y, i): 3
(y, s): 2       (i, n): 3
(s, _): 2       (n, g): 3
                (g, _): 3

Three pairs are tied for the top count: (p, l), (l, a), and (a, y), all at 14. This is expected, since every one of our four words starts with the same four letters. Real BPE implementations break ties with a fixed, consistent rule, commonly whichever pair was encountered first while scanning left to right; we'll do the same.

Merges 1 through 4. Merging the top pair each round, and recomputing every count after each merge, plays out like this:

Merge 1: (p, l)    -> "pl"     count 14
Merge 2: (pl, a)   -> "pla"    count 14
Merge 3: (pla, y)  -> "play"   count 14
Merge 4: (play, _) -> "play_"  count 5

After four merges, our four words look like this:

"play"     -> [play_]              (fully merged: 1 token)
"plays"    -> [play, s, _]         (3 tokens)
"player"   -> [play, e, r, _]      (4 tokens)
"playing"  -> [play, i, n, g, _]   (5 tokens)

This is the payoff: after only four merges, all four words already share the token play, even though the algorithm was never told anything about English morphology. It discovered the shared root purely by counting. The suffixes have not merged yet because, on their own, pairs like e+r or i+n occur less often (count 4 and 3) than p+l did (count 14). Run the process further and pairs like (e, r) and (i, n) would start merging next, gradually building er and ing into tokens of their own too.

The Same Algorithm as Code

The hand trace above is exactly what the following Python reproduces. get_pair_counts tallies every adjacent pair weighted by word frequency; merge_pair replaces every occurrence of the winning pair with its fused form:

from collections import defaultdict

corpus = {
    ("p", "l", "a", "y", "_"): 5,
    ("p", "l", "a", "y", "s", "_"): 2,
    ("p", "l", "a", "y", "e", "r", "_"): 4,
    ("p", "l", "a", "y", "i", "n", "g", "_"): 3,
}

def get_pair_counts(corpus):
    pairs = defaultdict(int)
    for word, freq in corpus.items():
        for i in range(len(word) - 1):
            pairs[(word[i], word[i + 1])] += freq
    return pairs

def merge_pair(pair, corpus):
    a, b = pair
    merged = a + b
    new_corpus = {}
    for word, freq in corpus.items():
        new_word, i = [], 0
        while i < len(word):
            if i < len(word) - 1 and word[i] == a and word[i + 1] == b:
                new_word.append(merged)
                i += 2
            else:
                new_word.append(word[i])
                i += 1
        new_corpus[tuple(new_word)] = freq
    return new_corpus

for step in range(4):
    pairs = get_pair_counts(corpus)
    best_pair = max(pairs, key=lambda p: pairs[p])
    print(f"Merge {step + 1}: {best_pair} -> count {pairs[best_pair]}")
    corpus = merge_pair(best_pair, corpus)

print(corpus)

Running this prints exactly the four merges traced above, because Python's max keeps the first maximal item it finds, and pairs are counted in the same left-to-right order used by hand:

Merge 1: ('p', 'l') -> count 14
Merge 2: ('pl', 'a') -> count 14
Merge 3: ('pla', 'y') -> count 14
Merge 4: ('play', '_') -> count 5
{('play_',): 5, ('play', 's', '_'): 2, ('play', 'e', 'r', '_'): 4, ('play', 'i', 'n', 'g', '_'): 3}

Two details matter beyond this toy example. First, training and using a tokenizer are different steps: training happens once, offline, on a huge corpus, and produces an ordered list of merge rules like the one above. Using the tokenizer afterward on new text (including a word the model has never seen) means applying that same ordered list of merges, in the order they were learned, to the new word's starting characters. Second, real training does not stop after four merges; it keeps going until the vocabulary reaches a target size chosen in advance.

How Big Are Real Vocabularies?

GPT-2's tokenizer runs BPE at the level of raw bytes rather than characters, and stops after 50,000 merges; together with 256 single-byte tokens and one special end-of-text marker, that gives a vocabulary of exactly 50,257 tokens. BERT uses a close relative of BPE called WordPiece, with a vocabulary of 30,522 tokens in its base model. WordPiece picks its merges slightly differently: instead of choosing whichever pair is simply the most frequent, it scores each candidate pair by dividing its frequency by the product of its two parts' individual frequencies, favoring pairs that show up together far more often than their individual popularity alone would predict, over pairs that are merely common on their own.

A third widely used approach, implemented in Google's SentencePiece toolkit, can run in the opposite direction from BPE: its Unigram Language Model algorithm starts from a large candidate vocabulary and repeatedly removes whichever token hurts the corpus's overall likelihood the least, shrinking down to the target size. SentencePiece also treats whitespace itself as an ordinary character to be tokenized (traditionally marking it with the symbol ▁) rather than assuming text arrives already split into words, which is exactly what makes it useful for languages that do not reliably use spaces to separate words.

Tokenization and Indian Languages

Return to the byte-level detail behind GPT-2's tokenizer, and something with real consequences for Indian users comes into focus. In the UTF-8 encoding that almost the entire internet uses, an ordinary English letter like a takes exactly one byte. Devanagari characters (the script used for Hindi, Marathi, and Sanskrit) sit in the Unicode range U+0900 to U+097F, which UTF-8 always encodes using three bytes per character. A single Hindi letter costs three times as many raw bytes as a single English letter before a tokenizer has made a single decision about merging anything. Tamil, Telugu, Kannada, Bengali, and every other major Indian script carry the same three-byte cost, since none of them fall in the one-byte ASCII range.

Byte count is only half the story. A BPE or WordPiece vocabulary only learns to merge patterns it saw often during training, and most large general-purpose tokenizers are trained on text collected predominantly from the English-dominated web. The practical result is exactly what the play example above predicts: frequent English words such as "the", "order", or "refund" collapse into single tokens, while less-frequent Hindi or Tamil words (even ordinary, everyday ones) end up chopped into many more, smaller pieces, because the tokenizer's training data never gave those patterns enough opportunities to merge. Code-mixed text of exactly the kind that opened this chapter is the hardest case of all: a single sentence forces the tokenizer to switch between two very different sets of learned patterns, sentence by sentence and sometimes word by word.

This is not a purely academic concern. Most commercial language-model APIs measure both how much you can send in one request and what you pay for it in tokens, not in words or characters. So a Hindi or Tamil sentence that needs noticeably more tokens than its English translation for the same meaning also consumes more of that budget and more of the model's limited context window for the same amount of information. This is exactly why Indian research groups build tokenizers deliberately trained on Indian-language text. AI4Bharat, a research initiative based at IIT Madras, has released open tokenizers and models such as IndicBERT, trained specifically on a broad mix of Indian languages using SentencePiece rather than assuming English-style, space-separated words, precisely so that Hindi, Tamil, Telugu, Bengali, Marathi, and other Indian languages get the same short, meaningful tokens for their common words that English speakers take for granted.

Back to the Help-Chat

Return one last time to "Mera order 20 minute late hai, refund chahiye". A subword tokenizer trained mostly on English text is likely to keep order, late, and refund as single, efficient tokens (common English words it has seen millions of times) while Mera, hai, and chahiye get broken into smaller, less efficient fragments, simply because the tokenizer's training data contained far fewer examples of them. Nothing about this reflects the importance or complexity of the words themselves; it is a direct, mechanical consequence of frequency counts during BPE training, running exactly like the merge-by-merge process you just traced by hand on "play", "plays", "player", and "playing".

Every AI text system you will ever build, prompt, or evaluate starts here. Before a single embedding is looked up, before any attention is computed, before any answer is generated, the raw text has already been cut into tokens, and those tokens have already decided how much of your sentence the model gets to work with per unit of computation. Get comfortable counting pairs and merging them by hand, because that same small idea — find what occurs together most often, fuse it into one unit, repeat — is the first working piece of nearly every language model in production today.

Think About It

Think about this: How would you explain tokenization: breaking text into pieces to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where tokenization: breaking text into pieces is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting tokenization: breaking text into pieces to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind tokenization: breaking text into pieces, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.