Text-to-Speech: Generating Natural Audio

Call the IRCTC enquiry line and you hear a machine say: "Train number one two nine five one, Mumbai Rajdhani Express, is running late by forty-five minutes. Expected arrival, platform number three, at eighteen hours four minutes." Nothing about that sentence sounds hard to produce — it is just a string of English words. But look at what the system actually received as input: the text "12951 Rajdhani Exp is running 45 min late, arr. PF 3 at 18:04 hrs". Every one of the tricky decisions already happened before a single sound was generated. The system had to know that 12951, a train number, is read digit-by-digit ("one two nine five one") while 45, a duration, is read as a whole number ("forty-five"). It had to expand PF to "platform" and 18:04 to "eighteen hours four minutes" rather than "eighteen colon oh four" or "six oh four PM." None of this is about pronunciation — it is about deciding, from bare text and no other context, what should be pronounced at all. This is the first and least appreciated lesson of text-to-speech (TTS): synthesizing a waveform that sounds like a human voice is the easy part of a mature pipeline. Deciding exactly what sequence of words the system is trying to say is where most real-world TTS systems still fail. This chapter builds the full pipeline — text normalization, phoneme conversion, acoustic modelling, and vocoding — from first principles, and works through the actual arithmetic that governs each stage.

The pipeline: four transformations, each closer to sound

A modern neural TTS system is not one model. It is a chain of four distinct transformations, each converting the input into a representation that carries more acoustic detail than the last. Understanding TTS means understanding what each stage receives, what it outputs, and why that particular representation was chosen as the hand-off point.

1. Text normalization converts arbitrary written text into the exact sequence of words that should be spoken — expanding 18:04, ₹250cr, Dr., and 23/08/2023 into their spoken forms.
2. Grapheme-to-phoneme (G2P) conversion converts that word sequence into a sequence of phonemes — the discrete units of sound a language uses (roughly 40–44 for English, depending on the accent standard) — resolving cases where the same spelling maps to different sounds depending on meaning.
3. The acoustic model is a neural network that consumes the phoneme sequence and predicts a mel-spectrogram: a time-frequency representation of what the audio should look like, frame by frame.
4. The vocoder is a second neural network that consumes the mel-spectrogram and generates the actual raw audio waveform — tens of thousands of amplitude samples per second — that get pushed to a speaker.

The diagram below lays out this chain along with the actual shape of the representation each stage hands to the next, using a running example (the word "namaste" and a 3-second utterance) that this chapter will trace numerically.

Stage 1: text normalization — the hardest problem nobody names

Text normalization takes arbitrary written text and produces the exact sequence of spoken words. This sounds like a lookup-table problem and is, in fact, a genuine natural-language-understanding problem, because the correct expansion of a token depends on its grammatical role and on neighbouring tokens, not on the characters alone. Consider the digit string "1947". Read as a year, it becomes "nineteen forty-seven." Read as a quantity — "1947 students registered" — it becomes "one thousand nine hundred forty-seven." A normalizer that always applies the year rule will make "1947 students registered" sound like a school founded a very specific year, not a headcount. The same ambiguity recurs everywhere in Indian text: "₹2.5 Cr" must become "two point five crore rupees," a phrase whose grammar (the Indian numbering system's crore/lakh units) has no direct expansion rule in TTS toolkits built for American English; "12/08/2023" is genuinely ambiguous between 12 August and December 8th depending on the source locale; and "Dr." expands to "doctor" before a name but to "drive" as a street suffix ("12 MG Dr.").

Below is a deliberately small, illustrative normalizer that handles exactly one such rule — a DD/MM/YYYY date — so the mechanism is fully traceable rather than hidden behind a library call.

def normalize_date(token):
    """Expand a DD/MM/YYYY token into spoken English. Handles only the
    ordinals this example needs; a production system would cover 1-31."""
    day_str, month_str, year_str = token.split("/")
    months = ["january", "february", "march", "april", "may", "june",
              "july", "august", "september", "october", "november", "december"]
    day = int(day_str)
    month_name = months[int(month_str) - 1]
    year = int(year_str)
    ordinals = {1: "first", 2: "second", 3: "third", 21: "twenty-first",
                23: "twenty-third"}
    day_word = ordinals.get(day, str(day))
    return f"{day_word} of {month_name} {year}"

print(normalize_date("23/08/2023"))

Trace it by hand. token.split("/") on "23/08/2023" gives ["23", "08", "2023"], so day_str, month_str, year_str = "23", "08", "2023". int(month_str) - 1 is int("08") - 1 = 7, and months[7] — counting from index 0 (january) — lands on "august". day = int("23") = 23, and ordinals.get(23, ...) finds the key 23 already in the dictionary, returning "twenty-third". year = int("2023") = 2023. The f-string then assembles "twenty-third of august 2023", which is exactly what print outputs. Notice the honest gap: the function still prints the numeral 2023 rather than expanding it to "twenty twenty-three," because year-expansion is its own rule set (two-digit-pair reading for 1100–2099, "two thousand and X" for the 2000s in some styles) layered on top of what is shown here. Real normalizers, such as the ones inside eSpeak NG or Google's Kestrel/Sparrowhawk toolkits, are large collections of exactly this kind of rule, each disambiguated by part-of-speech tags and surrounding tokens — which is precisely why normalization, not synthesis, is where most of a production TTS system's engineering effort and failure cases live.

Stage 2: grapheme-to-phoneme conversion and homographs

Once the text is normalized into plain words, G2P conversion maps those words onto phonemes — the roughly 40 discrete sound units of spoken English (this count varies by accent standard; the CMU Pronouncing Dictionary's ARPAbet lists 39). For most words this is a dictionary lookup: "namaste" (a loanword, so pronunciation must be supplied rather than derived from English spelling rules) maps to the phoneme sequence /n ə m ə s t eɪ/ — seven units: the consonant /n/, the reduced vowel /ə/ ("schwa," as in the "a" of "sofa"), /m/, /ə/ again, /s/, /t/, and the diphthong /eɪ/ as in "day." For novel or rare words not in any dictionary, a G2P model — historically rule-based, now typically a small sequence-to-sequence neural network trained on dictionary entries — predicts phonemes directly from spelling.

The genuinely hard cases are heteronyms: words spelled identically but pronounced differently depending on grammatical role. "Lead" is the clearest example. As a verb ("lead the team"), it is /liːd/, rhyming with "seed." As a noun for the metal ("a lead pipe"), it is /lɛd/, rhyming with "bed." A G2P system cannot resolve this from spelling alone — it needs a part-of-speech tag or a small classifier trained on surrounding context. "Record" is a stress-shift heteronym: the noun is RE-cord (/ˈrɛk.ɚd/, stress on the first syllable) and the verb is re-CORD (/rɪˈkɔːrd/, stress on the second). Get the stress wrong and the sentence is still intelligible but sounds distinctly foreign — which is exactly the class of error that separates a merely-functional TTS system from a natural-sounding one.

Stage 3: the acoustic model — phonemes to mel-spectrogram

The acoustic model is where the "neural" part of neural TTS begins in earnest. Its job is to consume a sequence of a few dozen phonemes and output a mel-spectrogram: a 2-D grid where one axis is time (in short overlapping frames) and the other is frequency, compressed onto the mel scale — a nonlinear frequency axis chosen because it matches how human hearing resolves pitch (finely at low frequencies, coarsely at high ones), so it wastes less representational capacity on frequency detail the ear cannot use.

The core architectural challenge is length mismatch: a phoneme sequence for "namaste" has 7 units, but the audio for that word, digitized at 22,050 samples per second, may last half a second — tens of thousands of raw samples, or (as derived below) dozens of spectrogram frames. Systems like Tacotron 2 (Google, 2017) solve this with an encoder-decoder architecture borrowed from sequence-to-sequence translation: an encoder turns the phoneme sequence into a sequence of embeddings, and a decoder generates the mel-spectrogram one frame at a time, at each step using an attention mechanism to decide which phoneme embeddings are currently "being spoken." The crucial difference from machine translation is that speech attention is expected to be monotonic — frame 40 of the audio should attend mostly to a phoneme at or after wherever frame 20 attended, because speech, unlike translation, never reorders its source units. When a trained Tacotron-style model's attention matrix is plotted with phonemes on one axis and output frames on the other, a well-trained model shows a roughly diagonal band; a badly trained one shows attention jumping around or collapsing onto a single phoneme, which is exactly the failure mode that produces repeated syllables or dropped words in bad synthetic speech. Newer non-autoregressive models (FastSpeech and its successors) replace this learned attention with an explicit duration predictor — a small network that directly estimates how many spectrogram frames each phoneme should occupy — trading some naturalness in rare cases for much faster, parallel generation and far more stable output (no repeated or skipped words).

Worked example: tracing the spectrogram's actual dimensions

The pipeline diagram above claims a 3-second utterance produces "80 mel bins × 255 frames." That number is not asserted — it follows from the standard windowing arithmetic used to compute a spectrogram, and it is worth deriving by hand once so the abstraction "mel-spectrogram" stops being a black box.

A spectrogram is built by sliding a fixed-length analysis window across the raw audio and computing one frequency spectrum per window position. Three parameters control the output size: the window length (how many raw samples each frame's spectrum is computed from), the hop length (how many samples the window advances between frames — this is smaller than the window length so consecutive frames overlap), and the total number of samples in the audio. Given n_samples raw samples, a window of length win_length, and a hop of hop_length, the number of complete frames that fit (without padding the signal's edges) is:

def n_mel_frames(n_samples, win_length, hop_length):
    """Number of un-padded STFT frames, matching librosa's center=False mode."""
    return 1 + (n_samples - win_length) // hop_length

sample_rate = 22050
duration_s  = 3.0
n_samples   = int(sample_rate * duration_s)   # 66150
win_length  = 1024
hop_length  = 256
n_mels      = 80

frames      = n_mel_frames(n_samples, win_length, hop_length)
mel_values  = frames * n_mels

print(n_samples, frames, mel_values)

Trace the arithmetic. n_samples = int(22050 * 3.0) = 66150 — this is simply "how many amplitude measurements make up 3 seconds of audio sampled 22,050 times per second," which is the standard rate used by Tacotron-family models (half of the 44,100 Hz CD-audio rate). Inside n_mel_frames: n_samples - win_length = 66150 - 1024 = 65126. Integer division 65126 // 256: since 256 × 254 = 65024 and 256 × 255 = 65280 > 65126, the quotient is 254. Adding the + 1 for the first frame gives frames = 255. Finally mel_values = 255 × 80 = 20400. So print(n_samples, frames, mel_values) outputs exactly 66150 255 20400 — matching the diagram.

The number 255 is worth sitting with. The acoustic model does not predict 66,150 raw numbers directly (that would be an enormous, extremely fine-grained sequence for an attention mechanism to manage one step at a time); it predicts only 255 frames, each a modest 80-dimensional vector. Dividing back, 66150 / 255 ≈ 259.4 raw samples correspond to each spectrogram frame — close to, but not exactly, the 256-sample hop length, because the last few samples at the tail of the signal do not form a complete window under this center=False convention and are simply dropped. This gap between "255 compact frames" and "66,150 raw samples" is precisely the job description of the next stage.

Stage 4: the vocoder — from spectrogram back to sound

A mel-spectrogram tells you the magnitude of energy at each frequency band, frame by frame. It deliberately discards phase — the precise timing offset of each frequency component within a frame — because phase is largely inaudible to human hearing but numerically expensive to predict accurately, and mel compression itself is lossy (many different raw waveforms can produce a very similar mel-spectrogram). The vocoder's job is to invert this: given only the 255×80 grid of magnitudes, synthesize a full 66,150-sample waveform that sounds correct, including a phase structure the spectrogram never specified.

The simplest approach, the Griffin-Lim algorithm, does this without learning anything: it iteratively guesses a phase, reconstructs a waveform, re-computes the spectrogram of that guess, and corrects the phase estimate, repeating until it converges. It works, is fast, and sounds noticeably metallic and buzzy — a telltale "robot voice" texture — because the phase it invents is only mathematically self-consistent, not perceptually natural. WaveNet (DeepMind, 2016) replaced this with a neural network trained to predict raw audio directly: an autoregressive model built from stacked dilated causal convolutions that predicts each waveform sample from the samples before it, conditioned on the mel-spectrogram, quantizing 16-bit audio down to 256 possible amplitude levels (via μ-law companding) so the output layer is a simple 256-way softmax. It produced dramatically more natural speech than anything before it, at the cost of generating audio one sample at a time — for a 3-second clip that means running the network roughly 66,150 times in strict sequence, which made real-time synthesis on ordinary hardware impractical for years. HiFi-GAN (2020) is the modern default precisely because it drops the autoregression: it is a generative adversarial network whose generator upsamples the entire 255-frame mel-spectrogram to the full 66,150-sample waveform in a single parallel forward pass (through a stack of transposed convolutions, each one expanding the time axis, together multiplying it by roughly the hop length of 256), while a discriminator network is trained simultaneously to tell real recorded audio from HiFi-GAN's output, pushing the generator toward waveforms that are not just spectrally plausible but texturally indistinguishable from a microphone recording. This parallel, non-autoregressive structure is why current assistants can synthesize speech faster than real time on modest hardware, in a way frame-by-frame WaveNet-style vocoders never could.

Correcting a common misconception

The mental model most students bring to TTS is that it works like an interactive voice response (IVR) menu — the phone-banking systems that say "for balance enquiry, press one" — stitching together pre-recorded word clips on demand. That was, in fact, how many production TTS systems worked before the mid-2010s: concatenative unit-selection synthesis recorded a voice actor saying thousands of sentences, sliced the audio into small units (often smaller than whole words — diphones, spanning the transition between two phonemes), and at runtime searched a huge database for the sequence of recorded units that best matched the target phoneme sequence, then smoothed the joins.

Modern neural TTS, the pipeline this chapter describes, does not do this. Tacotron2-style acoustic models and HiFi-GAN-style vocoders never play back a stored audio clip — every waveform is generated from scratch by a neural network's learned parameters, conditioned on the phonemes of that specific sentence. This is not a pedantic distinction. Concatenative systems degrade badly on exactly the material this chapter's opening hook was built from — numbers, train codes, rare proper nouns — because there is no pre-recorded unit for a word the voice actor never spoke, forcing the system to either fail outright or splice together jarringly mismatched fragments. A neural acoustic model has no such gap: because it generates a continuous mel-spectrogram from phonemes it has learned to predict for arbitrary sequences, it produces smooth, naturally co-articulated speech for "12951" or "Chandrayaan-3" exactly as fluently as for any sentence in its training data, since at inference time it is not retrieving anything — it is computing a new prediction from the model's parameters every time.

Active recall

Attempt these before reading the answers below.

1. Why would a text-to-speech system that only trains its neural network on recorded audiobook sentences still fail badly on the sentence "Dr. Rao's clinic is on MG Dr., near flat no. 4B"?
2. A TTS acoustic model is set up with a sample rate of 16,000 Hz, a window length of 800 samples, and a hop length of 200 samples. For a 2-second utterance, compute the number of raw samples and the number of spectrogram frames (using the same 1 + (n_samples - win_length) // hop_length formula derived in this chapter).
3. Why is the attention alignment inside a Tacotron2-style acoustic model expected to look roughly diagonal, and what does a badly collapsed (non-diagonal) alignment tend to produce in the output audio?
4. A mel-spectrogram already tells the vocoder how much energy is present at each frequency, frame by frame. Why is that not enough information to reconstruct the waveform directly, without any learning or iteration?
5. Give the two different pronunciations of the heteronym "live" (as in "I live here" versus "a live wire") and explain, in one sentence, what information a G2P system needs beyond the spelling to choose correctly.

Worked answers

1. The sentence has two separate normalization traps for the token "Dr." — the first "Dr." (before "Rao") should expand to "doctor," while the second "Dr." (after "MG," as a street-name suffix) should expand to "drive." A system trained only on already-normalized audiobook sentences never had to learn this disambiguation rule during acoustic-model training, because normalization is a separate, upstream stage — no amount of additional acoustic-model training fixes a normalization failure, since the acoustic model never sees the raw abbreviation at all, only whatever the normalizer decided to hand it.

2. n_samples = 16000 × 2 = 32000. Frames: 1 + (32000 − 800) // 200 = 1 + 31200 // 200. Since 31200 / 200 = 156 exactly, the integer division gives 156, so frames = 1 + 156 = 157.

3. Speech, unlike translation, never reorders its source units — the third phoneme spoken is always acoustically realized before the fourth, never after. So as the decoder steps forward through output frames, the attended phoneme index should only ever stay the same or advance, producing a roughly diagonal band when alignment weight is plotted with phonemes on one axis and frames on the other. A collapsed or erratic (non-diagonal) alignment means the decoder is attending to the wrong phoneme at some frame, which typically manifests as a skipped phoneme (a word or syllable silently dropped) or a repeated one (the model "gets stuck" attending to the same phoneme across many frames, producing a stutter or drawn-out sound).

4. A mel-spectrogram stores only magnitude (how much energy) per frequency band per frame; it discards phase (the precise timing offset of each frequency component). Two waveforms with identical magnitude spectra but different phase can sound nearly identical to a human ear but are numerically very different signals, so reconstructing a waveform requires either inventing a plausible phase through iteration (Griffin-Lim) or a model that has learned, from training data, what phase patterns are perceptually right for a given magnitude pattern (a neural vocoder).

5. As a verb ("I live here"), "live" is pronounced /lɪv/, with the short vowel of "give." As an adjective ("a live wire"), it is pronounced /laɪv/, with the long vowel of "five." A G2P system needs the word's part of speech (or equivalently, its syntactic context in the sentence) to choose correctly, since the spelling alone is identical in both cases.

Think About It

Think about this: How would you explain text-to-speech: generating natural audio to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind text-to-speech: generating natural audio, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.