Speech Recognition and Audio Processing

Call the Indian Railways enquiry line and ask for a PNR status by voice, and somewhere behind that phone call is a chain of numeric transformations turning air pressure into text. A caller says "eight one two four five six seven eight nine zero," the system has roughly a second per digit to decide what was said, and it has to do this over a compressed 8 kHz telephone line, in a noisy station, in an Indian-accented mix of Hindi and English digit words. This chapter builds that chain from first principles: how a continuous sound wave becomes numbers a computer can hold, how those numbers become features that expose phonetic content, how a neural network turns features into phoneme probabilities, and how a language model resolves the ambiguity that acoustics alone cannot.

From sound wave to numbers: sampling and quantization

A voice is a continuous pressure wave, x(t), a function of continuous time. A computer cannot store a continuous function; it must record x(t) at discrete instants. Sampling at rate fs means recording x(0), x(1/fs), x(2/fs), and so on. The question that decides everything downstream is: how often is often enough?

The Nyquist–Shannon theorem answers this precisely: to reconstruct a signal without ambiguity (aliasing), the sampling rate must be at least twice the highest frequency component present, fs ≥ 2·f_max. Human vocal-tract resonances (formants) that carry phonemic identity sit mostly below 3.5 kHz, but fricatives like /s/ and /f/ carry energy up to 8 kHz. Telephone networks were engineered around this trade-off decades before deep learning existed: they band-limit speech to about 4 kHz and sample at 8 kHz, exactly at the Nyquist bound for that band. This is why "telephone quality" audio sounds thin but is still fully intelligible: 8 kHz sampling captures everything needed to reconstruct a 4 kHz-limited voice signal without loss.

After sampling comes quantization: each sample's amplitude is rounded to one of 2^b levels, where b is the bit depth. Telephone-grade PCM audio uses 8 kHz sampling at 8 bits per sample. The bitrate is then 8000 samples/s × 8 bits/sample = 64,000 bits/s = 64 kbps. For a 300 ms spoken digit, the sample count is 8000 × 0.3 = 2400 samples, and the storage is 64,000 × 0.3 = 19,200 bits = 2400 bytes. These 2400 raw samples are the starting point for every later stage in this chapter, so keep that number in mind.

Framing: speech is only stationary in short windows

Speech is not statistically stable over long stretches: the vocal tract shape driving a "t" sound is completely different from the shape driving an "a" a few milliseconds later. But over a short enough window, roughly 20–30 ms, the signal is quasi-stationary: the same vocal-tract configuration holds still long enough for frequency-domain analysis to make sense. So the raw sample stream is chopped into overlapping frames, each analyzed independently. A frame length of 25 ms and a hop (stride) of 10 ms is a standard choice, giving 15 ms of overlap so that features do not change abruptly frame to frame.

Framing is exactly a sliding-window operation over the sample array. Here is the mechanic in code, traced on a small example before applying it to the real digit case:

def frame_signal(signal, frame_size, hop_size):
    frames = []
    start = 0
    while start + frame_size <= len(signal):
        frames.append(signal[start:start + frame_size])
        start += hop_size
    return frames

signal = list(range(20))                      # 20 raw samples: 0..19
frames = frame_signal(signal, frame_size=8, hop_size=4)
print(len(frames), frames)

Trace it: start begins at 0. The loop condition is start + 8 ≤ 20, i.e. start ≤ 12. Valid starts are 0, 4, 8, 12 (start=16 would give 16+8=24 > 20, so the loop stops). The four slices are [0:8], [4:12], [8:16], [12:20]. So the printed output is exactly:

4 [[0, 1, 2, 3, 4, 5, 6, 7], [4, 5, 6, 7, 8, 9, 10, 11], [8, 9, 10, 11, 12, 13, 14, 15], [12, 13, 14, 15, 16, 17, 18, 19]]

Now apply the same logic to the 2400-sample, 300 ms digit utterance from earlier, with a real 25 ms frame (200 samples at 8 kHz) and a 10 ms hop (80 samples). The condition start + 200 ≤ 2400 gives start ≤ 2200. Valid multiples of 80 up to 2200 run 0, 80, 160, …, 2160 (since 2160 ≤ 2200 but 2240 > 2200). That is 2160/80 + 1 = 27 + 1 = 28 frames covering one spoken digit. Each of these 28 frames now needs to be converted into a feature vector before any model sees it.

Before framing, each frame is usually multiplied by a smooth window function (a Hamming or Hann window) rather than cut with a hard rectangular edge. A rectangular cut introduces spurious high-frequency energy at the frame boundaries (spectral leakage) purely as an artifact of chopping, not because the speech signal actually contains that energy; a tapered window suppresses this artifact before the frequency analysis in the next step.

Feature extraction: why frequency, not amplitude

Feeding raw amplitude samples into a classifier is a poor choice, and this is worth stating precisely because it is a genuine engineering decision, not an arbitrary convention. Two recordings of the same word spoken by the same person half a second apart will almost never align sample-for-sample: pitch varies, speaking rate varies, phase varies. A model comparing raw amplitudes sees two very different inputs for the same underlying sound. What stays comparably stable across these variations is the distribution of energy across frequency bands. This is why every practical pipeline, whether a 1990s HMM system or a 2023 Transformer, transforms each frame into a spectral representation before any acoustic model consumes it.

The standard path is: take the Fast Fourier Transform of each windowed frame to get a power spectrum, pass that spectrum through a bank of triangular filters spaced on the Mel scale rather than linearly, take the logarithm of each filter's output energy, and finally apply a Discrete Cosine Transform to decorrelate the log-energies into Mel-Frequency Cepstral Coefficients (MFCCs). The Mel scale exists because human pitch perception is not linear in Hertz: listeners resolve differences between low frequencies far more finely than equally-sized differences at high frequencies. The standard conversion is:

mel(f) = 2595 × log10(1 + f / 700)

Check this at f = 1000 Hz: 1 + 1000/700 = 2.4286, and log10(2.4286) ≈ 0.3853, so mel(1000) ≈ 2595 × 0.3853 ≈ 1000. The constants 700 and 2595 were chosen precisely so that 1000 Hz maps to almost exactly 1000 mel, a convenient anchor point. Now check f = 8000 Hz: 1 + 8000/700 = 12.4286, log10(12.4286) ≈ 1.0944, giving mel(8000) ≈ 2595 × 1.0944 ≈ 2840. An 8-fold increase in Hertz (1000→8000) produced only a 2.84-fold increase in mel (1000→2840). That compression is the entire point: it allocates more filters, and therefore more discriminative resolution, to the low-frequency region where vowel formants live, matching what the human ear actually resolves.

The acoustic model: from phoneme probabilities to CTC

The acoustic model's job is to map the sequence of feature vectors (one per frame, 28 of them for the digit example) to a sequence of phoneme or character probabilities. Early systems (Gaussian Mixture Model–Hidden Markov Model, GMM-HMM, through the 1980s–2000s) modeled each phoneme state's feature distribution with a Gaussian mixture and used the HMM's transition structure to sequence phonemes into words. From roughly 2012 onward these Gaussians were replaced by deep neural networks (DNN-HMM hybrids), and by the late 2010s fully end-to-end neural architectures removed the HMM scaffolding entirely: Connectionist Temporal Classification (CTC) networks, RNN-Transducers, and attention-based encoder-decoders. Wav2vec 2.0 (Baevski et al., Facebook AI Research, 2020) pretrains a convolutional-plus-Transformer encoder on unlabeled audio and fine-tunes it with a CTC loss on limited labeled data; Whisper (Radford et al., OpenAI, 2022) instead trains a Transformer encoder-decoder directly on 680,000 hours of weakly labeled multilingual audio scraped from the web, learning transcription end to end without a separate pretraining stage.

CTC deserves a closer look because it solves a real alignment problem elegantly. A neural encoder outputs one probability distribution per input frame, but the frame count (28, in our example) rarely equals the number of output characters in the transcription, and the network is never told which frame corresponds to which letter during training. CTC's trick is to add a special "blank" symbol to the output vocabulary and define a deterministic collapsing rule: consecutive repeated labels collapse into a single instance, then blanks are stripped out entirely. This lets the network freely repeat a label across several consecutive frames (because a phoneme like the vowel in "cat" spans multiple 10 ms hops) without those repeats being misread as three separate letters, and it lets the network use blanks to separate two genuinely repeated letters (like the double "t" in "letter") from an accidental repeat.

Trace this on a worked example. Suppose the acoustic model, run frame-by-frame on the target word "CAT," produces the following arg-max label at each of 10 timesteps: C, C, -, A, -, -, T, T, T, - (using - for blank). The collapsing code is:

def ctc_collapse(labels):
    collapsed = []
    prev = None
    for l in labels:
        if l != prev:
            collapsed.append(l)
        prev = l
    return [l for l in collapsed if l != '-']

labels = ['C', 'C', '-', 'A', '-', '-', 'T', 'T', 'T', '-']
print(''.join(ctc_collapse(labels)))

Trace it step by step. prev starts as None. Step 1, l='C', differs from prev, append C, prev='C'. Step 2, l='C', equals prev, skip. Step 3, l='-', differs, append -, prev='-'. Step 4, l='A', differs, append A, prev='A'. Step 5, l='-', differs, append -, prev='-'. Step 6, l='-', equals prev, skip. Step 7, l='T', differs, append T, prev='T'. Steps 8–9, l='T' twice, equal prev both times, skip. Step 10, l='-', differs, append -. The collapsed list is ['C', '-', 'A', '-', 'T', '-']; stripping blanks leaves ['C', 'A', 'T']. The printed output is exactly CAT.

The diagram: the full pipeline in one view

The figure below traces every stage covered so far, from the raw waveform through framing, Mel-filterbank feature extraction, the neural acoustic model, CTC collapsing on the worked "CAT" example, and finally fusion with a language model inside a beam-search decoder to produce the transcript.

Why the acoustic model alone cannot finish the job

CTC decoding gives an acoustic hypothesis, but acoustics alone cannot always determine the correct word, because some words are pronounced identically. "Eight" and "ate" are true homophones: the acoustic model produces the same probability for both, by construction, since the pronunciation is the same phoneme sequence, /eɪt/. Say the acoustic model assigns P_acoustic("eight") = P_acoustic("ate") = 0.42 for a given frame span. A pure acoustic decoder has no way to break this tie. This is exactly where a language model earns its place in the pipeline: a bigram model trained on transcripts of digit-reading calls will have learned that after the word "PNR," a numeral word is overwhelmingly likely and a past-tense verb is not. Suppose the trained bigram probabilities are P("eight" | "PNR") = 0.08 and P("ate" | "PNR") = 0.00007. The decoder's joint score multiplies the two: score("eight") = 0.42 × 0.08 = 0.0336, versus score("ate") = 0.42 × 0.00007 = 0.0000294. The language model's contribution decides the outcome by more than three orders of magnitude, even though the acoustic evidence was perfectly tied. This is also where Indian ASR systems face a genuinely hard version of this problem: callers frequently code-switch between Hindi and English mid-utterance ("PNR ekdum eight one two hai," for instance), and a language model trained only on monolingual English text assigns near-zero probability to such a sequence regardless of how clearly it was spoken. Systems built for Indian languages, including India's Bhashini initiative for multilingual language technology, must train language models on genuinely code-mixed transcripts rather than assuming a single-language context.

The standard way to measure how well the whole pipeline performs is Word Error Rate (WER), defined as WER = (S + D + I) / N, where S is substitutions, D is deletions, I is insertions needed to turn the hypothesis into the reference, and N is the word count of the reference. Take reference "PNR EIGHT ONE TWO" (N = 4 words) against a hypothesis "PNR ATE ONE TWO": one substitution (EIGHT → ATE), zero deletions, zero insertions, so WER = 1/4 = 25% for that single utterance, even though only one homophone was mistaken.

Common misconception

Many students assume that higher sampling rate always means better recognition accuracy, on the logic that "44.1 kHz is CD quality, so it must transcribe better than 8 kHz telephone audio." This overstates a relationship that actually saturates. The information that distinguishes one phoneme from another lives mostly below 4 kHz (vowel formants) with some fricative energy extending to about 8 kHz; almost nothing phonetically discriminative exists above that. This is precisely why the telephone network standardized on 8 kHz decades ago, and why modern neural ASR systems such as Whisper and wav2vec 2.0 train on 16 kHz audio rather than 44.1 kHz: 16 kHz (Nyquist limit 8 kHz) captures the fricative energy that 8 kHz telephone sampling misses, giving a real accuracy gain over 8 kHz, but pushing further to 44.1 kHz buys almost nothing extra for transcription, because there is essentially no additional phoneme-discriminating content in the 8–22 kHz range; those extra bits exist to serve music fidelity and human listening pleasure, not phoneme identity. Sampling rate helps recognition only up to the point where it captures all phonetically relevant frequencies, and then further increases are wasted on this specific task.

Active recall

Attempt these before reading the worked answers below.

• 1. Speech energy relevant to phoneme identity extends up to about 8 kHz (including fricatives). What is the minimum sampling rate needed, by the Nyquist theorem, to avoid aliasing that content?
• 2. In the 300 ms, 2400-sample digit example, a 25 ms frame (200 samples) with a 10 ms hop (80 samples) gave 28 frames. Suppose the frame size is doubled to 50 ms (400 samples) while the hop stays at 10 ms (80 samples). (a) How many frames now cover the utterance? (b) Name one other property of the feature extraction, besides the frame count, that changes, and state the direction of the change.
• 3. Using the same CTC collapsing rule from the worked "CAT" example, decode this 9-timestep arg-max label sequence: P, P, A, A, -, -, N, -, -.
• 4. Why is feeding raw time-domain amplitude samples directly into a classifier a poor design choice for speech recognition, compared to feeding it Mel-spectral features?
• 5. Reference transcript: "TRAIN NUMBER ONE TWO FIVE FIVE" (6 words). ASR hypothesis: "TRAIN NUMBER WON TWO FIVE" (5 words: ONE was misheard as its homophone WON, and the second FIVE was dropped entirely). Compute the Word Error Rate.

Worked answers

• 1. Nyquist requires fs ≥ 2 × f_max = 2 × 8000 = 16,000 Hz, i.e. 16 kHz. This is exactly why modern ASR models train on 16 kHz audio rather than the 8 kHz telephone standard.
• 2. (a) Frame count uses the same rule as the code: valid starts satisfy start + 400 ≤ 2400, so start ≤ 2000. Multiples of 80 up to 2000 run 0, 80, …, 2000, which is 2000/80 + 1 = 25 + 1 = 26 frames (down from 28). (b) Frequency resolution improves: bin spacing is fs / N, which drops from 8000/200 = 40 Hz to 8000/400 = 20 Hz, giving finer spectral detail. But time resolution worsens in exchange: each frame now smears 50 ms of signal instead of 25 ms, blurring fast transients such as stop-consonant bursts (/p/, /t/, /k/). This is the standard short-time-Fourier-transform time-frequency trade-off, and it is a consequence of the same frame-size parameter the question asked about for the frame count.
• 3. Trace the collapse: prev=None; P differs, append P, prev=P; P equals, skip; A differs, append A, prev=A; A equals, skip; - differs, append -, prev=-; - equals, skip; N differs, append N, prev=N; - differs, append -, prev=-; - equals, skip. Collapsed list: [P, A, -, N, -]. Strip blanks: [P, A, N], giving decoded string "PAN".
• 4. Raw amplitude samples are highly sensitive to nuisance variation: the same phoneme spoken twice, even by the same speaker, rarely lines up sample-for-sample because pitch, speaking rate, and phase differ each time. A classifier comparing raw waveforms sees very different inputs for the same sound. Mel-spectral features summarize energy across perceptually meaningful frequency bands rather than instantaneous pressure, which stays far more stable across those variations, so every practical acoustic model, whether an old GMM-HMM or a modern Transformer, is fed a spectral or learned convolutional transformation of the audio rather than raw PCM values.
• 5. Comparing "TRAIN NUMBER ONE TWO FIVE FIVE" to "TRAIN NUMBER WON TWO FIVE": ONE was substituted with WON (S = 1), the second FIVE was deleted with no replacement (D = 1), and no extra words were inserted (I = 0). Reference length N = 6. WER = (1 + 1 + 0) / 6 = 2/6 ≈ 33.3%.

Think About It

Think about this: How would you explain speech recognition and audio processing to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where speech recognition and audio processing is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting speech recognition and audio processing to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind speech recognition and audio processing, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.