Picture a fraud-reporting line run by an Indian bank. A customer calls in a panic: "Sir mera paisa cut ho gaya, turant block karo card!" — half Hindi, half English, voice shaking, words tumbling over each other. Somewhere downstream, an agent (human or automated) has to decide whether to freeze the card in the next ten seconds or ask clarifying questions. Now picture the naive way to build this: run the call through a speech-to-text model, hand the resulting sentence to a language model, get back an instruction, hand that to a text-to-speech model, play it back. Three models, three hand-offs, and at every hand-off the system throws something away. The transcript "sir mera paisa cut ho gaya turant block karo card" reads identically whether the caller is mildly annoyed or in genuine distress — but the waveform doesn't. Pitch, tempo, breathiness, the tiny catch in the voice: all of it evaporates the moment audio becomes text. A model that must reason about urgency, sarcasm, or hesitation cannot do so from a transcript alone.
This is the actual engineering problem behind "audio-language models": not just converting speech to text (that's automatic speech recognition, ASR, a solved-enough problem since the 2010s), but building a single model that keeps both modalities — the continuous acoustic signal and the discrete symbolic language — inside one shared representation space, so that reasoning, generation, and paralinguistic information can flow across the modality boundary instead of being destroyed at it. This chapter builds that bridge from first principles: how a waveform becomes something a transformer can attend to, how it becomes something a transformer can generate, and what the two dominant architectural families — spectrogram-encoder models like Whisper, and discrete-token unified models like AudioLM, AudioPaLM, and Moshi — actually do differently.
Two Incompatible Representations
You already know, from Grade 11, that a text language model consumes a sequence of discrete integers — token IDs from a fixed vocabulary of tens of thousands of byte-pair-encoded subwords — and predicts a probability distribution over the next one. That works because text is already discrete: there are finitely many tokens, and a sentence is a short, sparse sequence of them (a sentence has maybe 20–40 tokens).
Raw audio is the opposite of that on every axis. A phone call sampled at 16,000 Hz produces 16,000 floating-point amplitude values per second. A single 10-second utterance is a sequence of 160,000 continuous numbers — roughly 4,000–8,000 times longer than the text transcript of the same utterance, and none of those numbers is drawn from a finite vocabulary. You cannot feed this directly into a standard transformer: self-attention costs O(n²) in sequence length, and 160,000 tokens of attention is computationally absurd for what is, informationally, one short sentence. Before any language modeling can happen, audio has to be compressed — in both senses of the word: made shorter, and made to live in a space compatible with what the rest of the model expects.
There are exactly two engineering answers to "what does that compressed representation look like," and each one defines a different family of audio-language model.
Path One: Continuous Encoders and Cross-Attention (Whisper)
The first answer, used by OpenAI's Whisper (Radford et al., 2022, "Robust Speech Recognition via Large-Scale Weak Supervision"), keeps the audio representation continuous but shortens it drastically using signal-processing tricks your G11 signals-adjacent math already supports: the short-time Fourier transform and mel-scale binning.
Whisper standardizes every input to a 30-second window at a 16,000 Hz sample rate. It slides a 25-millisecond analysis window across the audio, hopping forward 10 milliseconds each time, and at each window position computes the power spectrum and bins it into 80 mel-frequency channels (a perceptually motivated frequency scale, denser at low frequencies where the human ear — and most phonetic information — is most sensitive). The result is a "log-mel spectrogram": an 80-channel image where one axis is frequency and the other is time.
Let's derive the exact shape of that image, step by step, because every number in it is fixed by the sample rate and the window/hop choices above — nothing here is approximate.
Sample rate: 16,000 Hz
Chunk length: 30 seconds
Total samples: 30 × 16,000 = 480,000
Hop length: 10 ms × 16,000 = 160 samples/frame-step
Number of frames: 480,000 ÷ 160 = 3,000 (exact integer division)
So the spectrogram Whisper hands to its encoder is an 80 × 3,000 matrix — 80 mel channels, 3,000 time frames, one frame every 10 ms across the 30-second window. That is already a 160× reduction from 480,000 raw samples to 3,000 spectrogram frames, but 3,000 is still too long for efficient self-attention across many transformer layers, so Whisper's encoder front-end applies two 1-D convolutions over the time axis: the first with stride 1 (kernel size 3, padding 1, so it preserves length), the second with stride 2 (kernel size 3, padding 1). Convolution output length follows the standard formula
L_out = floor((L_in + 2 × padding - kernel_size) / stride) + 1
Second conv: L_in = 3,000, padding = 1, kernel_size = 3, stride = 2
L_out = floor((3,000 + 2 - 3) / 2) + 1 = floor(2,999 / 2) + 1 = 1,499 + 1 = 1,500
So after the convolutional stem, the encoder works with a sequence of exactly 1,500 audio embedding vectors — down from 480,000 raw samples, a 320× reduction, achieved with zero learned quantization, purely through fixed-hop framing and one strided convolution. Sinusoidal positional embeddings are added to these 1,500 vectors, and a standard multi-head self-attention transformer encoder (structurally identical to the encoder stack you studied in G11 for text) processes them. A separate autoregressive decoder then generates output text tokens one at a time, attending to the 1,500 encoder vectors through cross-attention at every layer — the same cross-attention mechanism from the original Vaswani et al. (2017) encoder-decoder transformer, just with audio embeddings standing in for the source-language tokens of a translation model. Special control tokens in the decoder's vocabulary (language ID, "transcribe" vs "translate," timestamp markers) let one architecture handle multiple tasks by conditioning generation on a task tag, rather than needing separate models.
Notice what this architecture never does: it never turns audio into a fixed vocabulary of audio "words." The 1,500 vectors are dense, continuous, and exist only to be cross-attended to — they are never generated, never sampled, never predicted as a next token. This makes Whisper excellent at the one-directional task it was built for (audio in, text out) but structurally unable to generate audio, and unable to interleave audio and text within a single autoregressive stream. For that, you need the second path.
Path Two: Discretize Audio, Then Unify the Vocabulary
The second answer — used by Google's AudioLM (Borsos et al., 2023), AudioPaLM (Rubenstein et al., 2023), and Kyutai's Moshi (Défossez et al., 2024) — takes a more radical step: it forces audio to become genuinely discrete, so that audio "tokens" and text tokens can sit in the exact same vocabulary and be modeled by one ordinary decoder-only transformer doing next-token prediction, the same objective you already know from text LLMs.
The tool that makes this possible is a neural audio codec built on residual vector quantization (RVQ), the architecture introduced in SoundStream (Zeghidour et al., 2021) and refined in EnCodec (Défossez et al., 2022). A codec encoder (a convolutional network, not a transformer) downsamples the raw waveform to a low frame rate — illustratively, tens of frames per second rather than 16,000 samples per second — producing one continuous vector per frame. That vector is then quantized not by one codebook but by a cascade: the first codebook finds its nearest of, say, 1,024 entries and represents the biggest chunk of the vector's information; the quantization error left over (the "residual") is passed to a second codebook, which quantizes what the first one missed; its residual goes to a third codebook, and so on. Stacking several codebooks this way lets a small number of small codebooks jointly approximate a rich continuous vector — this is literally the same residual-approximation idea as representing a number in binary one bit at a time, except each "digit" here is a 1,024-way choice instead of a 2-way one. The codec is trained end-to-end (encoder, quantizer, decoder) with reconstruction and adversarial losses so that decoding the chosen codes back through a decoder network reproduces a waveform perceptually close to the original — this decoder-back-to-waveform step is exactly what a "vocoder" does, and in a neural codec it's built in.
Once every audio frame is a fixed tuple of small integers (one per codebook), those integers can be treated exactly like BPE token IDs — the only remaining trick is making sure they don't collide with the text vocabulary. This is done by offsetting each audio token ID past the end of the text vocabulary, so the LLM's embedding table simply grows to include audio entries as new "words." Trace this precisely for a 4-codebook example:
TEXT_VOCAB_SIZE = 50257 # BPE vocabulary, e.g. GPT-2 style
N_CODEBOOKS = 4
CODEBOOK_SIZE = 1024
def audio_token_id(codebook_idx: int, code: int) -> int:
offset = TEXT_VOCAB_SIZE + codebook_idx * CODEBOOK_SIZE
return offset + code
# one audio frame's RVQ codes, one integer per codebook
frame_codes = [512, 7, 900, 3]
unified_ids = [audio_token_id(i, c) for i, c in enumerate(frame_codes)]
print(unified_ids)
Trace it by hand, codebook by codebook:
codebook 0: offset = 50257 + 0×1024 = 50257 → 50257 + 512 = 50769
codebook 1: offset = 50257 + 1×1024 = 51281 → 51281 + 7 = 51288
codebook 2: offset = 50257 + 2×1024 = 52305 → 52305 + 900 = 53205
codebook 3: offset = 50257 + 3×1024 = 53329 → 53329 + 3 = 53332
print(unified_ids) # [50769, 51288, 53205, 53332]
The total vocabulary is now 50,257 + 4 × 1,024 = 54,353 entries — text and audio sharing one embedding table, one softmax, one next-token-prediction loss. A training sequence can now genuinely interleave the two: text tokens for a user's typed question, audio tokens for a spoken reply, text tokens for a system instruction, more audio tokens — all one flat sequence for one ordinary transformer decoder to model causally, no cross-attention module required. This is the architectural core of AudioPaLM (which initializes from a pretrained text LLM, PaLM-2, and extends its vocabulary this way so it inherits the base model's language competence) and of Moshi (which pushes this further into real-time full-duplex dialogue, generating a stream of interleaved text and audio tokens continuously rather than waiting for a full utterance, and interestingly, generates time-aligned inner text tokens alongside the audio tokens as a form of "thinking in words" before or while speaking).
Where the Two Paths Diverge in Practice
Whisper's design commits fully to one direction — audio in, text out — and buys, in return, a smaller model that trains stably on weakly-labeled web-scraped audio-transcript pairs at enormous scale (680,000 hours in the original paper) and produces high-accuracy transcription and translation. But it cannot speak, and it processes audio in fixed 30-second blocks with a bidirectional encoder, so a full block generally has to be available before decoding proceeds — awkward for a live phone call where you want the system responding while the caller is still talking.
The unified-vocabulary path is architecturally symmetric — the same transformer that consumes audio tokens can emit them — and can therefore stream: generate the next token, whether text or audio, conditioned only on everything so far, exactly like a text LLM generating one word at a time. That symmetry is what makes real-time, full-duplex spoken dialogue (the system can start responding, or even interrupt, while still "hearing" the user, because there's no separate encoder pass that must finish first) achievable in a way the cross-attention design does not naturally support. The cost is a lossier audio representation — quantization to a handful of codebooks discards more acoustic detail than a dense 1,500-vector spectrogram encoding — and a training regime that has to learn audio generation, not just audio understanding, which is a harder objective.
What Cascading Loses: Correcting a Common Misconception
It's tempting, having seen Whisper (audio→text) and a text-to-speech model (text→audio) both exist, to conclude that chaining ASR → LLM → TTS is "essentially the same thing" as a unified audio-language model, just assembled from three off-the-shelf pieces instead of trained as one. This is the single most common misconception about this topic, and it is wrong in a specific, checkable way: at each seam in that cascade, the interface is plain text, and text is a lossy, one-dimensional summary of speech.
The bank-helpline caller's transcript is identical whether they whispered it in dread or snapped it in irritation — the ASR model's job is precisely to discard everything except word identity, because that is what "transcription accuracy" is measured against. Downstream, the LLM reasons only over that flattened text; it has no way to recover the caller's actual affect, hesitation length, or emphasis, because those signals were never encoded into its input in the first place. And at the final seam, the TTS model synthesizes a voice with whatever default prosody it was trained to produce — it cannot mirror the input caller's urgency because it never received the input audio, only the LLM's text output. Three independently-optimized black boxes, three lossy interfaces, and the paralinguistic information that made the original call urgent is gone by the second hand-off, not the third.
A genuinely unified model — one that keeps audio (or a rich discretization of it) inside the same representation the language model reasons over — has the physical possibility of conditioning its response on pitch, tempo, and tone, because that information never left the model's input space. Whether a given unified model actually exploits that possibility well is a separate, empirical question — but a cascaded pipeline cannot exploit it even in principle, because the information has already been deleted by the time the language-reasoning component sees anything.
Training Objectives: One More Contrast Worth Knowing
Both paths above train their decoding component with ordinary next-token cross-entropy — the identical objective from G11's language modeling unit, just with a vocabulary that may include audio-token IDs. But it's worth knowing there is a third, older approach to the audio→text alignment problem that neither Whisper nor the unified models use: Connectionist Temporal Classification (CTC), introduced by Graves, Fernández, Gomez, and Schmidhuber (2006) and used to fine-tune self-supervised encoders like wav2vec 2.0 (Baevski, Zhou, Mohamed, and Auli, 2020). CTC solves a harder-looking problem directly: the input (audio frames) is much longer than the output (text characters or subwords), and there's no ground-truth alignment telling the model which audio frame corresponds to which output symbol. CTC handles this by summing, via dynamic programming, over every possible frame-to-symbol alignment consistent with the target sequence (including a special "blank" symbol standing for "emit nothing here"), and maximizing the total probability across all of them. This produces a model that outputs one symbol-or-blank per input frame — a very different generation pattern from Whisper's autoregressive decoder, which emits one token at a time attending back over the whole audio, and from the unified transformer, which treats every token, audio or text, identically inside one causal sequence.
Active Recall
Q1. Whisper's log-mel spectrogram for a 30-second, 16,000 Hz clip uses a 25 ms window and a 10 ms hop, giving 3,000 frames and, after the stride-2 convolution, 1,500 encoder vectors. Suppose the hop length were changed to 20 ms (window unchanged). Recompute the number of mel frames, the post-convolution encoder length, and state one downstream computational consequence of the change.
Q2. Using the 4-codebook vocabulary-merge scheme from the worked example (TEXT_VOCAB_SIZE = 50,257, CODEBOOK_SIZE = 1,024), suppose the codec instead used 8 codebooks. What unified token ID represents codebook index 5, code 999? What is the new total vocabulary size?
Q3. A friend argues: "Since Whisper outputs text and a TTS model turns text into speech, chaining Whisper → an LLM → TTS is basically the same as a unified audio-language model, just three pieces instead of one." Identify the specific information destroyed at each of the two seams in that pipeline, and explain why a unified token-space model does not have the same limitation even in principle.
Q4. Why can a unified decoder-only audio-text model (like Moshi) support real-time, interruptible, full-duplex conversation more naturally than Whisper's encoder-decoder cross-attention design?
Q5. CTC (used to fine-tune wav2vec 2.0) and Whisper's cross-attention decoder both solve "audio in, text out," but they differ in how they handle the fact that audio has many more frames than the output has symbols. Describe, in one or two sentences each, how each approach handles this length mismatch.
Q6. In the residual vector quantization scheme, what does the "residual" being passed from codebook k to codebook k+1 actually represent, and why does adding more codebooks improve reconstruction quality rather than just adding redundant information?
Worked Answers
A1. New hop = 20 ms × 16,000 Hz = 320 samples. Total samples is unchanged at 480,000. New frame count = 480,000 ÷ 320 = 1,500 mel frames (half of the original 3,000, as expected since the hop doubled). Applying the same stride-2 convolution formula: L_out = floor((1,500 + 2 - 3)/2) + 1 = floor(1,499/2) + 1 = 749 + 1 = 750 encoder vectors. Downstream consequence: self-attention cost inside the encoder scales as O(n²) in sequence length; going from 1,500 to 750 vectors reduces attention compute by a factor of (750/1500)² = 0.25, i.e. a 4× reduction — at the cost of coarser temporal resolution, which would blur short phonetic events and likely hurt transcription accuracy on fast speech.
A2. Offset for codebook 5 = 50,257 + 5 × 1,024 = 50,257 + 5,120 = 55,377. Token ID = 55,377 + 999 = 56,376. New total vocabulary size = 50,257 + 8 × 1,024 = 50,257 + 8,192 = 58,449.
A3. At the first seam (audio → text via ASR), everything except word identity is discarded: pitch, tempo, loudness, breathiness, pauses — the acoustic correlates of urgency, sarcasm, or hesitation are gone the instant the transcript is produced, because ASR is trained and evaluated purely on word-level accuracy. At the second seam (LLM text output → TTS audio), the synthesized voice reflects only the TTS model's own learned prosody defaults; it cannot mirror the caller's original tone because the TTS model never receives the caller's audio at all, only the LLM's text. A unified model that keeps audio (or discretized audio tokens carrying acoustic detail) inside the same sequence the "reasoning" component operates over never deletes that information before the point of decision — it may or may not use it well, but the cascade cannot use it at all, because by the second hand-off the information no longer exists anywhere in the pipeline's state.
A4. The unified model has no separate "must finish first" encoding pass: the same causal transformer that consumes incoming audio tokens also produces outgoing audio and text tokens, one token at a time, conditioned on everything so far — structurally identical to how a text LLM streams a response. Whisper's design requires its bidirectional self-attention encoder to process a fixed window of audio before cross-attention decoding of that window can happen, which is a natural fit for batch transcription of a complete utterance but not for a system that must start responding, or react to an interruption, mid-utterance.
A5. CTC handles the length mismatch by having the model emit one symbol-or-"blank" prediction per input audio frame, then summing the probability over every possible frame-to-symbol alignment (via dynamic programming) that collapses to the correct target sequence — it never picks a single alignment, it marginalizes over all of them during training. Whisper's cross-attention decoder instead sidesteps alignment entirely: it generates the output autoregressively, one text token at a time, and at each step its cross-attention mechanism learns to weight (attend to) whichever audio encoder positions are currently relevant — the "alignment" is an emergent, soft byproduct of attention weights rather than an explicit sum over frame-to-symbol paths.
A6. The residual passed from codebook k to codebook k+1 is the quantization error left over after codebook k's nearest-entry approximation — i.e., residual = original_vector − codebook_k_entry. Codebook k+1 is trained to quantize that leftover error, not the original vector, so each additional codebook targets exactly the part of the signal none of the previous codebooks could represent. This is why stacking codebooks improves reconstruction rather than adding redundancy: it is analogous to successive-approximation in positional number representation, where each additional digit resolves finer detail the previous digits could not, rather than repeating information already captured.
Think About It
Think about this: How would you explain audio-language models: speech and text integration to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.