A food-delivery app running a monsoon-season ad push needs background music for eleven thousand short reels going out across cities and languages in a single week — one per hyperlocal micro-campaign, each fifteen seconds long, none of them allowed to repeat because repetition reads as cheap. Licensing eleven thousand distinct human-composed tracks is not a budget problem, it is a supply problem: no catalogue is that deep, and no composer roster can turn around that volume on that schedule. The obvious fix is to have a model generate the music. The question this chapter actually answers is the one that decides whether that fix works at all: what, mechanically, does it mean for a neural network to "generate" fifteen seconds of audio, and why does the most literal way of doing it — predicting the sound one tiny slice of air pressure at a time — turn out to be one of the most computationally punishing sequence-generation problems in all of machine learning. By the end you will be able to compute exactly how punishing, in samples and in milliseconds, and see precisely why production systems refuse to do it the literal way.
Digital audio, from first principles
A microphone converts a continuously varying air-pressure wave into a continuously varying voltage. A computer cannot store a continuous signal, so an analog-to-digital converter measures that voltage at fixed intervals and writes down a number each time — this is sampling. The rate at which it measures is the sample rate, in hertz (samples per second); a compact-disc-quality recording uses 44,100 Hz, meaning 44,100 numbers are stored for every second of sound. Each stored number is itself rounded to the nearest representable value out of a fixed set — 16-bit audio allows 2¹⁶ = 65,536 distinct amplitude levels — which is quantization. A digital audio clip is therefore nothing more than a long array of integers (or floats, once normalized), and every synthesis or generation technique in this chapter is, underneath, a technique for producing that array.
Sampling at a finite rate throws away information, and the Nyquist–Shannon sampling theorem states precisely how much: a sample rate of fs can faithfully represent frequencies only up to fs/2, called the Nyquist frequency. Feed a sampler a tone above that limit and it does not disappear — it reappears, disguised, at a lower frequency, an effect called aliasing. If a true frequency f lies between the Nyquist frequency and the full sample rate, the alias that appears is fs − f. This is not a rounding curiosity; it is the reason every practical synthesis routine has to know its sample rate before it decides how many harmonics it is safe to generate, which is exactly where the first worked example is headed.
Building a sound from scratch: additive synthesis
Long before neural networks, sound designers built tones the way Fourier's theorem says any periodic sound can be built: as a sum of pure sine waves at integer multiples of a fundamental frequency, called harmonics, each with its own amplitude. This is additive synthesis, and it is worth tracing in full because every later technique in this chapter is still, at bottom, producing the same kind of array — just with the harmonic weights chosen by a trained network instead of hand-picked.
import numpy as np
def synthesize_note(freq_hz, duration_s, sample_rate=44100, num_harmonics=5):
t = np.linspace(0, duration_s, int(sample_rate * duration_s), endpoint=False)
wave = np.zeros_like(t)
for n in range(1, num_harmonics + 1):
amplitude = 1.0 / n # each harmonic quieter than the last
wave += amplitude * np.sin(2 * np.pi * n * freq_hz * t)
wave = wave / np.max(np.abs(wave)) # normalize to [-1, 1]
return wave, t
wave, t = synthesize_note(freq_hz=440.0, duration_s=0.01) # A4, a 10 ms snippet
print(wave.shape)
print(round(wave[0], 4), round(t[1] - t[0], 8))Trace it line by line. duration_s=0.01 and sample_rate=44100 give int(44100 * 0.01) = 441 sample points, so t and wave are both arrays of shape (441,). The loop runs for n = 1..5, adding a sine at 440 Hz, 880 Hz, 1320 Hz, 1760 Hz, and 2200 Hz, each with amplitude 1/n — this is exactly the harmonic-amplitude falloff of a sawtooth wave's magnitude spectrum, so the note will sound bright and buzzy rather than pure. The highest harmonic, 2200 Hz, sits comfortably under the Nyquist frequency of 44100/2 = 22050 Hz, so nothing aliases. At t[0] = 0, every sin(2π·n·f·0) term is sin(0) = 0, so wave[0] is exactly 0.0 before normalization and stays 0.0 after it (0 divided by any nonzero maximum is still 0). The sample spacing t[1] - t[0] is just 1/sample_rate = 1/44100 ≈ 0.0000226757, which Python prints in scientific notation once rounded to eight places. The program therefore prints exactly:
(441,)
0.0 2.268e-05Additive synthesis is transparent and controllable, but it does not scale to realistic timbres — a real violin note has dozens of shifting, interacting harmonics plus noise from bow friction, and hand-tuning those by ear is a career, not a function call. That gap is exactly what learned models close: instead of a human choosing five harmonic weights, a network learns, from thousands of recordings, what weights (or, more generally, what waveform) a given instrument produces for a given pitch, dynamic, and articulation.
Two separate jobs: what to play versus how it sounds
Here is a mix-up worth naming and correcting directly, because it causes students to misread every music-AI system diagram they will ever see afterward: generating a MIDI file — a sequence of note-on and note-off events, pitches, velocities, timings — is not the same job as generating audio, and finishing the first does not mean you have produced a "song" you can play through a speaker. MIDI (and its neural equivalents, like a transformer trained on tokenized note sequences) answers the question what should be played: which pitch, how loud, how long, on which instrument. That symbolic description still has to pass through a second, entirely different system — a synthesizer, a sample library, or in modern pipelines a learned vocoder — that answers what should this actually sound like, turning abstract note events into the literal sequence of pressure-wave samples a speaker can reproduce. Confusing the two leads students to expect a "generate music" model to be one monolithic black box; in practice almost every serious system, symbolic or neural, is a pipeline with a distinct composition stage and a distinct acoustic-rendering stage, and knowing which stage a given model occupies is the first thing to check when reading its architecture diagram.
Predicting audio one sample at a time: the WaveNet approach
The most literal way to generate raw audio with a neural network is autoregressive, the same principle used for text: predict the next sample given all the previous ones, then feed that prediction back in and predict the one after it. WaveNet (van den Oord et al., 2016) showed this works, using stacks of dilated causal convolutions. "Causal" means a layer's output at time step t depends only on inputs at time t and earlier, never the future — essential for a model that has to generate sample t before sample t+1 even exists. "Dilated" means the convolution's filter taps skip over input positions at a fixed stride instead of reading every consecutive sample: a layer with dilation d and kernel size k reads inputs spaced d apart. Stacking layers with exponentially growing dilation — 1, 2, 4, 8, … — lets the receptive field (the span of raw input a single output depends on) grow exponentially with depth instead of linearly, which is the only reason this architecture is tractable at audio sample rates at all.
The diagram below traces exactly one output prediction back through four such layers (dilations 1, 2, 4, 8, kernel size 2) to see precisely which input samples it depends on.
Every one of the sixteen input nodes on the bottom row is highlighted, because with kernel size k=2 and dilations 1, 2, 4, 8 the receptive-field formula gives RF = (k−1)·(1+2+4+8) + 1 = 1·15 + 1 = 16. Follow the highlighted path yourself: the output depends on exactly one layer-4 position, {15}; that position depends on two layer-3 positions (dilation 8), giving {15, 7}; each of those depends on two layer-2 positions (dilation 4), giving {15, 11, 7, 3}; each of those depends on two layer-1 positions (dilation 2), giving {15, 13, 11, 9, 7, 5, 3, 1}; and each of those finally depends on two consecutive input positions (dilation 1), which between them cover every one of the sixteen input indices 0 through 15. That is the entire point of the exponential dilation schedule: four layers, each looking at only two inputs, together see sixteen raw samples — a receptive field that would need sixteen ordinary (non-dilated) layers of kernel size 2 to reach the same span linearly.
Scaling the stack — and why raw-sample autoregression still does not scale
Real WaveNet-style models repeat the full dilation schedule (1, 2, 4, …, up to some maximum) several times as separate stacked blocks, because one pass of 1-2-4-8-…-512 already reaches a wide span, and repeating it multiplies that span further without the parameter cost of ever-larger dilations. The code below computes this for a hypothetical stack — ten layers per block, dilations 1 through 512, repeated three times (thirty layers total) — using the same formula as above.
def receptive_field(kernel_size, dilations):
return (kernel_size - 1) * sum(dilations) + 1
dilations_per_stack = [2 ** i for i in range(10)] # 1, 2, 4, ..., 512
num_stacks = 3
all_dilations = dilations_per_stack * num_stacks # 30 layers total
rf_single_stack = receptive_field(2, dilations_per_stack)
rf_full_model = receptive_field(2, all_dilations)
print(rf_single_stack, rf_full_model)Trace it: dilations_per_stack sums to 1+2+4+...+512 = 1023, so rf_single_stack = 1·1023 + 1 = 1024. Repeating the list three times (Python's list-multiplication operator concatenates it three times, it does not multiply the values) gives all_dilations a sum of 3·1023 = 3069, so rf_full_model = 1·3069 + 1 = 3070. The program prints exactly 1024 3070. At a 16,000 Hz sample rate — typical for a speech- or music-focused raw-audio model — 3070 samples span 3070/16000 ≈ 0.192 seconds, or about 192 milliseconds, of context: enough to hear a couple of note onsets, not enough to hear a musical phrase.
That context window is what an autoregressive raw-audio model looks backward at. What it has to do forward is the real problem: to produce 15 seconds of audio at 44,100 Hz, it must predict 15 × 44,100 = 661,500 individual samples, and because each new sample depends on the ones already generated, those predictions are strictly sequential — 661,500 forward passes through the network, one after another, no parallelism across time during generation. Scale that to the eleven-thousand-reel week from the opening scenario and the naive approach needs on the order of 7 billion sequential sample predictions. This is precisely why production-grade audio generation systems do not generate raw waveform samples autoregressively at the final sample rate. Instead they compress audio first, with a learned neural audio codec (the same family of idea as SoundStream or EnCodec), into a much shorter sequence of discrete tokens — typically on the order of 50 to 75 tokens per second, regardless of the original 44,100 Hz sample rate — and then run an autoregressive transformer, the same kind of architecture used for text, over that token sequence, exactly the way Grade 11's introduction to NLP treats a sentence as a sequence of discrete tokens rather than a sequence of individual letters. A separate decoder network then expands the generated tokens back into a full-rate waveform. Systems like OpenAI's Jukebox (Dhariwal et al., 2020) and Google's MusicLM (Agostinelli et al., 2023) both follow this shape: compress audio into tokens, model the token sequence the way a language model models text, decode back to sound. The 15-second reel that would need 661,500 sequential raw-sample predictions needs only on the order of 750–1,125 sequential token predictions in this scheme — roughly three orders of magnitude fewer sequential steps, which is the difference between a system that ships eleven thousand tracks in a week and one that cannot.
Active recall
Attempt every question before reading its answer.
1. A synthesizer sums sine waves at 440, 880, 1320, 1760, and 2200 Hz (five harmonics of A4) at a sample rate of 8000 Hz. Does this alias? What if a sixth harmonic at 2640 Hz is added? What if a tenth harmonic at 4400 Hz is added instead?
2. Using RF = (k−1)·Σ(dilations) + 1, compute the receptive field for kernel size 3 with a single stack of dilations {1, 2, 4, 8}.
3. In the worked WaveNet-style example (kernel 2, three stacks of dilations 1..512, RF = 3070 samples ≈ 192 ms at 16,000 Hz), the sample rate is changed to 44,100 Hz while the architecture — thirty layers, same dilation pattern — stays exactly the same. Trace the full effect: does the receptive field change in samples? In milliseconds? How many stacks would be needed to restore roughly 192 ms of context at the new sample rate?
4. A model outputs a MIDI file with correct pitches, velocities, and timings for a full song. Has it produced playable audio? Why does the answer matter for how you'd design a music-generation pipeline?
5. Why do production-scale music-generation systems avoid modeling raw 44,100 Hz samples autoregressively? Give the argument in terms of the number of sequential prediction steps, using the eleven-thousand-reel scenario from the opening.
6. In synthesize_note, suppose amplitude = 1.0 / n is changed to amplitude = 1.0 / n**2. Will wave[0] still equal 0.0? How does the timbre change?
Answers.
1. At fs=8000 Hz, the Nyquist frequency is 4000 Hz. The original five harmonics (up to 2200 Hz) are all below 4000 Hz, so no aliasing occurs. A sixth harmonic at 2640 Hz is also below 4000 Hz — still safe. A tenth harmonic at 4400 Hz, however, exceeds the Nyquist frequency (4000 Hz) while staying below the sample rate itself (8000 Hz), so it aliases; its folded frequency is fs − f = 8000 − 4400 = 3600 Hz, which would be audible as a spurious 3600 Hz tone the synthesizer never intended to produce.
2. Σ(dilations) = 1+2+4+8 = 15. RF = (3−1)·15 + 1 = 30 + 1 = 31 samples.
3. The receptive-field formula depends only on kernel size and the dilation values, not on sample rate, so RF in samples is unchanged at 3070. But 3070 samples now spans 3070/44100 ≈ 0.0696 s ≈ 69.6 ms — far less than 192 ms, because each sample now represents a shorter slice of time. To recover roughly 192 ms at 44,100 Hz needs RF ≈ 0.192 × 44100 ≈ 8467 samples. Using RF = 1024 + (S−1)·1023 for S stacks (the first stack contributes 1024, each additional stack adds 1023 more), solving 1024 + (S−1)·1023 ≥ 8467 gives S−1 ≥ 7.28, so S ≥ 8.28, meaning at least 9 stacks are needed — three times the original stack count for roughly three times the sample rate, which is exactly why higher-fidelity raw-audio generation costs substantially more compute for the same amount of musical context.
4. No — a MIDI file is a symbolic description of what to play (pitch, velocity, timing), not an audio waveform. It still needs to pass through a synthesizer, sample library, or vocoder to become sound a speaker can reproduce. This matters for pipeline design because it means "generate the composition" and "generate the sound" are separable stages that can use entirely different techniques (for instance, a rule-based or transformer-based composer paired with a completely separate neural or sample-based renderer), rather than one model expected to do both jobs at once.
5. Fifteen seconds of audio at 44,100 Hz is 661,500 samples, and because raw-sample autoregression must generate each sample from the ones before it, that means 661,500 strictly sequential prediction steps per reel. Across eleven thousand reels that is roughly 7 billion sequential steps, which is computationally infeasible on any useful timeline. Compressing to a token sequence at roughly 50–75 tokens per second reduces a 15-second clip to on the order of 750–1,125 sequential steps — about three orders of magnitude fewer — which is what makes generation at this scale possible at all.
6. Yes, wave[0] is still exactly 0.0: every term in the sum is amplitude * sin(2π·n·f·0) = amplitude * sin(0) = amplitude * 0 = 0 regardless of what the amplitude weighting is, so the sum at t=0 is unaffected by the change. The timbre does change: 1/n² decays much faster than 1/n as harmonic number n increases, so more of the total energy stays concentrated in the fundamental and the higher harmonics contribute far less — the note sounds smoother and duller rather than bright and buzzy.
Key Takeaways — Summary and Recap
Let us recap what we covered: the core ideas behind music generation and audio synthesis with ai, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.