Music Generation: Modeling Temporal Sequences

The raga's hidden rulebook

Sit with a trained Hindustani classical vocalist while she performs raga Yaman and you will notice something a beginner never does: certain note-to-note moves simply never happen. Yaman uses all seven swaras — Sa, Re, Ga, Ma (the sharp, tivra Ma, not the natural one), Pa, Dha, Ni — but the ascent (aroha) and descent (avaroha) follow a fixed grammar. A phrase built around Ni almost always resolves toward Re or curls back to Sa; it does not jump straight to Pa. Ask the singer why, and she will not recite a probability table — she has internalized the grammar through years of riyaz (practice). But if you recorded a thousand of her phrases and simply counted, for every note, which note followed it, you would recover almost exactly the rule she is obeying. That count-and-predict operation, done formally, is the oldest and simplest algorithm for generating music: a Markov chain over notes. This chapter builds that model from scratch, shows precisely where it fails, and then builds the model that Indian and international composers, and companies like Google Magenta and OpenAI's MuseNet, actually use to fix that failure: a recurrent neural network that predicts each note conditioned on everything that came before it — the same core mechanism your Grade 11 NLP unit used to predict the next word in a sentence, applied to notes instead of words.

This equivalence is the spine of the chapter. Music generation is not a separate discipline from language modeling — it is sequence modeling applied to a different alphabet. A sentence is a sequence of tokens drawn from a vocabulary of words; a melody is a sequence of tokens drawn from a vocabulary of notes (or note-duration pairs, or MIDI events). Everything you already know about conditional probability, next-token prediction, and autoregressive generation transfers directly. What changes is the representation of the data and the specific statistical structure of the domain — and that structure is what we now make precise.

Representing music as a sequence

Before any model can learn transitions, music has to become a sequence of discrete symbols. Three representations dominate practice, and the choice matters because it determines what a "timestep" even means:

• Symbolic note sequence. Each element of the sequence is a pitch name (or scale degree), one per melodic event, with duration and rests handled separately or ignored. This is what we use below — it is the cleanest starting point because timesteps line up one-to-one with notes.
• Piano roll. A 2-D grid, pitch on one axis and time (in fixed small increments, e.g. one grid column per sixteenth note) on the other, with a cell lit up when that pitch sounds during that time slice. A chord is represented by lighting up several pitches in the same column simultaneously — a multi-hot vector, not a one-hot vector, because more than one note can be active at once.
• Event sequence (MIDI-style). A stream of discrete events — NOTE_ON(pitch), NOTE_OFF(pitch), TIME_SHIFT(Δt) — which is what real generation systems (Magenta's Performance RNN, for instance) use, because it captures expressive timing without forcing every duration onto a fixed grid.

We will use the symbolic note sequence for worked examples, because every step of arithmetic stays checkable by hand. Everything generalizes to piano rolls and event streams; only the vocabulary and the encoding of each timestep change.

The first model: a Markov chain over notes

A first-order Markov chain assumes the next note depends only on the current note, not on anything before it: P(x_{t+1} | x_t, x_{t-1}, ..., x_1) = P(x_{t+1} | x_t). To fit one, count, for every note, how often each other note follows it, and normalize each row into a probability distribution.

Take a small, simplified teaching corpus of Yaman-style phrases (not a musicological transcription — a toy dataset built to keep the arithmetic exact and checkable):

corpus = [
    ["N", "R", "G"],
    ["R", "G", "M"],
    ["G", "M", "D"],
    ["M", "D", "N"],
    ["N", "R", "G"],
    ["G", "M", "D"],
    ["D", "N", "S"],
    ["R", "G", "M"],
]

Eight phrases of three notes each give sixteen consecutive note-pairs (bigrams) total. Build the transition table by counting every bigram, then normalizing:

from collections import defaultdict

def build_transition_matrix(corpus):
    counts = defaultdict(lambda: defaultdict(int))
    for phrase in corpus:
        for prev, nxt in zip(phrase, phrase[1:]):
            counts[prev][nxt] += 1
    probs = {}
    for prev, next_counts in counts.items():
        total = sum(next_counts.values())
        probs[prev] = {nxt: c / total for nxt, c in next_counts.items()}
    return probs

transition = build_transition_matrix(corpus)
print(transition["N"])

Trace it by hand. The bigram (N, R) occurs in phrase 1 and phrase 5, so counts["N"]["R"] = 2. The bigram (N, S) occurs once, inside phrase 7's second pair. So counts["N"] = {"R": 2, "S": 1}, total 3, giving P(R|N) = 2/3 and P(S|N) = 1/3. The printed line is:

{'R': 0.6666666666666666, 'S': 0.3333333333333333}

Every other source note in this corpus turns out to have exactly one observed successor (check it yourself: R is always followed by G, G always by M, M always by D, D always by N), so N is the only place the chain actually branches — which matches the musical intuition that Ni is a "decision point" in Yaman, either curling back into the R–G motif or resolving down to the tonic Sa.

Generation means walking this chain, drawing a random number at each branch and picking whichever outgoing edge it lands in:

def generate(transition, start, draws):
    seq = [start]
    for u in draws:
        current = seq[-1]
        cumulative = 0.0
        for nxt, p in transition[current].items():
            cumulative += p
            if u < cumulative:
                seq.append(nxt)
                break
    return seq

draws = [0.1, 0.1, 0.1, 0.5, 0.1]
print(generate(transition, "G", draws))

Trace step by step. Start at G. Draw 0.1: transition["G"] = {"M": 1.0}, cumulative reaches 1.0 immediately, 0.1 < 1.0, append M. Draw 0.1: transition["M"] = {"D": 1.0}, append D. Draw 0.1: transition["D"] = {"N": 1.0}, append N. Draw 0.5: now at the one branch point, transition["N"] = {"R": 0.667, "S": 0.333} in that insertion order; cumulative after R is 0.667, and 0.5 < 0.667, so append R. Draw 0.1: transition["R"] = {"G": 1.0}, append G. Final output:

['G', 'M', 'D', 'N', 'R', 'G']

Notice the chain loops back into the N → R → G motif it started near — an emergent repetition the model never explicitly encoded, it simply fell out of the statistics. Also notice that S never appears as a source key in transition at all — no phrase in the corpus begins its second pair from S. In graph terms, S is a sink (an absorbing state): reach it and the walk has nowhere defined to go next, which is musically apt, since Sa is the tonic where a phrase comes to rest.

Where the Markov chain breaks down

Common misconception, named and corrected: students often assume that to give a Markov chain a "longer memory" — so it can capture a whole recurring motif rather than just one previous note — you simply raise the order from 1 to 2 or 3, and the table grows by a small, roughly proportional amount. It does not. It grows exponentially, because the number of contexts you must condition on grows as (alphabet size)^order, and each context still needs its own full row of outgoing probabilities.

With a 7-symbol swara alphabet, a first-order chain has 7 possible source contexts (one per note), each needing up to 7 entries: at most 7 × 7 = 49 table entries. Raise the order to 2 — condition on the previous two notes — and the number of distinct contexts is 7² = 49, each still needing up to 7 outgoing entries: 49 × 7 = 343 entries, a sevenfold increase, not a doubling. Order 3 pushes it to 7³ × 7 = 2,401.

This is exactly the combinatorial explosion you have already met in analyzing brute-force algorithms: fixing a window and enumerating every possible window content scales exponentially with window length. It is also why a raw high-order Markov table is a poor way to capture long musical structure (a phrase repeating 20 notes later, a return to the tonic 8 bars on) — you would need an astronomically large table, and most contexts in it would be seen zero or one times in any real training corpus, giving useless, overfit probability estimates. The fix is not a bigger table. It is a model whose memory does not require explicitly enumerating every possible context: a recurrent neural network, which compresses an arbitrarily long history into a single fixed-size vector instead of a table indexed by that history.

Recurrent networks: compressing history into a hidden state

An RNN music model keeps a hidden state vector h_t that is updated at every timestep from the current input note and the previous hidden state, then used to predict the next note:

h_t = tanh(W_x · x_t + W_h · h_{t-1} + b)
logits_t = V · h_t + c
P(next note) = softmax(logits_t)

This is the identical recurrence you used for next-word prediction in your NLP unit, with word tokens swapped for note tokens. h_t never grows in size no matter how long the piece runs — it is a fixed-length compressed summary of the entire prefix, which is precisely what a fixed-order Markov table cannot offer: an order-2 chain forgets everything before the last 2 notes by construction, while an RNN's hidden state can in principle carry information from far earlier, without needing a combinatorially large table to do it.

Work through one tiny, fully specified instance by hand. Vocabulary {G, M, D}, one-hot encoded: x_G = [1,0,0], x_M = [0,1,0], x_D = [0,0,1]. Scalar hidden state (dimension 1, purely for hand-tractability — real models use dozens to thousands of dimensions). Fixed weights: W_x = [0.8, -0.5, 0.2], W_h = 0.6, b = 0.1, output weights V = [1.0, 0.5, -1.0] mapping the scalar hidden state to one logit per vocabulary symbol. Start h_0 = 0 and feed the sequence G, M:

Step 1 (input G): W_x · x_G = 0.8(1) + (-0.5)(0) + 0.2(0) = 0.8. z_1 = 0.8 + 0.6(0) + 0.1 = 0.9. h_1 = tanh(0.9) ≈ 0.7163.

Step 2 (input M): W_x · x_M = -0.5. z_2 = -0.5 + 0.6(0.7163) + 0.1 = -0.5 + 0.4298 + 0.1 = 0.0298. h_2 = tanh(0.0298) ≈ 0.0298 (for arguments this small, tanh(x) ≈ x, since the cubic correction term x³/3 is under 10⁻⁵).

Now produce the prediction for what comes after G, M. Logits: logit_G = 1.0 × 0.0298 = 0.0298, logit_M = 0.5 × 0.0298 = 0.0149, logit_D = -1.0 × 0.0298 = -0.0298. Exponentiate: e^0.0298 ≈ 1.0302, e^0.0149 ≈ 1.0150, e^{-0.0298} ≈ 0.9707, summing to 3.0159. Dividing through gives P(G) ≈ 0.342, P(M) ≈ 0.337, P(D) ≈ 0.322 — a distribution close to uniform, which is exactly what you should expect: with only these two small-weight steps behind it, the network has not yet accumulated a strong preference. You can check this trace numerically:

import numpy as np

Wx = np.array([0.8, -0.5, 0.2])
Wh = 0.6
b = 0.1
V = np.array([1.0, 0.5, -1.0])

x_G = np.array([1, 0, 0])
x_M = np.array([0, 1, 0])

h0 = 0.0
h1 = np.tanh(Wx @ x_G + Wh * h0 + b)
h2 = np.tanh(Wx @ x_M + Wh * h1 + b)

logits = V * h2
probs = np.exp(logits) / np.sum(np.exp(logits))

print(round(h1, 4), round(h2, 4))
print(np.round(probs, 4))

This prints 0.7163 0.0298 and [0.3416 0.3366 0.3219], matching the hand derivation. To generate, sample the next note from this distribution (say D), feed x_D back in as the next input, compute h_3 from h_2, and repeat — this feedback loop, sample-then-feed-back, is what "autoregressive" means, and it is the only structural difference between training (where the true next note is always fed in, called teacher forcing) and generation (where the model's own sampled note is fed back in).

The generation loop, end to end

Sampling: why the "obviously correct" choice makes boring music

Given a predicted distribution like P(G)=0.342, P(M)=0.337, P(D)=0.322, the tempting move is greedy decoding: always emit the highest-probability note. Do this consistently and generated melodies degenerate into short repeating loops — because once the model emits its single most-likely note, the new context is only slightly different, so the next most-likely note is very often the same choice as before, and the walk falls into a fixed point or a short cycle. This is the standard failure mode of greedy decoding in any autoregressive model, text or music alike.

The fix is temperature sampling: divide the logits by a temperature T before the softmax, P_i ∝ exp(logit_i / T). At T = 1 you recover the model's raw distribution. As T → 0, the distribution sharpens toward greedy (all probability mass collapses onto the max), reproducing the repetitive-loop problem. As T grows past 1, the distribution flattens toward uniform, and generation becomes noisier and less faithful to what the model actually learned. In practice, generation systems sample stochastically at a moderate temperature (commonly T ≈ 0.8–1.2) specifically to avoid the greedy trap while staying musically coherent — the randomness in the walk is not a flaw to be minimized, it is the mechanism that produces variation at all.

Beyond plain RNNs

The plain RNN above has a real weakness that the numbers already hint at: h_2 ≈ 0.03 is much closer to 0 than h_1 ≈ 0.72 was, because the recurrent weight W_h = 0.6 shrank the previous state's contribution before it was added to the new input's contribution. Chain this shrinkage across many timesteps — as backpropagation-through-time must, to train the network on long sequences — and gradient signal from far back in a piece decays roughly geometrically, the vanishing gradient problem. This is precisely why production-grade music and text generators do not use the plain recurrence written above; they use LSTM or GRU cells, which add learned gates that can hold a memory value nearly unchanged across many steps instead of always shrinking it, and increasingly, attention-based Transformers, which skip the recurrence entirely and let every timestep look directly at every previous timestep's representation. The transition-matrix-versus-hidden-state distinction you learned here — explicit table versus compressed state — is the same distinction that separates n-gram language models from RNN language models from Transformers, one layer up.

Active recall

Attempt every question before reading its answer.

1. Using the transition table built from the corpus above, what is P(S | N), and from which raw counts is it derived?
2. If you upgrade the Markov chain from order 1 to order 2 over the same 7-symbol swara alphabet, how many (context → next-note) entries can the transition table need at most, and how many times larger is that than the order-1 table?
3. Starting at note N, with a single random draw u = 0.9, which note does the generate function above produce next, and why?
4. In the worked RNN trace, recompute h_2 if W_h were 0 instead of 0.6. What does this do to the model conceptually, and what numeric value results?
5. Why can't a plain one-hot-per-timestep note encoding represent a chord, and what change to the representation fixes this?
6. An order-2 Markov chain and a plain RNN can both, in principle, be influenced by something that happened many notes ago. Why does the RNN typically still fail to use information from 50 steps back, even though its architecture does not hard-limit its memory the way the Markov chain's fixed order does?

Answers.

1. counts["N"] = {"R": 2, "S": 1}, total 3 (from bigrams (N,R) in phrases 1 and 5, and (N,S) in phrase 7). P(S|N) = 1/3 ≈ 0.333.

2. Order-2 has 7² = 49 distinct two-note contexts, each needing up to 7 outgoing entries: 49 × 7 = 343 entries, versus 49 for order-1 — a sevenfold increase, not a doubling, because the entry count scales as 7^k × 7, exponential in the order k.

3. Still S. transition["N"] = {"R": 0.667, "S": 0.333} in that insertion order; cumulative after R is 0.667, and 0.9 ≥ 0.667, so the walk continues to S's bucket; cumulative reaches 1.0 there and 0.9 < 1.0, so S is selected.

4. With W_h = 0: z_2 = -0.5 + 0(0.7163) + 0.1 = -0.4, so h_2 = tanh(-0.4) ≈ -0.3799. Conceptually, zeroing W_h severs the recurrence entirely — the hidden state no longer carries any information from h_1, so the model degenerates into one that predicts the next note from only the current note, discarding all earlier history. It is strictly weaker than even a first-order Markov chain, since a Markov chain's fixed lookup table is at least being fit to real transition statistics, while this severed RNN's per-step output would depend only on the current symbol through fixed random weights, not on any learned conditional structure.

5. A one-hot vector has exactly one nonzero entry by construction, encoding "exactly one active symbol." A chord needs several pitches active in the same timestep, which requires a multi-hot vector (several 1s in one vector) — the standard piano-roll representation, where a time-slice column can have multiple pitch rows lit simultaneously, or an event stream with multiple simultaneous NOTE_ON events sharing a timestamp.

6. Backpropagation-through-time multiplies gradient contributions by the recurrent weight and the derivative of tanh at every step going backward; both factors are typically well below 1 in magnitude, so the product shrinks roughly geometrically with distance. After enough steps the gradient from something 50 timesteps back is numerically negligible, so the network is never effectively trained to use it — the architecture is theoretically unbounded in memory, but the optimization procedure that fits its weights is not, which is exactly the motivation for LSTM/GRU gating.

Think About It

Think about this: How would you explain music generation: modeling temporal sequences to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.