Deep Q-Networks: Atari Game Playing

Atari's Breakout gives a player a paddle, a ball, and a wall of bricks. A DeepMind agent trained in 2015 on nothing but the raw screen pixels and the score counter played it well enough to average 401.2 points per episode, against 31.8 for a professional human games tester and 1.7 for an agent pressing buttons at random — roughly 13× the human score, reported as 1327.2% of human performance in the paper's own results table. The agent was never told the rules of Breakout: not that bricks disappear when hit, not that the ball bounces, not even that the paddle should chase the ball. It received a 210×160-pixel image with a 128-colour palette and a number, 60 times a second, and had to discover everything else itself, across 50 million frames of play — about 38 days of continuous game experience. This chapter builds the algorithm that did it: the Deep Q-Network, introduced by Mnih and colleagues at DeepMind and published in Nature in 2015 as "Human-level control through deep reinforcement learning."

Why a lookup table cannot play Breakout

You have already met Q-learning in tabular form: for every state s and action a, you maintain a table entry Q(s,a) estimating the total discounted future reward of taking action a in state s and acting optimally afterward. The Bellman optimality equation defines the target this table should converge to:

Q*(s,a) = E[ r + γ · max Q*(s',a') | s, a ]
                    a'

and the tabular update nudges each entry toward that target on every visit:

Q(s,a) ← Q(s,a) + α [ r + γ · max Q(s',a') − Q(s,a) ]
                              a'

This works when the state space is small enough to enumerate — a grid world, a card game, a few hundred board positions. Now ask what "state" means for Breakout. DeepMind's own preprocessing pipeline defines it as a raw Atari frame: 210×160 pixels, each pixel one of 128 colours. The number of distinct frames the console can display is 128 raised to the power of the pixel count:

210 × 160 = 33,600 pixels
number of possible raw frames = 128^33,600

That exponent alone dwarfs the number of atoms in the observable universe (about 10^80) by a margin so large that "astronomically large" undersells it — 128^33,600 is roughly 10^70,800. A single game of Breakout only ever visits a vanishing sliver of this space, but a table has no way to know which sliver matters: every state is a stranger until visited, and two frames that differ by one pixel — the ball nudged one row down — are unrelated entries with no shared structure. Even DQN's own preprocessed input, a stack of four 84×84 grayscale frames, still allows 256^28,224 distinct tensors. No table, and no amount of memory, holds that. What DQN does instead is replace the table with a parametric function: a convolutional neural network Q(s,a;θ) with a fixed, modest number of weights θ that must generalise across visually similar states it has never exactly seen before — a ball two pixels to the left produces almost the same output as the frame actually seen, because nearby pixels share the same convolutional filters. That generalisation, not brute enumeration, is the entire reason function approximation is necessary here.

The network: from pixels to four numbers

The architecture Mnih et al. settled on (after testing informally on Pong, Breakout, Seaquest, Space Invaders and Beam Rider, then freezing the same architecture and hyperparameters across all 49 games they evaluated) takes the 84×84×4 stacked-frame tensor as its only input and produces one Q-value per legal action in a single forward pass. Three convolutional layers extract spatial features, two fully connected layers combine them:

• Conv1: 32 filters, 8×8, stride 4, ReLU
• Conv2: 64 filters, 4×4, stride 2, ReLU
• Conv3: 64 filters, 3×3, stride 1, ReLU
• FC1: 512 units, ReLU
• Output: one linear unit per valid action (4 for Breakout: NOOP, FIRE, RIGHT, LEFT)

You can derive the spatial size after each convolution yourself with the standard formula, output = ⌊(W − K)/S⌋ + 1, where W is the input width, K the kernel size, and S the stride:

Conv1: floor((84 - 8)/4) + 1 = floor(76/4) + 1 = 19 + 1 = 20  → 20×20×32
Conv2: floor((20 - 4)/2) + 1 = floor(16/2) + 1 =  8 + 1 =  9  →  9× 9×64
Conv3: floor(( 9 - 3)/1) + 1 = floor( 6/1) + 1 =  6 + 1 =  7  →  7× 7×64

Flattening the final 7×7×64 volume gives 7×7×64 = 3,136 features feeding into FC1. You can now count exactly how many numbers the whole network has to learn, layer by layer (weights plus one bias per output unit):

Conv1: (8·8·4  + 1) × 32  = (256+1) × 32   =     8,224
Conv2: (4·4·32 + 1) × 64  = (512+1) × 64   =    32,832
Conv3: (3·3·64 + 1) × 64  = (576+1) × 64   =    36,928
FC1:   3,136 × 512 + 512               = 1,605,632 + 512 = 1,606,144
Out:   512 × 4 + 4                     =     2,048 + 4   =     2,052
                                                          -----------
                                             total params = 1,686,180

About 1.69 million numbers approximate a function whose domain has roughly 10^70,800 possible raw inputs. That gap is the whole point: the network is not memorising states, it is learning reusable visual features — edges, the ball, the paddle, the brick wall — that transfer across the astronomical number of frames it will never see during training. Notice too what the output layer buys you architecturally. A network could instead take (state, action) as a joint input and output a single scalar, but that forces one forward pass per candidate action to find the best one. Mnih et al. deliberately architected it the other way: state alone goes in, and a vector of Q-values for every legal action comes out of one pass, so choosing the greedy action is a single argmax over four numbers, not four separate network evaluations.

Why plugging a neural network into the Bellman update alone does not work

Tabular Q-learning is proven to converge under mild conditions. The moment you replace the table with a neural network trained by ordinary online stochastic gradient descent — observe a transition, compute the TD target, take one gradient step, discard the transition, repeat — it tends to diverge or oscillate wildly in practice. Three properties combine to cause this, sometimes called the deadly triad in reinforcement learning: function approximation (weights are shared across states, so an update to one state's estimate silently perturbs the estimates of many other states), bootstrapping (the target itself is built from the network's own current output, r + γ max Q(s',a';θ), rather than a ground-truth return), and off-policy learning (the greedy target policy differs from the ε-greedy behaviour policy generating the data). Two further practical problems compound this for Atari specifically: consecutive frames from one episode are highly correlated (the ball barely moves between frames), so successive gradient steps are far from the independent, identically distributed samples that stochastic gradient descent assumes; and small changes to θ immediately change the very targets y = r + γ max Q(s',a';θ) used to train θ, so the network is chasing a target that moves every time it moves — a feedback loop that can spiral into divergence rather than settle into convergence.

The two fixes that make it stable

Experience replay. Instead of training on each transition once, in the order it happens, DQN stores every transition (s, a, r, s') in a circular buffer holding the most recent 1,000,000 transitions, and at each training step samples a uniformly random minibatch of 32 from that buffer. This breaks the correlation between consecutive training samples (a minibatch mixes moments from many different points in many different episodes), lets each transition be reused many times instead of being seen once and discarded (far more data-efficient), and smooths out the wild swings that would occur if training data mirrored whatever narrow behaviour the current policy happens to prefer this week.

A separate, frozen target network. DQN keeps two copies of the network's weights: the online network θ, updated by gradient descent every 4 actions taken, and a target network θ⁻, used only to compute the TD target y = r + γ max_a' Q(s',a';θ⁻), and frozen between updates. Every C = 10,000 parameter updates, θ⁻ is overwritten with a fresh copy of θ. For those 10,000 updates the regression target does not move even though the online network's estimate of Q(s,a;θ) does, converting an unstable moving-target problem into a sequence of ordinary, well-defined supervised regression problems, one every 10,000 steps.

A third, smaller stabiliser is worth knowing because it is easy to miss in a casual reading: the paper clips the TD error δ = r + γ max Q(s',a';θ⁻) − Q(s,a;θ) to the range [−1, 1] before it drives the gradient. This is equivalent to using a squared-error loss only for small errors and a constant-slope (absolute-value) loss for large ones — what is now usually called a Huber loss — so one wildly wrong prediction early in training cannot produce a huge, destabilising gradient spike.

The full training loop

Putting the pieces together, one full training run proceeds like this. A 30-step random no-op period starts each episode (so the agent does not memorise a single fixed starting frame). Every game frame is compared pixel-by-pixel against the previous one and the per-pixel maximum kept, which removes flicker artefacts specific to the Atari 2600 hardware; the result is converted to a single luminance (Y) channel and resized to 84×84. Rather than acting on every frame, the agent commits to one action and repeats it for 4 consecutive frames (action repeat = 4), which quadruples the effective amount of game experience for the same compute budget. The last 4 post-processed frames are stacked to form the 84×84×4 input — this is what lets the network infer velocity: a single frame cannot tell you which way the ball is moving, four frames in sequence can. Rewards are clipped to {−1, 0, +1} regardless of the game's actual scoring scale, so that one algorithm and one learning rate work across games as different as Pong (rewards of ±1) and Video Pinball (rewards in the thousands).

Action selection is ε-greedy: with probability ε take a uniformly random action, otherwise take argmax_a Q(s,a;θ). ε is annealed linearly from 1.0 down to 0.1 over the first 1,000,000 frames and held at 0.1 afterward — heavy random exploration early, mostly-greedy exploitation later. The replay buffer is pre-filled with 50,000 transitions from a purely random policy before any gradient step is taken, so the very first minibatches are not degenerate. Training then proceeds for 50 million frames, with one minibatch gradient update via RMSProp (learning rate 0.00025) every 4 new actions, and the target network refreshed every 10,000 of those updates.

Architecture and training loop

Worked example: one gradient step, by hand

Take a single stored transition sampled from the replay buffer. Suppose the current online network, given state s (a stack of 4 frames mid-rally in Breakout), outputs these four Q-values — the same numbers used in the diagram above:

Q(s, NOOP;  θ) = 2.0
Q(s, FIRE;  θ) = 3.5   ← the action actually taken, a
Q(s, RIGHT; θ) = 1.2
Q(s, LEFT;  θ) = 0.8

The agent took action FIRE, the paddle hit a brick, and the emulator returned reward r = 1 (clipped) and a non-terminal next state s'. The frozen target network, evaluated on s', outputs:

Q(s', NOOP;  θ⁻) = 1.0
Q(s', FIRE;  θ⁻) = 2.0
Q(s', RIGHT; θ⁻) = 4.0   ← the maximum
Q(s', LEFT;  θ⁻) = 0.5

Step 1 — TD target. With γ = 0.99:

y = r + γ · max Q(s',a';θ⁻)
       a'
  = 1 + 0.99 × 4.0
  = 1 + 3.96
  = 4.96

Step 2 — TD error. This compares the target to the online network's estimate for the action actually taken, Q(s,a;θ) = Q(s,FIRE;θ) = 3.5:

δ = y − Q(s,a;θ) = 4.96 − 3.5 = 1.46

Step 3 — error clipping. |δ| = 1.46 exceeds 1, so the paper's error clipping caps the value that actually drives the gradient at ±1:

δ_clipped = clip(1.46, −1, 1) = 1.0

Notice what this buys you: a raw squared-error gradient would have been proportional to −2δ = −2.92, meaning this one lucky transition (target far exceeds current estimate) would push the weights twice as hard as a transition with δ = 1. After clipping, every transition with |δ| ≥ 1 contributes the same unit-magnitude push, regardless of whether the network was wrong by 1.5 or by 150 — exactly the property that keeps one outlier transition from blowing up the weights.

Step 4 — the update. Only the Q(s,FIRE;θ) output unit's incoming weights receive a nonzero gradient from this transition, since the other three action outputs never entered the loss (Q(s,NOOP;θ), Q(s,RIGHT;θ) and Q(s,LEFT;θ) are untouched by this sample). With learning rate α = 0.00025 (RMSProp's adaptive scaling aside), the weight update is:

θ ← θ + α · δ_clipped · ∇θ Q(s,FIRE;θ)
      = θ + 0.00025 × 1.0 × ∇θ Q(s,FIRE;θ)

θ⁻ is left completely untouched by this step — it only changes in one atomic copy operation every 10,000 updates, which is exactly what keeps the regression target y stable across the 4,999 updates that follow this one before the next sync.

Reading the code

The network below reproduces the exact architecture just derived — note that 7 * 7 * 64 in the flatten step is not a magic number, it is the 3,136 computed above — and the training step implements the same TD target and masked gradient from the worked example, using gather to pick out only the Q-value of the action actually taken:

import torch
import torch.nn as nn
import torch.nn.functional as F

class DQN(nn.Module):
    def __init__(self, num_actions):
        super().__init__()
        self.conv1 = nn.Conv2d(4, 32, kernel_size=8, stride=4)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=4, stride=2)
        self.conv3 = nn.Conv2d(64, 64, kernel_size=3, stride=1)
        self.fc1 = nn.Linear(7 * 7 * 64, 512)
        self.fc2 = nn.Linear(512, num_actions)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        return self.fc2(x)          # shape: (batch, num_actions)


def train_step(policy_net, target_net, optimizer, batch, gamma=0.99):
    states, actions, rewards, next_states, dones = batch

    # Q(s, a; theta) for the action actually taken
    q_sa = policy_net(states).gather(1, actions.unsqueeze(1)).squeeze(1)

    with torch.no_grad():
        # max_a' Q(s', a'; theta_minus), theta_minus frozen this whole call
        next_q = target_net(next_states).max(1)[0]
        targets = rewards + gamma * next_q * (1 - dones)

    td_error = targets - q_sa
    loss = F.smooth_l1_loss(q_sa, targets)   # matches the paper's [-1,1] error clip

    optimizer.zero_grad()
    loss.backward()                          # gradient flows into policy_net only
    optimizer.step()
    return loss.item()


def select_action(state, policy_net, epsilon, num_actions):
    import random
    if random.random() < epsilon:
        return random.randrange(num_actions)
    with torch.no_grad():
        q_values = policy_net(state)
        return int(q_values.argmax(dim=1).item())

Trace train_step against the worked example: policy_net(states) produces the four numbers [2.0, 3.5, 1.2, 0.8], gather with actions = [1] (FIRE is index 1) selects 3.5, exactly q_sa in the derivation. target_net(next_states).max(1)[0] takes the row [1.0, 2.0, 4.0, 0.5] and returns 4.0, exactly the maximum used in Step 1. smooth_l1_loss is PyTorch's name for the Huber loss — squared for small errors, linear (slope ±1) beyond a threshold of 1 — which is precisely the error-clipping behaviour computed by hand in Step 3.

The misconception to unlearn

The most common wrong mental model a student carries into this topic is picturing the Q-network the way the Bellman equation is written: as a function of a state-action pair, Q(s,a), so surely the network takes both s and a as input and outputs one number, and choosing the best action means running the network once per candidate action and comparing the outputs. This is a completely reasonable guess — it mirrors the mathematical notation directly — and it is not what DQN does. As the architecture trace above shows, the network takes only the state (the 84×84×4 stack) as input, and produces a vector of Q-values, one per legal action, in a single forward pass; action selection is then just an argmax over that vector, not four separate forward passes. Mnih et al. made this architectural choice deliberately: a joint (state, action) input would need one forward pass per action to pick the greedy move, which is wasteful when the number of actions is small and fixed, as it is for any single Atari game. If you find yourself writing pseudocode with a loop over actions calling the network inside the loop, that is the misconception resurfacing — the loop should not exist.

Active recall

Attempt each question before reading its answer.

1. Why can tabular Q-learning not be applied directly to raw Atari frames? Give the actual number, not just "it's too big."
2. An Atari game has 6 legal actions instead of Breakout's 4. Which layer of the DQN architecture changes, and how many parameters does that layer gain or lose compared to the 4-action version?
3. A classmate proposes training DQN with only the online network, no target network, updating θ after every single transition. What specifically goes wrong, mechanistically?
4. Given Q(s,a;θ) = 5.0, r = −1 (clipped), γ = 0.99, and max_a' Q(s',a';θ⁻) = 2.0, compute the TD target y, the raw TD error δ, and the clipped error used in the actual gradient step.
5. True or false: because experience replay samples uniformly at random, a transition from 900,000 steps ago is exactly as likely to be sampled on the next training step as a transition from 10 steps ago. Justify from the algorithm's definition.
6. Why does DQN stack 4 consecutive frames as input instead of feeding the network a single frame?

Answers.

1. A raw frame is 210×160 pixels from a 128-colour palette, so the number of distinct possible frames is 128^33,600 ≈ 10^70,800 — a number vastly larger than the number of atoms in the observable universe (≈10^80). Even after preprocessing to an 84×84×4 grayscale stack, the input space is 256^28,224 possible tensors. No table with one entry per state can be built or even addressed at that scale, and even if it could, almost every entry would never be visited even once across 50 million training frames, so it would never receive an update.
2. Only the output layer changes. With 6 actions instead of 4: weights = 512 × 6 + 6 = 3,072 + 6 = 3,078 parameters, versus 512 × 4 + 4 = 2,052 for Breakout — a gain of 1,026 parameters. Every earlier layer (all three convolutions and FC1) is completely unaffected, because those layers only ever see the state, never the action count; this is a direct consequence of the "state-in, vector-of-Q-values-out" architecture from the misconception section.
3. Without a target network, the regression target y = r + γ max Q(s',a';θ) is recomputed from the same θ that the gradient step is about to change. Every update simultaneously moves the prediction Q(s,a;θ) toward y and moves y itself (since y depends on θ through the max term), so the network is chasing a target that recoils away from it on every step. Combined with correlated, non-i.i.d. consecutive samples if replay is also removed, this is the textbook recipe for oscillation or divergence rather than convergence to Q*.
4. y = r + γ × max Q(s',a';θ⁻) = −1 + 0.99 × 2.0 = −1 + 1.98 = 0.98. Raw TD error δ = y − Q(s,a;θ) = 0.98 − 5.0 = −4.02. Since |−4.02| > 1, the clipped error used in the actual gradient is clip(−4.02, −1, 1) = −1.0 — the network is told only "push this estimate down," with a fixed-magnitude push, not "push it down by 4.02."
5. True by the algorithm's definition: the buffer samples uniformly at random from whatever transitions currently sit in the 1,000,000-slot circular buffer, with no recency weighting. The important subtlety is that "900,000 steps ago" only stays in the buffer while the buffer holds fewer than 1,000,000 transitions or that slot has not yet been overwritten; once the buffer is full, transitions older than 1,000,000 steps have already been evicted and have probability zero, not just low probability, of being sampled.
6. A single static frame shows the ball's position but carries no information about its velocity or direction — two frames could show an identical ball position with the ball moving up-left in one case and down-right in the other, and the network could not tell them apart from one frame alone. Stacking 4 consecutive frames lets the convolutional filters detect motion directly, the same way your visual system infers a ball's trajectory from a short sequence of glimpses rather than one still image.

Think About It

Think about this: How would you explain deep q-networks: atari game playing to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind deep q-networks: atari game playing, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.