AlphaGo and AlphaFold: RL Success Stories

A Captain's Field Placement and a Go Stone

A T20 captain sets a field not for the ball that was just bowled but for the ball that has not been bowled yet. The state she is reasoning over is everything relevant right now — overs remaining, wickets in hand, required run rate, which batter is on strike, pitch wear — and the action she picks (third man in, extra cover out, spin over pace) changes that state for the next ball. Nothing about a single field placement is scored on its own. The only reward that exists is delivered at the end of the innings: win or lose. Everything in between is credit assignment across dozens of small decisions whose value can only be judged in hindsight, against an outcome that arrives many steps later. That structure — states, actions that change the state, and a reward that shows up only at the end — is exactly a Markov Decision Process (MDP), the mathematical object reinforcement learning is built to solve. DeepMind formalized the ancient board game Go in precisely this way, and in March 2016 the resulting system, AlphaGo, beat Lee Sedol, a 9-dan professional and one of the strongest Go players in history, four games to one. Four years later a differently built system from the same lab, AlphaFold, produced protein structure predictions accurate enough to be mistaken for laboratory measurements. The two are constantly mentioned in the same breath as "DeepMind's RL breakthroughs." Only one of them is a reinforcement learning success story. Getting clear on why is the point of this chapter.

Formalizing Go as a Markov Decision Process

Go is played on a 19×19 grid. Two players alternately place stones of their color; a stone or group is captured and removed when it has no empty adjacent point (no "liberty"); the player who controls more territory at the end wins. Casting this as an MDP: the state sₜ is the full board position at turn t; the action aₜ is a legal placement (or pass); the transition is deterministic — given a state and a legal action, the rules of Go fix the next state exactly, including any captures; and the reward is zero at every intermediate turn and only becomes ±1 at the terminal state, for a win or a loss. There is no partial credit for a stone that "looks good." An RL agent's job is to learn a policy π(a | s), a probability distribution over legal moves given a board, that maximizes the expected value of that terminal reward. The reason this is hard is the size of the state space. John Tromp and Gunnar Farnebäck's exact count of legal Go positions is on the order of 2.1 × 10¹⁷⁰ — a number vastly larger than the roughly 10⁴⁴–10⁴⁷ legal positions in chess, and larger than the number of atoms in the observable universe by many, many orders of magnitude. The DeepMind team's own estimate of the search space, from the 2016 Nature paper, put the number of possible move sequences at roughly 250¹⁵⁰ — an average branching factor of about 250 legal moves per turn, over a game that runs roughly 150 moves deep. Chess, by comparison, has a branching factor near 35. Brute-force search — the approach that let IBM's Deep Blue beat Garry Kasparov in chess in 1997 by evaluating on the order of tens of billions of positions per move (roughly 200 million positions per second over a multi-minute move budget) — is computationally hopeless here. AlphaGo's answer was to replace exhaustive search with learned intuition: two neural networks that narrow the search dramatically before any tree search happens at all.

Two Networks, Two Training Regimes

AlphaGo used two separate convolutional networks, trained in three stages. Stage 1 — supervised warm start. A policy network pσ was first trained by supervised learning on roughly 30 million board positions taken from the KGS Go Server's archive of games between strong human amateurs. Given a board, the network was trained to predict the move the human actually played next — an ordinary classification problem, no reward signal involved yet. This network alone predicted human expert moves correctly about 57% of the time and, played against itself, was already a competent amateur player. Its purpose was not to be the final policy; it was to give reinforcement learning a sane starting point instead of forcing self-play to begin from random noise. Stage 2 — self-play policy gradient. The network was then copied to pρ and improved by playing games against randomly sampled earlier versions of itself (sampling past checkpoints, not only the latest one, specifically to avoid overfitting to a single fixed opponent — a classic instability in self-play RL). Each finished game produced a trajectory of state-action pairs and a terminal reward z ∈ {+1, −1}. The network's weights were then nudged using the policy-gradient update ∇log π(a|s) · z, averaged over every move in the game: moves made in games that were won have their probability increased; moves in games that were lost have their probability decreased. This is REINFORCE, the most direct application of the policy-gradient theorem, and it is the actual reinforcement-learning step in the whole pipeline — the only place where a reward, rather than a labeled example, drives a weight update. Stage 3 — value network by regression. A separate value network vθ(s) was trained to predict, from a board position alone, the probability that the current player eventually wins. Training data came from self-play games generated by the RL policy network, with one crucial precaution: only a single position was sampled from each self-play game before discarding the rest, specifically because successive positions within one game are highly correlated and using many of them would badly overfit the value network to memorized games rather than general evaluation. The value network is trained by simple regression — minimize the squared error between its prediction and the actual outcome z of that game — so on its own it is supervised learning, not RL; it only becomes useful for reinforcement learning because the labels it regresses onto were generated by a reward-driven self-play process.

Monte Carlo Tree Search: Where Planning Meets Learning

Neither network plays Go by itself in the final system. AlphaGo combines both with Monte Carlo Tree Search (MCTS), a planning algorithm that runs thousands of simulated playouts from the current position before committing to a real move. Each simulation has four phases: select a path down the tree using a formula that balances exploitation of moves known to be good against exploration of moves that are promising but under-tried; expand the tree by one new node when a simulation reaches an unvisited position; evaluate that new leaf, in AlphaGo's case by blending the value network's prediction with a fast random rollout to the end of the game (value = 0.5·v_θ(leaf) + 0.5·(rollout outcome)); and backup that evaluation up the path, updating a running visit count N and mean action-value Q at every node it passed through. The selection step uses a formula called PUCT (Predictor + Upper Confidence bound applied to Trees):

a* = argmax_a [ Q(s,a) + c_puct · P(s,a) · sqrt(ΣN(s,b)) / (1 + N(s,a)) ]

Move   Prior P   Visits N   Mean value Q
A      0.5       3          0.4
B      0.3       1          0.6
C      0.2       0          0.0   (unvisited)

With c_puct = 1.5, total visits at the parent ΣN = 3 + 1 + 0 = 4, and sqrt(4) = 2:

PUCT(A) = 0.4 + 1.5 · 0.5 · 2 / (1+3) = 0.4 + 1.5/4        = 0.775
PUCT(B) = 0.6 + 1.5 · 0.3 · 2 / (1+1) = 0.6 + 0.9/2         = 1.050
PUCT(C) = 0.0 + 1.5 · 0.2 · 2 / (1+0) = 0.0 + 0.6/1         = 0.600

Move B wins, at 1.05, even though A has three times as many visits — its higher exploitation value (Q = 0.6 vs 0.4) outweighs A's exploration bonus. Move C, despite having the lowest prior and zero exploitation evidence, still edges out being ignored entirely, because an unvisited node with any non-trivial prior gets a guaranteed nonzero exploration bonus — this is precisely the mechanism that keeps MCTS from tunnel-visioning onto the policy network's single favorite move. The same computation, traced in code:

import math

def puct_score(Q, P, N, N_total, c_puct=1.5):
    return Q + c_puct * P * math.sqrt(N_total) / (1 + N)

children = {
    "A": {"P": 0.5, "N": 3, "Q": 0.4},
    "B": {"P": 0.3, "N": 1, "Q": 0.6},
    "C": {"P": 0.2, "N": 0, "Q": 0.0},
}
N_total = sum(c["N"] for c in children.values())  # 4

for name, c in children.items():
    score = puct_score(c["Q"], c["P"], c["N"], N_total)
    print(f"{name}: PUCT = {score:.3f}")

best = max(children, key=lambda a: puct_score(
    children[a]["Q"], children[a]["P"], children[a]["N"], N_total))
print("MCTS descends into:", best)

Tracing it: N_total sums to 4, so sqrt(N_total) = 2.0 exactly. For A, the exploration term is 1.5 · 0.5 · 2.0 / 4 = 0.375, giving 0.775. For B it is 1.5 · 0.3 · 2.0 / 2 = 0.45, giving 1.050. For C it is 1.5 · 0.2 · 2.0 / 1 = 0.6, giving 0.600. Formatted to three decimal places the floating-point arithmetic prints exactly:

A: PUCT = 0.775
B: PUCT = 1.050
C: PUCT = 0.600
MCTS descends into: B

After thousands of such simulations, the visit counts N(s,a) at the root themselves become the improved policy — a move that received more simulated visits is one the tree search, informed by both networks, judged stronger than the raw policy network's prior alone suggested. AlphaGo's real move is chosen from these visit counts, not directly from the policy network. This is the core insight: neural networks alone give fast, cheap, imperfect intuition; tree search alone is too expensive at this scale; combining a learned prior with a learned leaf evaluation makes the search tractable and lets it correct the networks' mistakes in real time.

AlphaGo Zero: Removing the Training Wheels

A year later, DeepMind published AlphaGo Zero, which discarded the human game records entirely. It trained a single network with two output heads — one for the move policy, one for the position value — starting from random weights, and improved purely through self-play, using MCTS's own visit-count distribution as the training target for the policy head at every move (rather than a human's actual move) and the self-play game's outcome as the target for the value head. This closes a loop that AlphaGo's original design left partially open: the tree search's improved decisions are distilled straight back into the network, which then makes the next round of tree search faster and stronger, entirely without a human data set. By DeepMind's own published figures, after roughly three days of self-play (about 4.9 million games) AlphaGo Zero surpassed the strength of the AlphaGo version that had beaten Lee Sedol; after around 40 days of training it exceeded AlphaGo Master, the version that had beaten world number one Ke Jie 3–0 in May 2017. This is reinforcement learning with essentially no supervised bootstrapping at all — reward and self-play are the only teacher.

AlphaFold: A Different Machine Behind a Similar Headline

Protein folding is a different kind of problem. A protein is a chain of amino acids that, in the cell, spontaneously folds into a specific 3D shape, and that shape determines almost everything about what the protein does — which drug molecules can bind to it, which other proteins it can interact with, whether a mutation breaks it. Determining that shape experimentally, by X-ray crystallography or cryo-electron microscopy, can take months to years and does not always succeed. AlphaFold2, DeepMind's 2020 system, takes an amino acid sequence and predicts the 3D coordinates of every atom in the folded protein, at accuracy that at the CASP14 blind-assessment competition scored a median of 92.4 on the Global Distance Test — a score competitive with the resolution of experimental methods, on a benchmark where prior computational methods across the field had struggled to clear 60. The architecture that gets there has almost no resemblance to AlphaGo's. The input is a multiple sequence alignment (MSA) — the same protein sequence found across many related species, aligned so corresponding positions line up — plus, where available, related known structures used as templates. This goes through 48 stacked Evoformer blocks, which maintain and repeatedly cross-update two representations: one over the rows of the MSA (capturing evolutionary co-variation — amino acids that mutate together across species tend to be physically close in the folded structure) and one over every pair of amino acid positions (capturing likely geometric relationships). Attention operates within and between these two representations. The refined pair representation then feeds a structure module built from Invariant Point Attention layers, which converts the abstract representation into actual 3D atomic coordinates, respecting the physical fact that a protein's identity doesn't depend on how it happens to be rotated or translated in space. One more piece matters here: AlphaFold2 recycles. The whole pipeline — Evoformer through structure module — is run once, and the resulting coordinates and representations are fed back in as additional input for another pass through the same weights, typically up to three recycling iterations, refining the prediction each time. Training minimizes a loss called FAPE (Frame Aligned Point Error), which measures, from the reference frame of each predicted amino acid, how far off every other predicted atom is from its position in the real, experimentally solved structure — an ordinary geometric regression loss, computed against ground-truth structures from the Protein Data Bank, optimized by standard backpropagation and gradient descent.

The Misconception: "AlphaFold Is Also Reinforcement Learning"

Here is the mistake almost every student makes on first encountering these two systems together: because both came from DeepMind, both were framed in the press as landmark "AI solves an unsolvable problem" stories, and AlphaFold's recycling loop superficially resembles a sequence of decisions being refined over time, it's natural to assume AlphaFold is also trained with reinforcement learning. It is not. There is no agent choosing actions from a policy. There is no reward signal, sparse or otherwise — FAPE is a direct, dense, per-atom geometric error against a known correct answer, available at every single training example, computed the moment a prediction is made. There is no exploration-exploitation tradeoff, no self-play, no Monte Carlo tree search, no credit-assignment problem across a trajectory of decisions with delayed feedback. Recycling looks sequential, but each recycling pass is optimized by ordinary backpropagation through the network's weights (the original implementation stops gradients through all but the final recycling iteration, precisely because backpropagating through many iterations is not doing anything reward-related — it's a memory-saving choice about how far back to differentiate a supervised loss, nothing more). AlphaFold is best classified as supervised deep learning with an attention-based architecture (the Evoformer is, structurally, a close cousin of the transformer you'd meet in an NLP chapter, applied to biological sequences instead of language) plus an engineered iterative-refinement trick. AlphaGo is reinforcement learning in the textbook sense: an MDP, a policy optimized by a reward-driven gradient, self-play generating its own training data. Both are extraordinary achievements from the same research lab in the same decade — that is the entire basis for the confusion, and it is not a basis for calling them the same kind of machine learning.

Mechanism, Side by Side

Active Recall

Attempt every question before reading its answer.

1. Write out the (state, action, reward) formalization of Go as an MDP. Why is the reward zero for every move except the last?
2. In AlphaGo's training pipeline, identify which single step is the actual reinforcement-learning update, and name the algorithm it uses.
3. Using the PUCT formula with c_puct = 2.0, and a node with P = 0.4, N = 5, Q = 0.5, and parent total visits ΣN = 20, compute the PUCT score.
4. Why does AlphaGo's value network only sample one position per self-play game when generating its training set, instead of using every position from every game?
5. Explain, in your own words, why AlphaFold's recycling mechanism does not make it a reinforcement-learning system, even though it involves the model revisiting its own output multiple times.
6. AlphaGo Zero removed the supervised warm-start entirely. What replaced human game records as the training target for its policy head?

Answers

1. State sₜ = the board position at turn t; action aₜ = a legal stone placement or pass; reward = 0 at every non-terminal turn, +1 or −1 only when the game ends (win or loss). The reward is zero elsewhere because no single Go move has an objectively correct value in isolation — only the final territory count determines the winner, so the game genuinely has a sparse, delayed reward, and any credit given earlier would be a heuristic guess rather than a fact about the game.
2. The actual RL step is stage 2: refining the policy network via self-play using the policy-gradient (REINFORCE) update ∇log π(a|s)·z, where z is the terminal outcome of the self-play game. Stage 1 (supervised imitation of human moves) and stage 3 (regression of the value network onto game outcomes) are both ordinary supervised learning; only stage 2 updates weights directly using a reward signal from actions the network itself chose.
3. sqrt(ΣN) = sqrt(20) ≈ 4.472. Exploration term = 2.0 · 0.4 · 4.472 / (1+5) = 3.578/6 ≈ 0.596. PUCT = Q + exploration = 0.5 + 0.596 ≈ 1.096.
4. Positions within a single game are highly correlated with each other and with that game's final outcome — the whole game shares one label. Using many positions from the same game would let the value network effectively memorize individual games rather than learn a general position evaluator, badly overfitting; sampling one position per game keeps training examples closer to independent.
5. Reinforcement learning requires an agent selecting actions to maximize an expected reward, typically under uncertainty about which action is best, with credit assigned across a trajectory of decisions. AlphaFold's recycling loop re-runs the same network on its own prior output purely to refine a prediction — there is no action being selected from alternatives, no reward function, and no exploration; every training example is scored by a fixed geometric error (FAPE) against a known ground-truth structure, and weights update by ordinary gradient descent on that error. Repetition of a computation is not the same thing as sequential decision-making under reward.
6. The visit-count distribution over moves produced by AlphaGo Zero's own MCTS at the root of the search became the training target for the policy head — the network was trained to predict what full tree search would decide, rather than to imitate a human's recorded move, closing the loop entirely within self-play.

Think About It

Think about this: How would you explain alphago and alphafold: rl success stories to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where alphago and alphafold: rl success stories is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting alphago and alphafold: rl success stories to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind alphago and alphafold: rl success stories, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.