Actor-Critic Methods: A2C and PPO

During an IPL death over, ESPNcricinfo's Win Predictor ticks a percentage on screen after every ball — 61%, then 68%, then 54%. That number is not a guess; it is a trained value function, a model that looks at a match state (runs needed, balls left, wickets in hand, batter-bowler matchup) and outputs a single scalar: the probability that this state eventually leads to a win. Meanwhile, the bowler is making a completely different kind of decision each ball — yorker, short ball, wide yorker, slower ball — without any way of knowing, in the moment, whether that specific choice was the reason the team's win probability rose or fell. One system commits to actions under uncertainty. A separate system, using far more accumulated data, judges how good the situation is. That split of labor — a decision-maker and an evaluator, trained together but doing different jobs — is exactly the actor-critic architecture that powers modern reinforcement learning, from Atari-playing agents to the reward-model-guided fine-tuning behind large language models. This chapter builds actor-critic methods from the ground up, derives A2C precisely, and shows why Proximal Policy Optimization (PPO) — the algorithm behind most production RL systems today, including RLHF for language models — had to be invented to fix a flaw that A2C cannot fix on its own.

From REINFORCE to Actor-Critic: Taming Variance

Recall the policy gradient theorem: for a stochastic policy π_θ(a|s), the gradient of expected return with respect to parameters θ is

∇θ J(θ) = E_τ [ Σ_t ∇θ log π_θ(a_t | s_t) · G_t ]

where G_t is the Monte Carlo return from time t to the end of the episode. This is REINFORCE, and it works — it is an unbiased estimator of the true gradient. But it has a crippling practical flaw: variance. G_t is the sum of every reward from now until the episode ends, and in almost any real environment that sum is noisy for reasons that have nothing to do with the quality of the action at time t. A bowler bowls a perfect low full-toss outside off; the batter still gets a top edge that flies for four. The action was correct — the return was bad. Multiply that noise across an entire over, an entire innings, and the gradient estimate you compute from a handful of episodes swings wildly from batch to batch. High-variance gradients mean you need enormous numbers of samples to see a clear direction, and training crawls.

The standard fix is a baseline. Instead of scaling the log-probability gradient by the raw return G_t, subtract any function b(s_t) that does not depend on the action taken:

∇θ J(θ) = E [ ∇θ log π_θ(a_t | s_t) · (G_t − b(s_t)) ]

Why is this still an unbiased estimator of the same gradient? Because the extra term introduced by the baseline vanishes in expectation over actions, for any state-only function b(s):

E_{a~π}[ ∇θ log π_θ(a|s) · b(s) ]
  = b(s) · Σ_a π_θ(a|s) ∇θ log π_θ(a|s)
  = b(s) · Σ_a ∇θ π_θ(a|s)          (since π∇logπ = ∇π)
  = b(s) · ∇θ ( Σ_a π_θ(a|s) )
  = b(s) · ∇θ (1)  =  0

The sum of probabilities over all actions is always exactly 1, a constant, so its gradient is zero regardless of what b(s) is. That means any baseline leaves the mean of the gradient estimator untouched — it only changes the variance. A well-chosen baseline can shrink that variance enormously, because G_t − b(s_t) is small and centred near zero whenever the outcome was roughly what you'd expect from that state, and only large when the outcome genuinely surprised you. The best possible choice of baseline, it turns out, is the true state-value function itself.

The Critic as Baseline: Advantage and TD-Error

Define V^π(s) = E[G_t | s_t = s] — exactly the win-probability number the commentary graphic shows: the expected return from this state onward, under the current policy. And define Q^π(s,a) = E[G_t | s_t=s, a_t=a], the expected return if you additionally commit to a specific action. The quantity

A(s,a) = Q(s,a) − V(s)

is the advantage: how much better (or worse) taking action a is than the policy's own average behaviour from that state. This is precisely G_t − b(s_t) with the optimal baseline plugged in, and it has an intuitive reading: positive advantage means "that action beat expectations, do more of it"; negative means "that action underperformed, do less of it."

Computing Q(s,a) directly usually requires a second network taking both state and action as input. Actor-critic methods sidestep this with a cheaper, one-step estimate. Instead of waiting for the full Monte Carlo return G_t, bootstrap from the critic's own value estimate one step ahead:

δ_t = r_t + γ · V(s_t+1) − V(s_t)

This is the temporal-difference (TD) error, and under a correct V, E[δ_t | s_t, a_t] = A(s_t, a_t) — it is an estimate of the advantage that only needs one transition, not a whole episode, which is dramatically lower variance than raw Monte Carlo returns. The critic's entire job is to make V accurate enough that δ_t is a trustworthy advantage signal; it is trained by ordinary regression, minimizing (V_φ(s_t) − (r_t + γV_φ(s_t+1)))². The actor's job is to use δ_t in place of G_t − b(s_t) in the policy gradient. Two learners, two losses, one shared stream of experience.

Actor-Critic Architecture

Concretely this means two function approximators. The actor is a network (often ending in a softmax over discrete actions, or a Gaussian mean/variance for continuous control) that outputs π_θ(a|s). The critic is a network outputting a single scalar V_φ(s). They can be entirely separate networks, or — very common in practice, since both need to understand the same state — share early layers and split into two heads near the output, cutting compute and letting representation learning help both jobs at once. The diagram below traces one full update: a state goes into both the actor and the critic; the actor samples an action that the environment executes; the environment returns a reward and a next state; the critic scores both states to form the TD-error; and that single scalar δ_t flows back to update both networks.

The two dashed cyan arrows are the entire point of this chapter: a single scalar δ_t, computed once per transition, simultaneously tells the actor which action to reinforce and tells the critic how wrong its value estimate was. Everything from here — A2C, PPO, and the RLHF pipelines that fine-tune large language models — is a variation on how carefully that scalar is used to move the actor's parameters.

A2C: Synchronous Advantage Actor-Critic

A2C ("Advantage Actor-Critic") is the direct implementation of the loop above, with two engineering choices layered on. First, it runs several copies of the environment in parallel — several simulated "net sessions" collecting experience simultaneously — and averages the gradient across them before each update, which further reduces variance without needing longer episodes. This is the synchronous cousin of the earlier A3C algorithm (Mnih et al., 2016), which let worker threads update a shared network asynchronously and lock-free; A2C instead waits for every worker to finish its batch and applies one synchronized update, which turned out to use GPU batching more efficiently for equal or better final performance. Second, A2C adds an entropy bonus to the loss to stop the policy from collapsing to a single deterministic action too early, which would kill exploration. The combined loss per transition is:

L(θ,φ) = − log π_θ(a_t|s_t) · δ_t        (actor / policy loss)
        + c1 · δ_t²                       (critic / value loss, MSE)
        − c2 · H(π_θ(·|s_t))              (entropy bonus, encourages exploration)

with c1, c2 small weighting constants (typically c1≈0.5, c2≈0.01). Note the sign: minimizing −log π · δ is the same as gradient-ascending on log π · δ, which pushes probability mass toward actions with positive advantage and away from actions with negative advantage, exactly as intended.

Let's trace one real update, in the bowler's chair. Suppose the critic currently estimates the win probability at the start of this ball as V(s_t) = 0.55. The actor's policy over two shortlisted deliveries is π(bouncer) = 0.3, π(yorker) = 0.7 — the bowler leans yorker, but samples the less likely action, a bouncer, this ball. It draws a top-edge chance that goes begging but still shifts momentum: the critic re-evaluates the resulting state at V(s_t+1) = 0.75. Take r_t = 0 (no terminal reward mid-innings; the win-probability shift itself carries the signal) and γ = 0.99.

def actor_critic_step(pi_bouncer, pi_yorker, reward, gamma, V_s, V_s_next):
    td_target = reward + gamma * V_s_next
    advantage = td_target - V_s
    # softmax gradient: d logpi(chosen)/dz_chosen = 1 - pi(chosen)
    #                    d logpi(chosen)/dz_other  = -pi(other)
    grad_z_bouncer = (1 - pi_bouncer) * advantage   # chosen action = bouncer
    grad_z_yorker  = -pi_yorker * advantage
    return td_target, advantage, grad_z_bouncer, grad_z_yorker

td_target, advantage, g_bouncer, g_yorker = actor_critic_step(
    pi_bouncer=0.3, pi_yorker=0.7, reward=0.0, gamma=0.99, V_s=0.55, V_s_next=0.75)

print(f"TD target      r_t + gamma*V(s_t+1): {td_target:.4f}")
print(f"Advantage delta_t                  : {advantage:.4f}")
print(f"grad wrt bouncer logit (ascent dir): {g_bouncer:.5f}")
print(f"grad wrt yorker  logit (ascent dir): {g_yorker:.5f}")

TD target      r_t + gamma*V(s_t+1): 0.7425
Advantage delta_t                  : 0.1925
grad wrt bouncer logit (ascent dir): 0.13475
grad wrt yorker  logit (ascent dir): -0.13475

Work the arithmetic by hand to see why: 0.99 × 0.75 = 0.7425; 0.7425 − 0.55 = 0.1925 — the advantage is positive, so the sampled action outperformed what the critic expected from this state. For a two-action softmax, the gradient of the log-probability of the chosen action with respect to its own logit is 1 − π(chosen) = 1 − 0.3 = 0.7, and with respect to the other logit is −π(other) = −0.7. Scaling both by the advantage 0.1925 gives ±0.13475: the update nudges the bouncer logit up and the yorker logit down by the same magnitude, since the two must move oppositely to keep probabilities summing to one. Simultaneously, the critic's own parameters move to shrink the gap between V(s_t)=0.55 and the TD target 0.7425 — the critic is, in effect, updating its estimate of "how good is this match situation" using exactly the same transition. In production A2C, this per-step TD target is often replaced with an n-step return or a Generalized Advantage Estimate (GAE(λ)), which blends multiple bootstrap horizons to trade off bias against variance more smoothly — the core mechanics of the actor and critic losses are unchanged.

Why A2C Still Breaks: The Step-Size Problem

A2C is a strict improvement over REINFORCE, but it inherits a structural danger from all vanilla policy-gradient methods: there is no guaranteed relationship between the size of a parameter update and the size of the resulting change in the policy's behaviour. A small step in θ-space can, for a policy near a decision boundary, cause a large jump in π(a|s) — sometimes flipping the policy's preferred action almost entirely. Because the next batch of training data is collected using the new, possibly-collapsed policy, a single overly aggressive update can permanently destroy performance: unlike supervised learning, where a bad gradient step just gets corrected by the next mini-batch of fixed data, in RL a bad step corrupts the very data-generating process used for every future step. The agent can get trapped in a policy that never recovers.

The first rigorous fix was Trust Region Policy Optimization (TRPO), which explicitly constrains each update so that the KL-divergence between the old and new policy stays below a hard threshold, computed via the Fisher information matrix and conjugate-gradient optimization. It works, but it is expensive and finicky to implement correctly — exactly the kind of complexity that discourages wide adoption. PPO was designed to get nearly the same safety guarantee with a first-order method any standard optimizer (Adam, SGD) can run directly.

PPO: The Clipped Surrogate Objective

PPO keeps the actor-critic loop entirely intact — same critic, same advantage estimate, same entropy bonus — and only changes the actor's objective. Define the probability ratio between the current policy being optimized and the policy that actually collected the data:

r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t)

At the moment an update begins, r_t = 1 everywhere, since θ = θ_old. As the optimizer takes several gradient steps on the same batch of collected trajectories (PPO reuses each batch for multiple epochs, unlike A2C's single pass), r_t drifts away from 1. The clipped surrogate objective is:

L^CLIP(θ) = E_t [ min( r_t(θ) · A_t,  clip(r_t(θ), 1−ε, 1+ε) · A_t ) ]

with ε = 0.2 almost universally in practice. Reusing the bowler's numbers — π_old(bouncer) = 0.3, advantage A = 0.1925 — trace what happens across several epochs of reusing the same batch, as π_new(bouncer) is pushed upward by the gradient:

def ppo_clip(pi_new, pi_old, advantage, epsilon=0.2):
    ratio = pi_new / pi_old
    unclipped = ratio * advantage
    clipped_ratio = min(max(ratio, 1 - epsilon), 1 + epsilon)
    clipped = clipped_ratio * advantage
    objective = min(unclipped, clipped)
    return ratio, unclipped, clipped, objective

for pi_new in [0.3, 0.4, 0.5, 0.6]:
    ratio, unclipped, clipped, obj = ppo_clip(pi_new, pi_old=0.3, advantage=0.1925)
    print(f"pi_new={pi_new:.1f}  ratio={ratio:.3f}  unclipped={unclipped:.4f}"
          f"  clipped={clipped:.4f}  L_CLIP={obj:.4f}")

pi_new=0.3  ratio=1.000  unclipped=0.1925  clipped=0.1925  L_CLIP=0.1925
pi_new=0.4  ratio=1.333  unclipped=0.2567  clipped=0.2310  L_CLIP=0.2310
pi_new=0.5  ratio=1.667  unclipped=0.3208  clipped=0.2310  L_CLIP=0.2310
pi_new=0.6  ratio=2.000  unclipped=0.3850  clipped=0.2310  L_CLIP=0.2310

At pi_new = 0.3 the ratio is exactly 1 and nothing is clipped — the objective simply equals the advantage. Once pi_new passes roughly 0.36 (where the ratio crosses 1.2), the clip term freezes the reward the optimizer receives from pushing the probability further, at 1.2 × 0.1925 = 0.2310, even though the unclipped surrogate keeps climbing past that point (0.2567, then 0.3208, then 0.3850). Because the objective is a min, the optimizer sees no further gradient benefit from moving pi_new beyond the point where the ratio hits 1+ε — the incentive to keep amplifying an already-rewarded action is capped, which is precisely how PPO prevents a single good batch from swinging the policy too far. For actions with negative advantage the same clipped-min formula is designed to symmetrically prevent the optimizer from over-correcting the ratio too far below 1 — Schulman et al. (2017) show the clip only removes the extra incentive to move the ratio further outside [1−ε, 1+ε] in whichever direction would otherwise increase the objective, while leaving a corrective gradient available if the ratio has to move back toward 1. No KL divergence, no Fisher matrix, no conjugate gradient — just a clamp, and it works well enough that PPO became the default policy-optimization algorithm across game-playing agents, robotics, and RLHF fine-tuning of language models.

Common Misconception: The Critic Doesn't Vote on Actions

Students who have just studied GANs, or Q-learning with two networks (online and target), tend to picture the critic as a second decision-maker — as if the actor proposes an action and the critic approves or vetoes it, the way a discriminator judges a generator's output. That is wrong, and the confusion causes real bugs when students later implement actor-critic code and wire the critic's output into the action-selection step. The critic in A2C or PPO never sees an action as input, and it never influences which action gets sampled at run time — it is a function of state alone, V_φ(s), not Q_φ(s,a). Its entire role is retrospective and indirect: after the actor has already committed to an action and the environment has already responded, the critic estimates how good the resulting states were, so that the advantage δ_t can tell the actor whether that already-taken action deserves more or less probability next time. Behaviour is decided by the actor alone; the critic only ever scores, never chooses. This is also why actor-critic methods remain on-policy in their classic form (A2C, PPO) — the critic's value estimates and the advantage they produce are only valid for the policy that is currently generating the data, unlike the Q-function in DQN, which can legitimately be trained from stale, off-policy replay data because it conditions on the action explicitly.

Active Recall

Attempt these before reading the answers.

1. Why does subtracting a state-dependent baseline b(s_t) from the return in a policy-gradient estimator leave the expected gradient unchanged?
2. Given V(s_t)=0.55, V(s_t+1)=0.75, r_t=0, γ=0.99, compute the TD-error δ_t.
3. A two-action softmax policy has π(bouncer)=0.3, π(yorker)=0.7. The bouncer is sampled and the advantage comes out to 0.1925. What is the sign and rough magnitude of the gradient-ascent push on the yorker logit, and why does it move in that direction?
4. Why can't A2C simply be trained with a much larger learning rate to speed up convergence, in the way you might for a supervised image classifier?
5. Compute the PPO clipped surrogate objective for π_old(bouncer)=0.3, π_new(bouncer)=0.5, advantage =0.1925, ε=0.2.
6. True or false: the critic network in A2C/PPO takes both the state and a candidate action as input and picks the one with the highest score.

Answers.

1. Because Σ_a π_θ(a|s) ∇θ log π_θ(a|s) = Σ_a ∇θ π_θ(a|s) = ∇θ (Σ_a π_θ(a|s)) = ∇θ(1) = 0. The sum of action probabilities is always exactly 1 no matter what θ is, so its gradient is the zero vector — any state-only baseline contributes exactly zero to the expected gradient, regardless of its value. It only reduces the variance of the sample estimate.

2. δ_t = 0 + 0.99 × 0.75 − 0.55 = 0.7425 − 0.55 = 0.1925.

3. The gradient with respect to the non-chosen action's logit is −π(yorker) × advantage = −0.7 × 0.1925 = −0.13475. It is negative — gradient ascent decreases the yorker logit — because the bouncer (the action actually taken) outperformed expectations, so probability mass has to move away from the alternative to increase the probability of the bouncer; in a two-action softmax the two logit gradients are always equal and opposite.

4. A large step in parameter space does not correspond predictably to a small or safe step in policy-distribution space — near a decision boundary, a modest parameter change can flip which action the policy prefers almost entirely. Since the next batch of training data is collected using the freshly updated policy, an overly large step can corrupt the data-generating process itself, unlike supervised learning where bad gradient steps are self-correcting against a fixed dataset. This can permanently collapse performance, which is the exact failure PPO's clipping is built to prevent.

5. Ratio = 0.5/0.3 = 1.667. Unclipped = 1.667 × 0.1925 ≈ 0.3208. Clip bounds are [0.8, 1.2], so the ratio clips down to 1.2, giving clipped = 1.2 × 0.1925 = 0.2310. L_CLIP = min(0.3208, 0.2310) = 0.2310 — the optimizer's incentive to keep increasing this action's probability is capped once the ratio exceeds 1.2.

6. False. The critic V_φ(s) is a function of state alone and never takes an action as input in A2C/PPO; it does not choose or rank actions. Action selection is entirely the actor's job — the critic only evaluates states, after the fact, to produce the advantage signal that trains the actor.

Think About It

Think about this: How would you explain actor-critic methods: a2c and ppo to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind actor-critic methods: a2c and ppo, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.