Actor-Critic: Policy + Value Learning

An IRCTC Tatkal booking agent has to decide, ten times a second during the opening rush, which server route to send a request down. Pick the fast, lightly loaded path and the booking clears; pick the congested one and it times out. The agent could learn a value for every route — "this route is worth 0.7, that one is worth 0.3" — and always pick the highest, the way Q-learning does. Or it could learn a probability distribution directly — "send 70% of traffic here, 30% there" — and sample from it. The first is a value method. The second is a policy method. Actor-critic is what happens when you stop choosing between them and run both at once, each one fixing the other's biggest weakness.

This chapter builds actor-critic from the two methods it fuses, derives its update equations from first principles, traces a full numeric episode by hand, and shows where the method breaks if you get the arithmetic wrong.

Two failure modes, one architecture

Recall from Q-learning (Grade 11, model-free RL) that a value method learns Q(s, a), the expected long-run return of taking action a in state s, then acts greedily: always pick argmax_a Q(s, a). This works cleanly when actions are discrete and few. It breaks down in two situations that actor-critic exists to solve.

Continuous or high-dimensional action spaces. If the IRCTC load balancer isn't choosing among 4 fixed routes but setting a continuous traffic-split percentage between 0% and 100%, argmax_a Q(s, a) requires solving an optimization problem over a continuous variable at every single decision — for every state, every step. That's computationally ruinous at Tatkal-rush query rates.

The need for stochastic policies. A pure value method is deterministic once trained: same state, same action, forever. But an adversary — or just another user hammering the same route — can exploit a predictable policy. A cricket team on the field can't always send the same fielder to cover a low full toss; a mixed strategy is what game theory calls a Nash equilibrium under uncertainty. Policy methods learn π(a|s) directly, a probability distribution the agent samples from, which handles both continuous actions and useful randomness naturally.

So why not just use a pure policy method (REINFORCE, which you may have seen as the simplest policy-gradient algorithm) and skip value functions altogether? Because REINFORCE has a variance problem so severe it can take orders of magnitude more episodes to converge. Actor-critic exists specifically to fix that variance problem while keeping the policy-gradient machinery.

The policy gradient, from scratch

A policy π_θ(a|s) is a function — parameters θ, typically a neural network's weights — that outputs a probability distribution over actions given a state. Training it means adjusting θ to make good actions more probable and bad actions less probable. The question is: what counts as "good," and how do we compute a gradient for something that involves sampling?

Define J(θ) as the expected total return of following policy π_θ. The policy gradient theorem (you can take this as given at G11 level, but the shape matters) states:

∇_θ J(θ) = E_π[ ∇_θ log π_θ(a|s) · G_t ]

where G_t is the actual return (sum of discounted future rewards) obtained from time t onward in a sampled trajectory. Read this carefully: it says push up the log-probability of action a in state s, scaled by how good the outcome turned out to be. If G_t is large and positive, θ moves to make that action more likely next time. This is REINFORCE, and it is an unbiased estimator of the true gradient — but G_t is a single Monte Carlo sample of a random variable that depends on every reward from step t to the end of the episode. Two runs of the identical policy from the identical state can produce wildly different G_t values just from environment randomness (a different train's Tatkal window opened a second later, a different server got a network hiccup). That's high variance, and high variance means noisy gradients, means slow, unstable learning.

The critic's job: replace the noisy return with a baseline-corrected estimate

The fix does not touch the unbiasedness of the gradient — it targets the variance. Subtract any baseline b(s) that does not depend on the action, and the expectation is provably unchanged:

E_π[ ∇_θ log π_θ(a|s) · (G_t − b(s)) ] = ∇_θ J(θ)

This holds because E_a[∇_θ log π_θ(a|s)] = 0 for any fixed s — a standard identity of probability distributions (the expected gradient of a normalized log-density is zero; intuitively, probabilities must still sum to 1, so there's no direction in which "on average" they can rise). Subtracting a state-dependent baseline removes a term that averages to zero, so it costs nothing in expectation but can shrink variance enormously if b(s) is close to G_t's typical value.

The best choice of baseline is the state's own value function, V(s) — the expected return from s under the current policy. That gives the advantage function:

A(s, a) = G_t − V(s)   (or, using one-step bootstrapping: A(s,a) ≈ r + γV(s′) − V(s))

A(s, a) answers a sharper question than G_t alone: not "was the outcome good," but "was this action better or worse than what I'd expect from this state anyway." That is exactly the signal the actor needs, and it is far less noisy because it's centered around zero rather than around whatever raw magnitude rewards happen to have.

This is the whole architecture in one sentence: the actor is the policy π_θ(a|s), updated by policy gradient using the advantage as its signal; the critic is a learned value function V_w(s) (separate parameters w, usually a second small network or the same network's second head), trained by ordinary TD learning to estimate returns accurately, and its only job is to supply that low-variance advantage estimate to the actor.

Diagram: the two-network loop

Trace the loop: the environment hands state s to both networks. The actor samples an action from π_θ(a|s) and fires it into the environment, which returns a reward r and next state s′. The critic evaluates V_w(s) and V_w(s′) and combines them with r into the TD error, which doubles as the advantage estimate. That single number flows two directions: it scales the actor's policy-gradient update (dashed blue), and its square is the loss the critic minimizes to make its own value estimate more accurate (dashed purple). Two networks, one shared scalar, two different uses of it.

Worked example: a 3-state ISRO ground-station queue

Consider a stripped-down scheduler choosing between two actions — A (process the priority downlink packet) and B (process the routine telemetry packet) — across 3 states {S0, S1, S2} where S2 is terminal. Discount γ = 0.9, learning rates α_actor = 0.1, α_critic = 0.5.

Initialize: V_w(S0) = 0, V_w(S1) = 0, V_w(S2) = 0 (terminal states are always 0 by definition). Policy at S0 starts as a softmax over two logits, both 0, giving π(A|S0) = π(B|S0) = 0.5.

Step 1. Agent is in S0, samples action A (priority downlink), environment returns reward r = 2 and next state S1.

Compute the TD error, which serves directly as the one-step advantage estimate:

δ = r + γV_w(S1) − V_w(S0)
  = 2 + 0.9(0) − 0
  = 2

Critic update (gradient descent on squared TD error, which for a tabular/linear critic reduces to moving V_w(S0) toward the TD target):

V_w(S0) ← V_w(S0) + α_critic · δ = 0 + 0.5(2) = 1.0

Actor update. The policy gradient step for a softmax policy has two halves: for the chosen action's own logit, ∇_θ log π_θ(a|s) = (1 − π_θ(a|s)); for every other action b, the same gradient with respect to b's logit is −π_θ(b|s) (the standard softmax-gradient identity — probability mass gained by the chosen action's logit is mirrored by a push down on every other logit). For action A, chosen with probability 0.5, and its only alternative B:

Δ(logit of A at S0) = α_actor · δ · (1 − π(A|S0)) = 0.1 × 2 × 0.5 = 0.1
Δ(logit of B at S0) = α_actor · δ · (0 − π(B|S0)) = 0.1 × 2 × (−0.5) = −0.1

The A-logit at S0 rises from 0 to 0.1; the B-logit falls from 0 to −0.1 — both actions move, in opposite directions. Recomputing the softmax: π(A|S0) = e^0.1/(e^0.1+e^−0.1) = 1.105/2.010 ≈ 0.550. Action A — the one that just paid off — is now 5.0 percentage points more likely. That is the entire mechanism of actor-critic learning, executed once.

Step 2. Agent is now in S1, samples action B (routine telemetry), gets r = 5, reaches terminal state S2.

δ = r + γV_w(S2) − V_w(S1) = 5 + 0.9(0) − 0 = 5
V_w(S1) ← 0 + 0.5(5) = 2.5

Actor logit for B at S1 rises by 0.1 × 5 × 0.5 = 0.25 — a bigger push than step 1 got, because the advantage was larger. This is exactly the design intent: actions that beat the critic's expectation by more get reinforced more.

Step 3 (why the critic matters — a second pass through S0). Suppose the agent later returns to S0, again samples A, again gets r = 2, lands in S1 — but now V_w(S1) = 2.5 from step 2's update:

δ = 2 + 0.9(2.5) − 1.0 = 2 + 2.25 − 1.0 = 3.25

Same immediate reward as step 1, but a larger advantage — because the critic now knows S1 leads somewhere good, and that knowledge propagates backward into the value of taking A at S0, without waiting for the episode to finish and without needing a full Monte Carlo rollout. That backward propagation through V_w — bootstrapping — is precisely what a pure REINFORCE agent cannot do; REINFORCE would have to wait for the whole trajectory's actual return, replay it, and only then update, which is both slower and noisier per update.

Common misconception: "the critic tells the actor what to do"

Students who've just learned Q-learning tend to assume the critic is choosing actions — that V_w(s) is somehow being maximized over actions the way Q(s,a) was. It is not. V_w(s) takes only a state as input; it has no notion of "which action is best," because it never evaluates individual actions at all. It only estimates "how good is it to be in state s, on average, under my current policy." The actor is the only component that ever decides an action, via sampling from π_θ(a|s). The critic's entire contribution is grading how surprising the outcome of that action was, after the fact, so the actor's next update is less noisy. Swap the critic for a random constant and the actor still learns (slowly, with high variance) — that's REINFORCE with a bad baseline. Swap the actor for "pick argmax over some function of state" and you no longer have actor-critic; you have Q-learning wearing a different name. The two components are not interchangeable, and neither one alone performs the other's role.

Why this scales where Q-learning doesn't

Return to the IRCTC continuous traffic-split example. A Q-learning agent would need argmax over a continuous action at every decision, which either requires discretizing the action space (losing precision, and the discretization grid itself becomes a hyperparameter) or solving a nested optimization per step (too slow for real-time routing). An actor-critic agent's actor can output the parameters of a continuous distribution directly — for instance a Beta distribution over the [0,1] traffic-split — and sampling from it is a single cheap draw, no inner optimization loop. This is why actor-critic variants (A2C, A3C, PPO — which you may meet later as extensions that stabilize the same core idea) are the standard for robotics, continuous control, and any system, like adaptive request routing, that acts in a continuous space many times per second.

Active recall

Q1. A pure REINFORCE agent and an actor-critic agent both use the policy gradient theorem. What specific quantity differs between their update rules, and why does that difference reduce variance?

Q2. In the worked example, why did subtracting V_w(S0) = 1.0 from the target in step 3 not bias the actor's gradient in expectation?

Q3. Suppose the critic is badly miscalibrated — say V_w(s) is initialized to a huge positive number for every state and never corrected. What happens to the actor's learning, qualitatively?

Q4. Why can't the critic alone (without an actor) be used to select actions in a continuous action space the way argmax_a Q(s,a) works for discrete actions?

Q5. In step 1 of the worked example, recompute the actor update if the reward had instead been r = 0 (no benefit from processing the priority packet) instead of r = 2. What sign does the logit update take, and what does that mean for π(A|S0) going forward?

Q6. Explain in one sentence why the critic is trained with a squared-error (regression) loss while the actor is trained with a policy-gradient loss — why can't both use the same kind of loss?

Answers

A1. REINFORCE uses the raw Monte Carlo return G_t (or G_t − baseline with a fixed, state-independent baseline) as the scaling signal; actor-critic replaces it with the advantage A(s,a) = r + γV_w(s′) − V_w(s), a bootstrapped one-step estimate. G_t is the sum of every future reward in a full trajectory sample, so its variance accumulates over the entire episode length. The advantage only depends on one transition plus the critic's (already averaged, lower-variance) estimate of the future, so its variance is far smaller — at the cost of introducing a small bias from an imperfect critic, a classic bias-variance tradeoff.

A2. Because V_w(S0) does not depend on the action a being taken (it's a function of state alone), and the identity E_a[∇_θ log π_θ(a|s)] = 0 guarantees that subtracting any action-independent quantity leaves the expected gradient unchanged. The 1.0 subtracted in step 3 shifts the specific sampled gradient's magnitude, but averaged over many samples from the same state, it contributes nothing net — pure variance reduction, no bias.

A3. If V_w(s) is stuck at a huge constant for every state, the TD error δ = r + γV_w(s′) − V_w(s) becomes dominated by the difference of two huge, nearly-equal numbers (γV_w(s′) − V_w(s)), which floating-point arithmetic renders noisy and roughly zero regardless of the actual reward — so the actor receives almost no useful learning signal and effectively stalls, or updates on numerical noise. This is why the critic needs its own working TD-learning loop; a frozen or badly initialized critic starves the actor.

A4. V_w(s) takes only a state as input — it was never built to be evaluated at different candidate actions, so there is nothing to take an argmax over. (Compare Q(s,a), which does take an action, but suffers the same continuous-optimization problem discussed above.) Only the actor, which outputs an actual distribution over actions, can produce a specific action or action-distribution to execute.

A5. δ = 0 + 0.9(0) − 0 = 0, so Δ(logit of A) = 0.1 × 0 × 0.5 = 0. Zero advantage means zero update: the actor makes no change to π(A|S0) at all, because a reward of exactly what the critic already expected (0, matching V_w(S0)=0 at that point) carries no new information about whether A was a good or bad choice relative to baseline.

A6. The critic is solving a regression problem — predict a real-valued number (expected return) as accurately as possible — which squared error is the natural loss for; the actor is solving a credit-assignment problem over a probability distribution — increase or decrease the log-probability of a sampled action in proportion to a scalar signal — which is what the policy gradient theorem specifies directly and squared error has no meaningful role in.

Think About It

Think about this: How would you explain actor-critic: policy + value learning to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where actor-critic: policy + value learning is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting actor-critic: policy + value learning to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind actor-critic: policy + value learning, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.