Policy Gradient Methods in Reinforcement Learning

It is the final over of a tight T20 run chase. Nineteen runs are needed off six balls, and the ball is in the hands of a death-over specialist — the kind of bowler whose entire craft, as commentators love to point out about someone like Jasprit Bumrah, lies in never letting the batter settle into a pattern. Over the course of a match, and across an entire season, this bowler is quietly running an experiment on every single delivery: try a full yorker at the base of off stump, watch what happens; try a chest-high bouncer, watch what happens; try a slower ball that dies off the pitch, watch what happens. A dot ball or a wicket nudges the bowler toward repeating that choice in a similar situation next time. A boundary nudges the bowler away from it. Nobody sits the bowler down with a spreadsheet listing the value of every possible delivery in every possible match situation — the adjustment happens directly, ball by ball, on the decision itself.

A machine can learn the same way: by directly adjusting its decision-making rule, called a policy, in the direction that produced better outcomes, instead of first learning to estimate the value of every state and action and only then behaving greedily with respect to those values. This family of techniques is called policy gradient methods, and it is one of the two great branches of modern reinforcement learning — the branch that scales to continuous, high-dimensional action spaces, that naturally produces the kind of deliberate unpredictability a good death-over bowler needs, and that sits underneath some of the most consequential AI systems built in the last few years, including the human-feedback fine-tuning stage of large language models.

From Estimating Values to Learning Actions Directly

By this point you have almost certainly met value-based reinforcement learning: an agent learns a function such as Q(s, a), the expected long-run return of taking action a in state s and behaving well afterward, and it acts by picking argmax_a Q(s, a) — always the action with the highest estimated value, with some exploration bolted on top. This works well for problems like grid worlds or Atari-style games with a small, discrete menu of actions. But it runs into three real limits.

First, many real decision problems have continuous action spaces — the steering angle of a self-driving car, the torque applied to a robot-arm joint, the exact bid placed in an ad auction. There is no way to loop over "every possible action" and take an argmax when the action space is a continuum of real numbers.

Second, the best possible policy is sometimes genuinely stochastic, not deterministic. Our death-over bowler is a good illustration: if the bowler always, without exception, bowled a yorker on the last ball of a chase, a good batting side would simply set up in advance for the yorker, and the value of that "optimal-looking" deterministic choice would collapse. The truly optimal strategy is to bowl each delivery with some carefully tuned probability, so that no single ball is ever fully anticipated. Value-based methods, which greedily chase the single best-looking action, cannot represent this kind of deliberate randomness; policies, described directly as probability distributions, can.

Third, tiny changes in estimated Q-values can flip the argmax and cause an agent's behaviour to change abruptly, which tends to make value-based learning unstable when combined with function approximation. A policy that is adjusted a little at a time, gradient step by gradient step, tends to change smoothly instead.

Policy gradient methods respond to all three limits with one idea: instead of learning values and deriving a policy from them, parameterize the policy itself — write it as π_θ(a|s), read as "the probability of action a in state s, according to parameters θ" — and adjust θ directly, using gradient ascent, so that actions which led to high returns become more likely and actions which led to low returns become less likely.

Parameterizing the Policy

A policy is a mapping from states to a probability distribution over actions. In our simplified cricket setting, the "state" might describe the match situation — the over number, the runs required, the batter's known weaknesses — and the "action" is which delivery to bowl from a fixed menu: yorker, bouncer, slower ball, off-cutter, wide yorker, and so on. A parameterized policy computes this distribution from a set of numbers θ, which could be as simple as one number per state-action pair or as complex as the weights of a deep neural network that takes the state as input and outputs action probabilities.

The most common way to turn raw scores into a valid probability distribution is the softmax function. If h(s, a) is a numerical "preference" for action a in state s — a higher preference means the policy leans toward that action — softmax converts preferences into probabilities:

π_θ(a|s) = e^h(s,a) / Σ_b e^h(s,b)

where the sum in the denominator runs over every action b available in that state, guaranteeing the outputs are all positive and sum to exactly 1. This is the same softmax used at the output layer of a classification network; here it turns "how much the agent currently favours each delivery" into "the probability of bowling each delivery."

The Objective: What Gradient Ascent Is Climbing

Training a policy means finding the parameters θ that maximize the expected return the agent collects while behaving according to π_θ. Formally, define the objective

J(θ) = E_{τ~π_θ} [ G(τ) ]

the expected return G(τ) — the possibly discounted sum of rewards — averaged over trajectories τ (complete sequences of states, actions, and rewards from the start of an episode to its end) generated by letting the agent act according to π_θ. The learning rule is ordinary gradient ascent:

θ ← θ + α ∇_θ J(θ)

for some learning rate α. The entire difficulty of policy gradient methods is packed into that gradient term. J(θ) is an expectation over an episode played out inside an environment that is, from the agent's point of view, a black box: you cannot ask a real batter what would have happened had a different ball been bowled, and you certainly cannot differentiate through the physics of bat meeting ball. So how can you compute the gradient of an expectation whose randomness comes partly from a policy you control and partly from a world you do not?

The Log-Derivative Trick

The resolution is a short piece of calculus that every policy gradient algorithm in use today, from the 1992 original to the large-scale methods used to fine-tune modern language models, is built on. Write the expectation out as a sum over all possible trajectories, weighted by their probability under the policy:

J(θ) = Σ_τ P(τ;θ) G(τ)

Differentiating with respect to θ and using the identity ∇_θ P = P · ∇_θ log P — which follows directly from the chain rule applied to log P — turns the derivative of a probability into the original probability times the derivative of a log-probability:

∇_θ J(θ) = Σ_τ P(τ;θ) · ∇_θ log P(τ;θ) · G(τ) = E_{τ~π_θ} [ ∇_θ log P(τ;θ) · G(τ) ]

The expression has turned back into an expectation, which means it can be estimated just by sampling trajectories and averaging — no differentiation through the environment required. The remaining step is to expand P(τ;θ). A trajectory's probability factors as an alternating chain of policy decisions and environment transitions, P(τ;θ) = p(s_0) · π_θ(a_0|s_0) · p(s_1|s_0,a_0) · π_θ(a_1|s_1) ⋯. Taking the logarithm turns this product into a sum, and taking the gradient with respect to θ makes every environment-transition term vanish, since the environment's own dynamics do not depend on the agent's parameters — only the policy terms survive:

∇_θ log P(τ;θ) = Σ_t ∇_θ log π_θ(a_t|s_t)

One more refinement matters for building a practical algorithm: the reward earned at any timestep cannot depend on an action chosen later in the episode, so a whole-trajectory return G(τ) multiplying an early timestep's log-probability term can, without changing the expectation, be replaced by just the return-to-go G_t from that timestep onward — a lower-variance quantity built only from rewards the action could actually have influenced. Substituting this in gives the Policy Gradient Theorem, first proved in this general form by Richard Sutton and colleagues in 1999:

∇_θ J(θ) = E_{π_θ} [ Σ_t ∇_θ log π_θ(a_t|s_t) · G_t ]

Read this as an instruction: for every action the agent actually took, nudge its parameters in the direction that increases the log-probability of that action, scaled by how good the outcome turned out to be from that point on. Actions followed by large returns get pushed up harder; actions followed by poor returns get pushed down. Nothing here required knowing how the environment works internally — only that the agent could sample from it.

REINFORCE: Turning the Theorem into an Algorithm

Ronald Williams introduced the first working version of this idea in 1992, in an algorithm he named REINFORCE. It estimates the Policy Gradient Theorem's expectation with plain Monte Carlo sampling — play out full episodes, observe the actual returns, and use them directly in place of the true, unknown expected return:

• Initialize the policy parameters θ, typically to small random values.
• Repeat for many episodes: run the current policy π_θ from the start of an episode to the end, recording every state s_t, action a_t, and reward r_t along the way.
• For every timestep t in that episode, compute the return-to-go G_t — the discounted sum of rewards from t until the episode ends.
• Update the parameters for every timestep, θ ← θ + α · G_t · ∇_θ log π_θ(a_t|s_t), then move on to the next episode with the freshly updated policy.

Notice that G_t is doing exactly the job that Q^π(s_t, a_t) — the true action-value — plays in the Policy Gradient Theorem; a full-episode return is simply an unbiased, if noisy, sample of that quantity. That noise is REINFORCE's main weakness, and we will return to it after working through exactly what one update looks like in practice.

Worked Example: One Ball, One Gradient Step

Back to that final over: nineteen needed off six. To see the update rule act on real numbers, freeze the trajectory down to a single decision — the very first ball of the over, treated as a one-step episode, so the return G is simply the reward from that one ball and no discounting is needed. The exact same update applies unchanged inside a full six-ball over; only the length of the trajectory and the definition of G_t change.

Setup. A coach has been tracking this bowler's tendencies and has narrowed the choice down to two deliveries, Yorker (Y) and Bouncer (B). Using the simplest possible parameterization — where the preference for each delivery is just its own learnable number, so h(s,Y) = θ_Y and h(s,B) = θ_B directly — the current values are:

θ_Y = 1.0 θ_B = 0.5

Step 1 — Compute the current policy. Applying the softmax formula:

π(Y) = e^1.0 / (e^1.0 + e^0.5) = 2.7183 / (2.7183 + 1.6487) = 2.7183 / 4.3670 ≈ 0.6225

π(B) = e^0.5 / (e^1.0 + e^0.5) = 1.6487 / 4.3670 ≈ 0.3775

Before this ball is bowled, the policy leans toward the yorker about 62% of the time and the bouncer about 38% of the time — not a coin flip, but not a certainty either, which is exactly the deliberate unpredictability policy gradient methods can represent.

Step 2 — Act, and observe a reward. The bowler samples from this distribution and bowls a Yorker. It is a good one: the batter can only jam down on it, producing a dot ball with a close shave at the stumps. The coach's reward design assigns R = +1 to an outcome this good.

Step 3 — Compute the gradient of the log-probability of the action taken. For a softmax policy over preferences, the gradient of log π(a) with respect to each preference has a clean closed form: it equals 1 − π(a) for the action actually taken, and −π(a') for every other action a'. Since the bowler chose Y:

∂ log π(Y) / ∂θ_Y = 1 − π(Y) = 1 − 0.6225 = 0.3775

∂ log π(Y) / ∂θ_B = −π(B) = −0.3775

Step 4 — Apply the REINFORCE update. With learning rate α = 0.1 and G = R = +1:

θ_Y ← 1.0 + 0.1 × 1 × 0.3775 = 1.03775

θ_B ← 0.5 + 0.1 × 1 × (−0.3775) = 0.46225

Step 5 — Recompute the policy. The preference gap between Y and B has widened slightly, from 0.5 to 0.5755, so softmax now favours the yorker a little more strongly:

π(Y) = e^1.03775 / (e^1.03775 + e^0.46225) ≈ 2.8229 / 4.4105 ≈ 0.6400

π(B) ≈ 0.3600

One gradient step, from a single successful ball, moved the probability of bowling a yorker in that situation from 62.25% to 64.00% — a nudge of about 1.75 percentage points. Had the same yorker instead been creamed for a boundary (R = −1), the arithmetic in Step 4 would flip sign: θ_Y would fall to 0.96225 and θ_B would rise to 0.53775, pulling π(Y) down to about 0.6046 instead of up. The direction of the nudge always tracks the sign of the reward; the size of the nudge always tracks how much the outcome could still surprise the current policy, through the 1 − π(a) and π(a') terms — an action the policy was already nearly certain about barely moves, while an action it was unsure about swings harder.

Checking the arithmetic in code. The same five steps, written as a short NumPy script, reproduce every number above exactly:

import numpy as np

theta = np.array([1.0, 0.5])          # [theta_Yorker, theta_Bouncer]

def policy(theta):
    exp_theta = np.exp(theta)
    return exp_theta / np.sum(exp_theta)

pi = policy(theta)
print(np.round(pi, 4))                # [0.6225 0.3775]

action, reward, alpha = 0, 1.0, 0.1   # action 0 = Yorker, dot ball -> reward +1

grad_log_pi = -pi.copy()
grad_log_pi[action] += 1.0            # 1 - pi[a] for the chosen action, -pi[a] elsewhere

theta = theta + alpha * reward * grad_log_pi
print(np.round(theta, 5))             # [1.03775 0.46225]
print(np.round(policy(theta), 4))     # [0.64 0.36]

Multiply this single ball's nudge across thousands of deliveries, across many overs and many matches, and the coach's intuition — vary the yorker and the bouncer, and lean harder into whatever has been working — emerges automatically from gradient ascent on J(θ), with no value table and no explicit model of the batter ever being built.

The Variance Problem, and a Fix Called the Baseline

REINFORCE is correct on average — its expected update really does point along ∇_θ J(θ) — but any single episode's return G_t is a noisy, high-variance sample of the true action-value Q^π(s_t, a_t). In cricket terms: a well-executed yorker can still occasionally be scooped for six by a batter improvising a reverse-ramp, and a slightly loose delivery can still occasionally beat the bat and clip the stumps. Using raw returns to scale every update means the policy sometimes takes a large step in the wrong direction purely because of a lucky or unlucky bounce, which makes learning slow and erratic.

The standard fix is to subtract a baseline b(s) from the return before using it to scale the gradient:

θ ← θ + α · (G_t − b(s_t)) · ∇_θ log π_θ(a_t|s_t)

A natural choice of baseline is the state-value function V(s) — an estimate of the return the policy expects to get from state s on average, regardless of which action is chosen. Subtracting it converts the learning signal from "how good was this outcome, in absolute terms" to "how much better or worse than expected was this outcome," precisely the quantity reinforcement learning calls the advantage, A(s,a) = Q(s,a) − V(s). A yorker that produces a dot ball when a dot ball was already the expected outcome should barely move the policy; a yorker that produces a wicket when the bowler was expected to leak four runs should move it a great deal.

Crucially, subtracting any baseline that depends only on the state and not the action leaves the gradient's expected value completely unchanged. This is because the term ∇_θ log π_θ(a|s) has zero expectation under the policy's own action distribution: E_{a~π_θ}[∇_θ log π_θ(a|s)] = Σ_a π_θ(a|s) ∇_θ log π_θ(a|s) = Σ_a ∇_θ π_θ(a|s) = ∇_θ Σ_a π_θ(a|s) = ∇_θ(1) = 0, since action probabilities in any state always sum to exactly one. Multiplying this zero-mean quantity by a baseline b(s) that does not depend on a keeps its expectation at zero — the baseline is "free," reducing variance without introducing any bias.

Learning V(s) alongside the policy, rather than throwing it away after a single use, is precisely the idea behind actor-critic methods, where the "actor" is the policy π_θ being trained with the gradient rule above, and the "critic" is a learned value function supplying the baseline, and in fuller versions, replacing the raw Monte Carlo return with a lower-variance bootstrapped estimate. That combination is the natural next step beyond REINFORCE, and it underlies most large-scale policy gradient methods used today.

Where Policy Gradients Matter in Practice

Three settings make the policy-gradient family the tool of choice rather than a value-based alternative. The first is continuous control — robotics, autonomous vehicles, industrial process control — where the action is a real-valued vector and there is no discrete menu to search over; a parameterized policy can output, say, the mean and standard deviation of a Gaussian distribution over torques or steering angles directly. The second is any setting that provably needs randomized behaviour to be optimal, which includes every adversarial or partially observable situation where a predictable pattern can be exploited — our death-over bowler is one small example; poker and other imperfect-information games are the classic large-scale ones.

The third, and the one most likely to touch you directly, is fine-tuning large language models with human feedback, usually called RLHF. After a language model is pretrained on text, it is further trained to prefer responses that human raters judge more helpful and reliable. Here the "state" is the conversation so far, the "action" is the next token or full response the model generates, and the "reward" comes from a separate model trained to predict human preference judgments. This is, precisely, a policy gradient problem — the language model's output distribution is the policy π_θ — and the workhorse algorithm for it is Proximal Policy Optimization (PPO), introduced by John Schulman and colleagues at OpenAI in 2017, which adds a safeguard against the policy changing too drastically in any single update. Every time an AI assistant's response feels more careful or better calibrated than raw next-word prediction alone would produce, a policy gradient update, run at enormous scale, is very likely part of the reason why.

Back to the Death Over

The coach in the opening scene never needed to build a lookup table of the value of a yorker, in every possible combination of over number, target score, and batter. All that was needed was a way to observe what happened after each ball, and a rule for nudging the probability of each delivery in the direction of what worked. That is the entire content of policy gradient methods, formalized: define a differentiable policy, roll it out, weight the log-probability of each action taken by how good its outcome turned out to be, and climb. REINFORCE is the simplest instance of this idea, and the direct ancestor of the actor-critic methods and PPO-style algorithms that now train robots to walk, agents to play games at a professional level, and large language models to hold a more useful conversation. Whenever the space of actions is too rich to tabulate, or the best strategy genuinely calls for a bowler — or a policy — that refuses to be predictable, this is the family of methods reinforcement learning reaches for.

Think About It

Think about this: How would you explain policy gradient methods in reinforcement learning to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind policy gradient methods in reinforcement learning, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.