PPO: Proximal Policy Optimization

In 2022, OpenAI turned GPT-3 — a model that could complete text but often ignored instructions, invented facts, or produced toxic completions — into InstructGPT, the model whose interface became ChatGPT. The last stage of that process was reinforcement learning: a reward model, trained on human preference rankings, scored the language model's outputs, and the language model's parameters were updated to raise the score. The algorithm that performed that update was PPO. Nothing about that choice was incidental. The "policy" being updated was a 175-billion-parameter network already fluent at generating language; if a single noisy batch of reward signal pushed it too far in one step, it could collapse into repeating a handful of high-reward phrases, or produce fluent-sounding garbage that "games" the reward model without actually being useful. Proximal Policy Optimization exists specifically to prevent that collapse — to let a policy learn from its own reward signal while guaranteeing that no single update moves it too far from where it stood a moment ago.

That is the entire idea condensed into one sentence, but the word "proximal" is doing real mathematical work, and unpacking it requires starting from the algorithm PPO was built to replace.

Where vanilla policy gradients break

A policy π_θ(a|s) is a probability distribution over actions given a state, parameterized by θ. The REINFORCE algorithm updates θ using the policy gradient theorem:

∇θ J(θ) = Et [ ∇θ log πθ(at|st) · At ]

where A_t is the advantage — how much better action a_t was than what the policy would typically do in state s_t (usually A_t = Q(s_t,a_t) − V(s_t), estimated in practice with Generalized Advantage Estimation, GAE-λ, from a value network's TD residuals). You sample a trajectory, compute this gradient, take a gradient-ascent step, and — critically — you must then throw the trajectory away. The gradient was derived under the assumption that the actions were sampled from the current π_θ; the moment θ changes, that assumption is stale, and reusing the same data for a second gradient step is invalid.

This makes REINFORCE brutally sample-inefficient, but the sharper problem is stability. In supervised learning, a bad step size just means slower or noisier convergence, because the training distribution is fixed. In reinforcement learning, the policy generates its own training data — a large step doesn't just move θ, it moves the distribution of states and actions the agent will *see next*. A badminton coach who reacts to one spectacular smash by telling a student to overhaul their entire swing risks producing worse shots for weeks, because the new form was never tested against anything except that one point. An RL policy that overreacts to one good trajectory can walk itself into a region of parameter space it can't recover from — the on-policy data it collects there is now generated by a broken policy, so there's no gradient signal pointing back to where it used to work. This is the "trust region" problem: policy space has cliffs that supervised-learning loss landscapes mostly don't, and gradient descent has no built-in notion of a safe step size.

TRPO's answer, and why PPO exists instead

Trust Region Policy Optimization (TRPO, Schulman et al., 2015) attacks this head-on: maximize the policy improvement subject to a hard constraint that the KL divergence between the new and old policy stays below a threshold δ. This works, but solving a KL-constrained optimization problem exactly requires the conjugate gradient method with Hessian-vector products against the Fisher information matrix — second-order machinery that's expensive per update and awkward to combine with the shared parameters and dropout/batchnorm layers common in modern networks.

PPO (Schulman et al., 2017) asks a narrower question: can we get most of TRPO's stability using only first-order gradients — the same machinery as REINFORCE — plus one cheap trick? The answer is the clipped surrogate objective, and it is the entire mechanism of the algorithm.

The importance-sampling ratio

Since PPO wants to reuse a batch of trajectories across multiple epochs of gradient updates, it needs to correct for the fact that, after the first update, the data was collected by a policy π_{θ_old} that no longer matches the current π_θ. Importance sampling supplies exactly this correction: define the probability ratio

rt(θ) = πθ(at|st) / πθ_old(at|st)

r_t = 1 means the current policy assigns this action the same probability the data-collecting policy did — no drift. r_t = 1.75 means the current policy has become 75% more likely to take that action than when the data was collected. The naive surrogate objective r_t(θ)·A_t, maximized directly, reduces exactly to the policy gradient when r_t ≈ 1 — but importance sampling has a well-known failure mode: when the two distributions diverge, the ratio can blow up and the gradient estimate becomes high-variance or actively misleading. Left unconstrained, maximizing r_t·A_t pushes θ to make r_t arbitrarily large for actions with positive advantage — precisely the destructive over-step TRPO's KL constraint was built to prevent.

The clipped surrogate objective

PPO's fix is to cap the reward the objective gives for pushing r_t away from 1, using:

LCLIP(θ) = Et [ min( rt(θ)·At,  clip(rt(θ), 1−ε, 1+ε)·At ) ]

with ε typically 0.1–0.2. Read this term by term: the first argument to min is the plain, unclipped importance-weighted advantage. The second clamps the ratio to the band [1−ε, 1+ε] before multiplying by A_t. Taking the min of the two — not the clipped term alone — is the entire trick, and it produces an asymmetric effect that most students get backwards on first read.

Walk through what happens in each of the two advantage-sign cases, holding A_t fixed and letting r_t vary:

A_t > 0 (the action was better than expected — you want to encourage it). For r_t ≤ 1+ε, clip(r_t) = r_t, so both arguments to min are identical and the objective is the plain linear term r_t·A_t. Once r_t exceeds 1+ε, the clipped term freezes at (1+ε)·A_t while the unclipped term keeps growing — so min now always selects the frozen, smaller value. The objective goes flat: increasing r_t further earns no additional reward, so the gradient with respect to θ through this sample becomes zero. Below r_t = 1−ε, nothing is clipped at all — the objective still falls linearly as r_t shrinks, so there's still full gradient pressure pushing the probability of a good action back up if it's being suppressed.

A_t < 0 (the action was worse than expected — you want to discourage it). The picture flips. For r_t ≥ 1−ε the objective is unclipped and linear, decreasing (more negative) without bound as r_t grows — so if the update is making a bad action more likely, the penalty keeps intensifying no matter how far r_t drifts above 1+ε. Below r_t = 1−ε, the clipped term freezes at (1−ε)·A_t, which is less negative than the still-falling unclipped term, so min selects the frozen value: the objective goes flat, and there's no further reward for suppressing that action's probability past the boundary.

The pattern behind both cases: clipping only ever removes the incentive to move the ratio further in the direction that would *already* count as improvement — it never removes the penalty for moving in the direction that would make things worse. That asymmetry is what makes the clip a genuine (if approximate) trust-region device rather than just a magnitude cap.

Worked example: computing L^CLIP by hand

Suppose a rollout produced three (state, action) samples, and GAE has already estimated an advantage for each. The old policy (the one that generated the data) and the current policy (mid-update) assign these probabilities to the actions actually taken:

With ε = 0.2, the clip band is [0.8, 1.2]. Compute r_t = π_θ/π_old for each: r₁ = 0.70/0.40 = 1.75, r₂ = 0.20/0.30 = 0.6667, r₃ = 0.90/0.70 = 1.2857. All three fall outside the clip band — sample 1 and 3 above 1.2, sample 2 below 0.8. That code below computes both surrogate terms and the min for each:

import numpy as np

old_probs   = np.array([0.40, 0.30, 0.70])
new_probs   = np.array([0.70, 0.20, 0.90])
advantages  = np.array([2.0, -1.5, 0.5])
epsilon     = 0.2

ratio         = new_probs / old_probs
clipped_ratio = np.clip(ratio, 1 - epsilon, 1 + epsilon)

surr1 = ratio * advantages
surr2 = clipped_ratio * advantages
L_clip_per_sample = np.minimum(surr1, surr2)

print("ratio:          ", np.round(ratio, 4))
print("clipped_ratio:  ", np.round(clipped_ratio, 4))
print("surr1 (raw):    ", np.round(surr1, 4))
print("surr2 (clipped):", np.round(surr2, 4))
print("per-sample L:   ", np.round(L_clip_per_sample, 4))
print("L_CLIP (mean):  ", np.round(np.mean(L_clip_per_sample), 4))

Tracing it by hand confirms every line: surr1 = [1.75×2.0, 0.6667×−1.5, 1.2857×0.5] = [3.5, −1.0, 0.6429]. surr2 = [1.2×2.0, 0.8×−1.5, 1.2×0.5] = [2.4, −1.2, 0.6]. Taking the elementwise minimum: min(3.5, 2.4) = 2.4, min(−1.0, −1.2) = −1.2, min(0.6429, 0.6) = 0.6. The printed output is:

ratio:           [1.75   0.6667 1.2857]
clipped_ratio:   [1.2    0.8    1.2   ]
surr1 (raw):     [3.5    -1.0   0.6429]
surr2 (clipped): [2.4    -1.2   0.6   ]
per-sample L:    [2.4    -1.2   0.6   ]
L_CLIP (mean):   0.6

Notice that in all three cases the clipped term was selected, and each one matches the pattern derived above: sample 1 has A > 0 and r > 1+ε — flattened, exactly as predicted. Sample 2 has A < 0 and r < 1−ε — flattened. Sample 3 has A > 0 and r > 1+ε — flattened. None of these three samples happened to land in the "still linear, unbounded penalty" regime, but that's a property of this particular batch, not of the algorithm — swap sample 2's advantage to +1.5 instead of −1.5 and its r = 0.6667 (below 1−ε, but now with positive advantage) would select the unclipped term: 0.6667×1.5 = 1.0 versus clipped 0.8×1.5=1.2, and min picks 1.0, the unclipped value, since here A > 0 and r < 1−ε is the "still-linear" regime.

The full training objective and loop

L^CLIP alone is only the policy term. Because PPO is typically implemented as an actor-critic method sharing a network trunk between the policy head and a value-function head, the actual loss combines three pieces — the clipped policy loss, a value-function regression loss, and an entropy bonus that keeps the policy from collapsing to a deterministic one too early and losing exploration:

L(θ) = LCLIP(θ) − c1·LVF(θ) + c2·S[πθ]

where L^VF = (V_θ(s_t) − V_target)² and S is the policy's entropy, typical coefficients c₁ ≈ 0.5, c₂ ≈ 0.01. Wrapped in the outer training loop, one PPO update cycle looks like this (PyTorch-style; rollout, compute_gae, and minibatches are assumed helpers, not shown — they handle environment stepping, the GAE-λ advantage recursion, and shuffling the batch into minibatches respectively):

def ppo_update(policy, value_fn, optimizer, env,
                epsilon=0.2, epochs=4, batch_size=64):
    trajectories = rollout(env, policy, steps=2048)              # assumed helper, not shown
    advantages, returns = compute_gae(trajectories, value_fn)    # assumed helper, not shown
    old_log_probs = policy.log_prob(
        trajectories.states, trajectories.actions
    ).detach()

    for epoch in range(epochs):
        for batch in minibatches(
            trajectories, advantages, returns, old_log_probs, batch_size
        ):                                                       # assumed helper, not shown
            new_log_probs = policy.log_prob(batch.states, batch.actions)
            ratio = (new_log_probs - batch.old_log_probs).exp()
            clipped_ratio = ratio.clamp(1 - epsilon, 1 + epsilon)

            surr1 = ratio * batch.advantages
            surr2 = clipped_ratio * batch.advantages
            policy_loss = -torch.min(surr1, surr2).mean()

            value_loss = (value_fn(batch.states) - batch.returns).pow(2).mean()
            entropy_bonus = policy.entropy(batch.states).mean()

            loss = policy_loss + 0.5 * value_loss - 0.01 * entropy_bonus
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

Notice the ratio is computed as (new_log_probs - old_log_probs).exp() rather than dividing raw probabilities — subtracting log-probabilities and exponentiating is the numerically stable way to compute a ratio of probabilities that can be very small, which is exactly why real implementations store log-probabilities rather than probabilities in the first place; the hand-worked example above used raw probabilities only because the arithmetic stays exact and traceable. old_log_probs is computed once, outside the epoch loop, and held fixed across all four epochs — that fixed snapshot is precisely π_{θ_old}, and it only advances to match the current θ at the start of the *next* rollout, once fresh on-policy data has been collected.

The misconception that trips most students

The name "clipped" strongly suggests that PPO enforces a hard rule: after this update, no probability ratio in the batch will lie outside [1−ε, 1+ε]. This is false, and it's false in an important way. The clip operates only inside the loss computation, on the *value used to compute the gradient* for that sample — it is not a constraint applied to θ, to π_θ's output, or to the ratio itself. Nothing in the algorithm stops gradient descent from moving θ far enough that r_t ends up at 1.6 or 0.3 after the step. In fact, across the four-to-ten epochs a real PPO run takes per batch, it is completely ordinary for a meaningful fraction of samples to drift outside the clip band by the final epoch — that's not a bug to fix, it's the expected behavior of a first-order approximation to a trust region rather than an enforced one. What the clip actually guarantees is narrower and more precise: it guarantees that, once a sample's ratio has moved past the boundary in the *rewarding* direction (further increasing an already-good action's probability, or further decreasing an already-bad action's probability), that sample stops contributing gradient signal urging it to move even further that way. It never stops a sample from being told to move back — the correction pressure in the harmful direction is always live, unbounded, and immediate. Confusing "clipped objective" with "clipped ratio" leads students to expect PPO to behave like TRPO's hard KL wall, when it's actually closer to a soft speed limit that only applies going one direction.

Active recall

Attempt each question before reading its answer.

1. Sample: π_old = 0.5, π_θ = 0.9, A_t = +3, ε = 0.2. Compute r_t, the clipped ratio, and state which surrogate term (raw or clipped) the min operator selects.
2. Sample: π_old = 0.6, π_θ = 0.1, A_t = −4, ε = 0.1. Same three computations.
3. Why does PPO take min(surr1, surr2) instead of always training on the clipped term surr2 alone?
4. What specifically does "proximal" refer to, and how does PPO's mechanism for staying "proximal" differ from TRPO's?
5. True or false: after a PPO update finishes, every sample's ratio r_t lies inside [1−ε, 1+ε]. Justify your answer.
6. In the training loop above, old_log_probs is computed once before the epoch loop and reused across all four epochs of minibatch updates. What would break if it were instead recomputed with the current θ at the start of every single minibatch step?

Answers

1. r_t = 0.9/0.5 = 1.8. Clip band is [0.8, 1.2], so clipped ratio = 1.2 (clamped down from 1.8). surr1 = 1.8 × 3 = 5.4; surr2 = 1.2 × 3 = 3.6. min(5.4, 3.6) = 3.6 — the clipped term is selected. This matches the rule: A > 0 and r > 1+ε ⇒ flattened.

2. r_t = 0.1/0.6 = 0.1667. Clip band is [0.9, 1.1], so clipped ratio = 0.9 (clamped up from 0.1667). surr1 = 0.1667 × −4 = −0.6667; surr2 = 0.9 × −4 = −3.6. min(−0.6667, −3.6) = −3.6 — the clipped term is selected. This matches: A < 0 and r < 1−ε ⇒ flattened.

3. Using only surr2 would make the objective flat (zero gradient) in *both* directions past the clip boundary — including the direction where the policy is drifting toward a worse action or away from a better one, which is exactly the case where you most need gradient signal to correct it. Taking the min keeps the raw, unclipped term active whenever it is the smaller (more pessimistic) of the two, which is precisely the harmful-direction case; clipping alone would silence the correction along with the runaway reward.

4. "Proximal" means the updated policy π_θ is kept close to the data-collecting policy π_{θ_old}, so that the importance-sampling correction and the advantage estimates computed under π_{θ_old} remain approximately valid for π_θ. TRPO enforces this with a hard constraint — KL(π_old, π_new) ≤ δ — solved via second-order optimization. PPO enforces it only approximately and only in the reward-maximizing direction, via the clip term inside a plain first-order objective, trading a guarantee for cheap, simple SGD-based updates.

5. False. The clip changes the loss and hence the gradient, not the ratio itself. There is no mechanism preventing gradient descent from pushing r_t outside [1−ε, 1+ε]; it routinely happens, especially in later epochs on the same batch, once the flattened term stops contributing pressure to pull that sample back inside the band.

6. If old_log_probs were recomputed with the live θ at every minibatch step, the ratio r_t would always equal exactly 1 (new and old would be the same network), making surr1 = surr2 = A_t for every sample — the clip would never trigger, the algorithm would degrade to vanilla policy-gradient-with-a-value-baseline, and it would lose the entire mechanism that lets it reuse one rollout across several epochs safely.

Think About It

Think about this: How would you explain ppo: proximal policy optimization to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind ppo: proximal policy optimization, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.