Policy gradient methods have a seductive simplicity. Collect trajectories, estimate which actions helped, and shift the policy toward those actions. The problem is that the same simplicity makes them fragile. One bad update can move the policy so far that the data it collects next is useless, and recovery takes a long time. Proximal Policy Optimization, or PPO, is the practical answer to that fragility. It does not introduce a new learning objective from first principles. It takes the ordinary policy gradient objective, adds a cheap guardrail that stops the policy from moving too far in a single step, and thereby turns a method that often diverges into one that reliably improves.
That guardrail is why PPO became the default algorithm for fine-tuning language models with human feedback, for training game agents, and for many robotics tasks. It is not the most sample-efficient method, and it is not the most theoretically elegant. It is the method that works across a wide range of problems without careful per-problem tuning, and that robustness is worth understanding in detail.
The policy gradient starting point
Consider a policy πθ(a∣s) parameterised by θ , which gives the probability of taking action a in state s . The goal is to maximise expected return
where γ is the discount factor and rt is the reward at step t . The policy gradient theorem says the gradient of this objective can be written as an expectation over states and actions visited by the current policy:
Here A(s,a) is the advantage: how much better action a is than the average action in state s . In practice the advantage is estimated from returns, often with a baseline or with generalised advantage estimation, which trades bias against variance. The key point for PPO is what happens when you try to reuse data.
If you collect data with an old policy πθold and then update to a new policy πθ , you can correct for the mismatch with importance sampling. Define the probability ratio
Then a surrogate objective that approximates the true objective near the old policy is
CPI stands for conservative policy iteration. Maximising this surrogate is equivalent to the policy gradient step when θ=θold , because then rt(θ)=1 . The trouble starts when you optimise it for several steps on the same batch of data. The ratio can drift far from 1, the surrogate stops approximating the true objective, and the update can destroy the policy. A single large ratio on a high-advantage action can dominate the gradient, pushing the policy to take that action almost always, even if the advantage estimate was noisy.
Trust regions, without the second-order machinery
The principled fix is a trust region: only allow the new policy to move a limited distance from the old policy. Trust Region Policy Optimization, or TRPO, enforces this with a hard constraint on the KL divergence between the old and new policies, and solves the constrained problem with second-order methods involving conjugate gradients and line search. TRPO works well, but it is complicated to implement, sensitive to implementation details, and awkward to combine with architectures that use dropout, parameter sharing, or recurrent layers.
PPO keeps the trust region idea and drops the machinery. Instead of a hard KL constraint, it uses a clipped surrogate objective that penalises moving the ratio too far, but only when that movement would improve the surrogate. The clipped objective is
where ϵ is a small constant, typically 0.2, and clip forces the ratio into the interval [1−ϵ,1+ϵ] .
The minimum is doing the real work, and it behaves differently depending on the sign of the advantage:
- If At>0 , the action was good, so increasing its probability is desirable. But once rt(θ) exceeds 1+ϵ , the clipped term stops growing, and the minimum picks the clipped term. The gradient with respect to that sample becomes zero. The policy is not pushed further in that direction.
- If At<0 , the action was bad, so decreasing its probability is desirable. But once rt(θ) falls below 1−ϵ , the clipped term again caps the gain, and the gradient becomes zero.
- If the ratio moves in the wrong direction, for example the ratio increases for a bad action, the unclipped term is smaller than the clipped term, so the minimum picks the unclipped term and the gradient still pushes the policy back. Clipping never prevents correction, only excessive improvement.
In short, clipping creates a flat region beyond the trust region boundary for moves that would otherwise be rewarded, while leaving the gradient intact for moves that undo a mistake. That asymmetry is the whole trick.
A useful diagnostic is the fraction of samples that are clipped in a given epoch. If almost nothing is clipped, the updates are small and you could afford a larger step or more epochs. If a large fraction is clipped, the policy is trying to move far, and the clip is doing heavy work. Monitoring that fraction, together with the KL divergence between the old and new policies, tells you whether the trust region is binding in practice.
Why multiple epochs on the same data become safe
Vanilla policy gradients are on-policy: each gradient step needs fresh data, because the gradient formula assumes data came from the current policy. That is sample-inefficient. PPO reuses each batch for several epochs of minibatch updates, which would be dangerous with the unclipped surrogate, because repeated steps on the same batch can compound into a large move. With clipping, the first epoch may move the ratio to the boundary, but later epochs on the same samples contribute no gradient once they are clipped, so the total move stays bounded.
This is what makes PPO practical. You collect a batch of trajectories, compute advantages once, and then run several epochs of stochastic gradient ascent on the clipped objective, plus a value function loss and optionally an entropy bonus. The standard combined loss, written as a loss to minimise, is
where LVF is the squared-error loss for the value function, S is the entropy of the policy, and c1 and c2 are small coefficients, commonly 0.5 and 0.01. The entropy term encourages exploration by penalising a policy that collapses to a single action too early. The value loss is usually clipped as well in modern implementations, so that the value function does not move too far in one update either.
A minimal sketch of the PPO loop looks like this:
for iteration in range(num_iterations): trajectories = collect(policy_old, num_steps) advantages = compute_gae(trajectories) # generalised advantage estimation returns = advantages + values for epoch in range(num_epochs): for minibatch in minibatches(trajectories): ratio = policy.prob(minibatch) / policy_old.prob(minibatch) clipped = clip(ratio, 1 - eps, 1 + eps) policy_loss = -mean(min(ratio * advantages, clipped * advantages)) value_loss = mean((value(minibatch) - returns) ** 2) entropy = policy.entropy(minibatch) loss = policy_loss + 0.5 * value_loss - 0.01 * entropy optimiser.step(loss) policy_old = policy The code is deliberately simple. There is no conjugate gradient solver, no line search, and no Fisher matrix. That simplicity is a feature: it means PPO can be implemented in a few dozen lines on top of a standard deep learning library, and it composes cleanly with shared actor-critic networks, recurrent policies, and distributed data collection.
Advantage estimation matters as much as clipping
Clipping bounds the policy move, but it cannot fix a bad advantage estimate. If advantages are noisy, PPO will confidently move in the wrong direction, just not very far in one step. Generalised advantage estimation, or GAE, is therefore part of the standard PPO recipe, not an optional extra.
GAE builds the advantage from a weighted sum of temporal difference errors. Let δt=rt+γV(st+1)−V(st) be the one-step temporal difference error, where V is the value function. Then the GAE advantage with parameter λ is
Related posts
Our Guard Checked That We Still Talked About the Fix, Not That We Still Did It
