Skip to main content
The default GRPO run of Chapter 9 succeeded, and its logs contain two numbers that a post-training engineer would ask about before calling it healthy. The entropy of the policy tokens fell from 0.57 to 0.08 nats in 80 steps, and by the end four of every five groups had identical rewards and carried no gradient. The policy had become nearly deterministic. On this gym that is what success looks like, because each task has one right procedure. On a real coding distribution, a policy that has stopped exploring cannot discover a better procedure later, and a collapsed sampler makes pass@kk no better than pass@1. The opposite worry is also real: an RL objective with nothing holding the policy near its starting point is free to drift into regions where the reward is wrong, the language degrades, or skills that the reward does not measure are forgotten. This chapter studies the two standard regularizers that address these worries. A KL penalty pulls the policy toward a frozen reference policy, usually the starting checkpoint. An entropy bonus pushes it away from determinism. Both look like one extra line in a loss, and both hide subtleties that change the gradient a trainer actually computes: the three common KL estimators produce three different gradients when used as losses, one of them zero in expectation and one the gradient of a different divergence; putting the KL in the reward and putting it in the loss differ for sequences; and an entropy bonus over a full vocabulary pushes probability onto tokens the policy should never emit. We derive each fact, check it numerically on distributions small enough to enumerate, then compare the regularizers on TinyAgent: no regularizer, KL penalties of several strengths and placements, entropy bonuses, and a higher temperature, measured by entropy, held-out pass rate, and pass@kk; clip-higher gets a one-batch experiment of its own. Two functions are saved, kl_estimate and entropy_bonus, together with the update, training loop, and sequence-KL helper that use them.
The TinyAgent experiments use the book’s default setting from Chapter 9: the anonymized one-edit gym, the RL base agent, and the training and held-out splits.

​
10.1 The reference policy

The reference policy πref\pi_{\text{ref}} is a fixed policy, in practice a frozen copy of the model at the start of RL (the SFT or base checkpoint). The KL-regularized objective is Jβ(θ)=Eτ∼πθ[R(τ)]−β KL(πθ ∥ πref),J_\beta(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\big[R(\tau)\big] - \beta\, \mathrm{KL}\big(\pi_\theta \,\|\, \pi_{\text{ref}}\big), with the sequence KL decomposed into per-token terms along the policy’s own prefixes (Appendix A.6). Three reasons for it recur in the literature, and they are different reasons. First, in RLHF the reward is a learned model that is accurate only near the data it was trained on; the KL keeps the policy where the reward can be trusted. Second, the reference carries capabilities the reward does not measure (fluency, formatting, knowledge), and the KL limits how much of them the policy may trade away. Third, the KL is a cumulative trust region: PPO’s clip (Chapter 8) limits how far one update moves from πold\pi_{\text{old}}, while the KL limits how far the whole run moves from πref\pi_{\text{ref}}. With a verifiable reward such as hidden tests, the first reason disappears, which is the core of the case for dropping the KL (Section 10.7). What does the regularized objective want? For a single decision (a bandit) the answer is in closed form. Write π\pi for a distribution over KK actions with rewards r(a)r(a). Then Eπ[r]−β KL(π ∥ πref)=−β∑aπ(a)log⁡π(a)πref(a) er(a)/β=βlog⁡Z−β KL(π ∥ π∗),\mathbb{E}_\pi[r] - \beta\,\mathrm{KL}(\pi \,\|\, \pi_{\text{ref}}) = -\beta \sum_a \pi(a) \log \frac{\pi(a)}{\pi_{\text{ref}}(a)\, e^{r(a)/\beta}} = \beta \log Z - \beta\, \mathrm{KL}(\pi \,\|\, \pi^*), where π∗(a)=πref(a) er(a)/β/Z\pi^*(a) = \pi_{\text{ref}}(a)\, e^{r(a)/\beta} / Z and Z=∑aπref(a) er(a)/βZ = \sum_a \pi_{\text{ref}}(a)\, e^{r(a)/\beta}. Since the KL is nonnegative and zero only at π=π∗\pi = \pi^*, the optimum is π∗\pi^* and the optimal value is βlog⁡Z\beta \log Z. The optimum tilts the reference by the exponentiated reward: β\beta is a temperature that interpolates between πref\pi_{\text{ref}} (β→∞\beta \to \infty) and the best action in the reference’s support (β→0\beta \to 0). An action the reference never takes stays at probability zero for every β\beta. The same derivation with πref\pi_{\text{ref}} uniform gives the entropy bonus, because KL(π ∥ Uniform)=log⁡K−H(π)\mathrm{KL}(\pi \,\|\, \mathrm{Uniform}) = \log K - H(\pi): maximizing E[r]+αH(π)\mathbb{E}[r] + \alpha H(\pi) is a KL penalty toward the uniform distribution, with optimum softmax(r/α)\mathrm{softmax}(r/\alpha). Chapter 15 uses the same closed form to derive DPO.
The bandit below has five actions. Action 4 has the highest reward, but the reference never takes it; actions 0 and 1 are good, and the reference prefers the weaker one.
Output:
At β=10\beta = 10 the optimum is the reference, with KL 0.001. As β\beta falls, mass moves first to action 1, which the reference already likes, and then to action 0, the better action the reference rarely takes; at β=0.01\beta = 0.01 the policy is deterministic on action 0 with expected reward 1.0 and entropy 0. The KL saturates at log⁡(1/0.05)=2.996\log(1/0.05) = 2.996, the cost of putting all mass on an action the reference takes 5% of the time. Action 4, worth 1.2, never receives any mass. The entropy bonus with α=0.3\alpha = 0.3 behaves differently: its implicit reference is uniform, so it puts 0.59 on action 4 and reaches a higher expected reward, 1.065, than any KL setting. In a language model the “actions outside the reference’s support” are mostly garbage tokens, and the uniform reference’s indifference to them is the entropy bonus’s main hazard (Section 10.6). We check the closed form against random perturbations of π∗\pi^*:
Output:
The value at π∗\pi^* equals βlog⁡Z\beta \log Z, and no perturbation does better, as the identity above requires.

​
10.2 Estimating KL from samples

The trainer never sees full distributions over sequences. It sees sampled tokens y∼πθ(⋅∣c)y \sim \pi_\theta(\cdot \mid c) and two numbers per token, log⁡πθ(y∣c)\log \pi_\theta(y \mid c) and log⁡πref(y∣c)\log \pi_{\text{ref}}(y \mid c). Write their difference as ℓ=log⁡πref(y∣c)−log⁡πθ(y∣c)\ell = \log \pi_{\text{ref}}(y \mid c) - \log \pi_\theta(y \mid c). John Schulman’s note Approximating KL Divergence (2020) compares three per-sample estimators of KL(πθ ∥ πref)\mathrm{KL}(\pi_\theta \,\|\, \pi_{\text{ref}}): k1=−ℓ,k2=12ℓ2,k3=eℓ−1−ℓ.k_1 = -\ell, \qquad k_2 = \tfrac{1}{2}\ell^2, \qquad k_3 = e^{\ell} - 1 - \ell . k1k_1 is unbiased by definition and takes negative values on many samples. k3k_3 adds eℓ−1e^\ell - 1, which has mean zero under πθ\pi_\theta because Eπθ[πref/πθ]=1\mathbb{E}_{\pi_\theta}[\pi_{\text{ref}}/\pi_\theta] = 1; it is k1k_1 with a control variate (Appendix A.3), unbiased and never negative since eℓ≥1+ℓe^\ell \ge 1 + \ell. k2k_2 is biased: a Taylor expansion shows that it matches the KL to second order in ℓ\ell and overestimates beyond. Chapter 9 saved all three as kl_penalty. Appendix A.7 sampled them at two distances; here we compute their exact mean and per-sample standard deviation by enumerating every outcome of a 50-way distribution, at four distances between policy and reference.
Output:
The means are exact, so k1k_1 and k3k_3 show a ratio of 1.000 at every distance, and k2k_2‘s bias grows with the distance: 1% at KL 0.001, 41% at KL 0.32, and a factor of 2 at KL 1.08. The spread tells the other half. At KL 0.001, typical of a per-token KL between adjacent checkpoints, one k1k_1 sample has a standard deviation 45 times the quantity it estimates, against 1.25 times for k3k_3: averaging k1k_1 to a given precision takes about (44.84/1.25)2≈1300(44.84/1.25)^2 \approx 1300 times as many tokens. The reason is that k1k_1 is first order in ℓ\ell, so its spread scales like ∣ℓ∣|\ell|, while the KL itself is second order. The advantage of k3k_3 shrinks as the distributions separate, and at KL 1.08 its spread is comparable to k1k_1‘s. For logging a KL, k3k_3 is the right default: unbiased, nonnegative, and low-variance in the regime where training lives. Using an estimator as a loss is a separate question.

​
10.3 The gradient each estimator produces as a loss

A KL term in the loss is implemented by evaluating an estimator on the batch’s sampled tokens and adding β\beta times its mean to the loss. Autograd then differentiates the estimator through log⁡πθ(y∣c)\log \pi_\theta(y \mid c), with the sampled tokens held fixed. It does not differentiate through the fact that the tokens were sampled from πθ\pi_\theta. The expected gradient of such a loss is therefore Ey∼πθ[∇θk(y)]\mathbb{E}_{y \sim \pi_\theta}[\nabla_\theta k(y)], which is not ∇θEy∼πθ[k(y)]\nabla_\theta \mathbb{E}_{y\sim\pi_\theta}[k(y)]: the score-function term E[k(y) ∇θlog⁡πθ(y)]\mathbb{E}[k(y)\, \nabla_\theta \log \pi_\theta(y)] of Chapter 6 is missing. Compute each one, with ∇ℓ=−∇log⁡πθ(y)\nabla \ell = -\nabla \log \pi_\theta(y) and Eπθ[∇log⁡πθ]=0\mathbb{E}_{\pi_\theta}[\nabla \log \pi_\theta] = 0. For k1=−ℓk_1 = -\ell, the per-sample gradient is ∇log⁡πθ(y)\nabla \log \pi_\theta(y), whose expectation is zero: a k1k_1 loss contributes only noise. For k2=ℓ2/2k_2 = \ell^2/2, the per-sample gradient is −ℓ ∇log⁡πθ(y)=(log⁡πθ(y)−log⁡πref(y))∇log⁡πθ(y)-\ell\, \nabla \log\pi_\theta(y) = \big(\log\pi_\theta(y) - \log\pi_{\text{ref}}(y)\big)\nabla\log\pi_\theta(y), and its expectation is the true gradient of the reverse KL, ∇θKL(πθ ∥ πref)=∑y∇πθ(y)(log⁡πθ(y)−log⁡πref(y))+∑yπθ(y) ∇log⁡πθ(y)=Eπθ[(log⁡πθ−log⁡πref) ∇log⁡πθ].\nabla_\theta \mathrm{KL}(\pi_\theta \,\|\, \pi_{\text{ref}}) = \sum_y \nabla \pi_\theta(y) \big(\log \pi_\theta(y) - \log \pi_{\text{ref}}(y)\big) + \sum_y \pi_\theta(y)\, \nabla \log \pi_\theta(y) = \mathbb{E}_{\pi_\theta}\big[(\log \pi_\theta - \log \pi_{\text{ref}})\, \nabla \log \pi_\theta\big]. The biased estimator gives the unbiased gradient. For k3=eℓ−1−ℓk_3 = e^\ell - 1 - \ell, the per-sample gradient is (1−eℓ)∇log⁡πθ(y)(1 - e^{\ell})\nabla\log\pi_\theta(y), and Eπθ[(1−πrefπθ)∇log⁡πθ]=0−∑yπref(y) ∇log⁡πθ(y)=∇θ KL(πref ∥ πθ).\mathbb{E}_{\pi_\theta}\big[(1 - \tfrac{\pi_{\text{ref}}}{\pi_\theta})\nabla \log \pi_\theta\big] = 0 - \sum_y \pi_{\text{ref}}(y)\, \nabla \log \pi_\theta(y) = \nabla_\theta\, \mathrm{KL}(\pi_{\text{ref}} \,\|\, \pi_\theta). The unbiased, low-variance estimator gives the gradient of the forward KL, the other direction. Near πθ=πref\pi_\theta = \pi_{\text{ref}} the two directions agree to second order, so their gradients agree to first order, and the difference grows with the distance. We check all of this exactly by weighting each outcome by its (detached) sampling probability, which is the expectation a large batch of samples would approach.
The saved function is a thin layer over Chapter 9’s kl_penalty, which it calls for the values. It adds one thing: kinds ending in + use the straight-through construction g - g.detach() + v.detach(), whose forward value is the estimator vv (so the logged number is, say, k3k_3) and whose gradient is that of g=k2g = k_2 (so the loss pushes along the reverse-KL gradient). verl implements the same option under the same + suffix (core_algos.kl_penalty, commit 00094bd, inspected 2026-10-13), with a comment that k1k_1 and k3k_3 have the right expected value but not the right expected gradient.
Output:
Each line is the derivation in numbers. The expected k1k_1 gradient is zero, so its relative error against either target is exactly 1. The k2k_2 gradient matches the reverse-KL gradient to every printed digit, and the k3k_3 gradient matches the forward-KL gradient while missing the reverse one by 50%. The straight-through k3+ has k2k_2‘s gradient exactly. How large is k3k_3‘s error at the distances training actually reaches?
Output:
At a per-token KL of 0.002, the k3k_3 loss is within 3.4% of the reverse-KL gradient, and at 0.06 it is 17% off. Chapter 9’s GRPO preset averaged a per-token k3k_3 of 0.17 against its reference, between the second and third rows, where this table puts the error at 17% to 34%. In that regime, a “reverse-KL penalty” implemented as a k3k_3 loss, as in DeepSeekMath’s GRPO and verl’s default low_var_kl, is closer to a forward-KL penalty. Section 10.5 shows why the direction matters, and Section 10.7 measures the difference on TinyAgent. Figure 10.1: KL estimators as the policy moves away from the reference, computed exactly on a 50-outcome distribution. Left: relative bias of each estimator's value. Middle: per-sample standard deviation relative to the true KL (log scale). Right: relative error of each estimator's expected loss gradient against the true reverse-KL gradient. Figure 10.1 puts the three properties side by side over two decades of true KL. Read the panels from left to right as three different questions. Is the logged value right? k1k_1 and k3k_3, yes. Is one sample informative? k2k_2 and k3k_3, yes, by an order of magnitude at small KL. Does the loss push in the right direction? Only k2k_2 (and therefore k3+), at any distance. No single estimator wins all three, which is why the straight-through combination exists.
Pitfall. Rows of a batch in a PPO-style update come from πold\pi_{\text{old}}, not from the current πθ\pi_\theta, after the first minibatch. The derivations above assume on-policy samples. With the small ratios of Chapter 9’s default (clip fraction about 0.1%), this is a minor additional bias; with many epochs it is another reason the KL term’s gradient is only approximately what its name says.

​
10.4 KL in the reward versus KL in the loss

There are two places to put the penalty. KL in the reward subtracts β\beta times a per-token k1k_1 (the log-ratio) from the reward and lets the policy gradient handle it, as RLHF-style PPO does. KL in the loss adds β\beta times a per-token estimator to the loss, as GRPO does. For one token, the reward version is REINFORCE on the reward −k1-k_1, whose gradient is −E[k1∇log⁡πθ]-\mathbb{E}[k_1 \nabla\log\pi_\theta] ; this is exactly the negative reverse-KL gradient derived above, so for a single token, ”k1k_1 in the reward” and ”k2k_2 in the loss” agree. For sequences they do not. By Appendix A.6, the sequence KL is E[∑tKLt(ct)]\mathbb{E}\big[\sum_t \mathrm{KL}_t(c_t)\big], where KLt(ct)\mathrm{KL}_t(c_t) is the per-token KL at prefix ctc_t. Its gradient has two parts: how each KLt\mathrm{KL}_t changes with the parameters at a fixed prefix, and how the choice of token yty_t changes which prefixes, and therefore which future KLs, are visited. A per-token loss captures only the first part, because each token’s loss term is a function of that token’s own log-probability. The policy gradient on the reward captures both, because each token’s log-probability is weighted by the KL of everything that follows it (the reward-to-go of Section 6.5). The difference is credit for future divergence: a token that steers the episode into a region where the policy and reference disagree is penalized by the reward version and not by the loss version. We check this on a two-token sequence model with a vocabulary of 4, small enough to enumerate all 16 sequences: the first token has its own logits, and the second token’s logits depend on the first.
surrogate builds the expected loss that autograd would see for each placement. For the loss placements, each sequence contributes the estimator at both tokens. For the reward placement, each token’s log-probability is multiplied by the detached sum of k1k_1 from that token to the end.
Output:
The reward placement reproduces the exact gradient of the sequence KL for both tokens. The k2k_2 loss is exact for the second token, the last one, where there is no future, and 68% wrong for the first token, whose effect on the second token’s KL it ignores. k3k_3 is wrong on both, and k1k_1 is zero on both. The per-token k2k_2 loss is the exact gradient of a different objective: ∑tKLt\sum_t \mathrm{KL}_t with the distribution over prefixes held fixed. Which placement is better is a design question rather than a correctness one. The reward placement penalizes the right quantity, but it enters the advantage and so inherits the policy gradient’s variance; with GRPO’s group normalization, it is also rescaled by each group’s reward spread, so its strength varies from task to task. The loss placement is a low-variance, local regularizer that keeps every token’s distribution near the reference without accounting for where the episode goes. verl exposes both: algorithm.use_kl_in_reward with a k1k_1 penalty by default and an optional adaptive controller, and actor.use_kl_loss with kl_loss_type: low_var_kl (k3k_3, clamped to [−10,10][-10, 10]) and a default coefficient of 0.001 (commit 00094bd, inspected 2026-10-13). For agents, the future-divergence term matters more than for single-turn tasks, because the tokens of an early call decide which observations, and therefore which later contexts, the episode reaches.

​
10.5 Forward versus reverse KL

The reverse KL KL(πθ ∥ πref)=Eπθ[log⁡πθ−log⁡πref]\mathrm{KL}(\pi_\theta \,\|\, \pi_{\text{ref}}) = \mathbb{E}_{\pi_\theta}[\log \pi_\theta - \log \pi_{\text{ref}}] is an expectation under the policy. It is large wherever the policy puts mass that the reference does not, and indifferent to reference mass that the policy ignores. The forward KL KL(πref ∥ πθ)\mathrm{KL}(\pi_{\text{ref}} \,\|\, \pi_\theta) is an expectation under the reference, and it is large wherever the reference has mass that the policy lacks. The first is called mode-seeking (or zero-forcing), the second mass-covering. Maximum likelihood and SFT minimize a forward KL from the data; on-policy distillation in Chapter 16 minimizes a reverse KL to the teacher. The difference is clearest when a simple family must approximate a distribution it cannot represent. We fit one discretized Gaussian to a mixture of two by gradient descent in each direction.
Output:
The forward fit centers between the modes with a standard deviation of 2.84, splitting its mass evenly and putting 30% of it in the valley, where the target has 0.8%. The reverse fit sits exactly on the right mode (mean 2.50, standard deviation 0.60), puts nothing on the left, and reaches a KL of 0.693, which is log⁡2\log 2: the cost of covering one of two equal modes perfectly. Which mode it picks depends on the initialization (we started it at 1.0, to the right of center). Figure 10.2: A single Gaussian fitted to a bimodal target on a 121-point grid. The forward-KL fit spreads across both modes and the valley; the reverse-KL fit commits to one mode. Figure 10.2 shows the two fits over the target. For RL the consequences are direct. A reverse-KL penalty to πref\pi_{\text{ref}} lets the policy abandon reference modes (strategies the starting model sometimes used) at no cost, so it does not by itself preserve diversity, and it forbids mass where the reference has none, as the bandit’s action 4 showed. A forward-KL pull, which is what a k3k_3 loss implements at moderate distances (Section 10.3), penalizes abandoning reference modes and so acts more like a diversity-preserving regularizer. That is a side effect of an estimator’s gradient, not a design choice anyone wrote down, and it is worth knowing which one a configuration actually computes.

​
10.6 Entropy collapse and the levers against it

Entropy collapse is the steady fall of the policy’s per-token entropy during RL, toward a policy that samples the same tokens every time. It is not a bug in any one component: it is what a policy gradient does to a softmax policy when its high-probability choices are rewarded. The mechanism fits in one line. For logits zz over KK tokens with π=softmax(z)\pi = \mathrm{softmax}(z) and H=−∑jπjlog⁡πjH = -\sum_j \pi_j \log \pi_j, differentiating gives ∂H/∂zj=−πj(log⁡πj+H)\partial H / \partial z_j = -\pi_j(\log \pi_j + H), so a small logit change Δz\Delta z changes the entropy by ΔH≈−∑jπj (log⁡πj+H) Δzj=−Covj∼π(log⁡πj, Δzj),\Delta H \approx -\sum_j \pi_j\, (\log \pi_j + H)\, \Delta z_j = -\mathrm{Cov}_{j \sim \pi}\big(\log \pi_j,\ \Delta z_j\big), using Eπ[log⁡π]=−H\mathbb{E}_\pi[\log \pi] = -H. Entropy falls when the update raises the logits of tokens that are already likely, and rises when it raises the logits of unlikely ones. Cui et al., The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (2025-05-28), derive this covariance form for policy-gradient and natural-gradient updates and report that, without intervention, entropy and validation performance trade off along a fitted curve. One policy-gradient step on a sampled token aa with advantage AA changes the logits by Δz=ηA(ea−π)\Delta z = \eta A (e_a - \pi). Four cases cover the possibilities.
Output:
The first-order prediction matches the actual change to within 1% in every case, and the signs follow the covariance. Rewarding the likely token (probability 0.647) lowers the entropy; rewarding the rare token (probability 0.0044) raises it; penalizing either does the reverse. Now consider a GRPO batch late in training. Within a group, the successful attempts are mostly made of the policy’s likely tokens, so most positive-advantage updates are of the first kind. The failures are made of a mix, and penalizing a rare token (the fourth row) also lowers entropy. Three of the four cases reduce entropy whenever the policy is already right more often than not, which is why collapse is the default dynamics of on-policy RL, not an accident of one run. The levers against collapse act on this covariance in different ways.
  • An entropy bonus adds −αH-\alpha H to the loss, whose gradient on the logits is α πj(log⁡πj+H)\alpha\,\pi_j(\log\pi_j + H) per token: it raises every token whose log-probability is below −H-H and lowers the likely ones. Over a vocabulary of 316 words (or 151k tokens for Qwen3), “every unlikely token” is mostly tokens the policy should never emit. Section 10.1 showed the same thing in closed form: the entropy bonus is a KL penalty toward the uniform distribution.
  • A KL penalty pulls each token’s distribution toward the reference’s, which has the reference’s entropy. It resists collapse only as far as the reference is uncertain, and, as a reverse KL, it does not penalize dropping alternatives the reference had (Section 10.5).
  • Clip-higher (DAPO) raises ϵhigh\epsilon_{\text{high}}. With A>0A > 0, PPO’s clip removes a token’s gradient once πθ(a)>(1+ϵhigh) πold(a)\pi_\theta(a) > (1 + \epsilon_{\text{high}})\,\pi_{\text{old}}(a), so the absolute increase a token can receive before the clip binds is ϵhigh πold(a)\epsilon_{\text{high}}\,\pi_{\text{old}}(a): 0.18 for a token at 0.9, and 0.002 for a token at 0.01. The upper clip therefore binds on rare tokens with positive advantage, which are exactly the updates that raise entropy in the table above. DAPO’s stated reason is the same: “the probability of an unlikely ‘exploration’ token is too tightly bounded to be uplifted”, and it raises ϵhigh\epsilon_{\text{high}} from 0.2 to 0.28. The lever works only if the clip binds, which requires the policy to move within a batch (several minibatches or epochs).
  • Sampling temperature TT samples from softmax(z/T)\mathrm{softmax}(z/T). Because the trainer computes log-probabilities at the same TT (Chapter 9’s rl_update divides the logits by cfg.temperature), the optimized policy is the tempered one, and the network can sharpen zz to undo any fixed TT. A higher temperature delays collapse rather than preventing it; ProRL reports that with raised rollout temperature “entropy continues to decline steadily as training progresses”. Evaluating at a lower temperature than training is then a separate, deliberate choice (Chapter 17).
train_rl has no hook for an extra loss term, and Chapter 9’s functions must not change. We therefore save entropy_bonus and a regularized version of Chapter 9’s update and loop, built from the same saved pieces (sample_step, behavior_stats, compute_advantages, policy_loss). entropy_bonus is the masked mean of the full-vocabulary entropy at policy positions; unlike the KL estimators, it is exact at each visited prefix rather than a one-sample estimate, because the logits for the whole vocabulary are already computed.
regularized_update is Chapter 9’s rl_update with two differences: the loss subtracts entropy_coef times the entropy bonus, and the KL term uses kl_estimate, so cfg.kl_kind may be 'k3+'. The logged entropy is the bonus’s value at each minibatch, computed before the step.
train_regularized is train_rl without periodic evaluation, plus the reward placement of Section 10.4. With kl_in_reward=True, each episode’s reward becomes ri−β∑tk1,tr_i - \beta \sum_t k_{1,t}, computed from the sampler’s log-probabilities and the reference before the advantages, and the loss carries no KL term. sequence_kl computes that sum.
With both regularizers off, the loop must be Chapter 9’s. It consumes the random generator in the same order as train_rl, so three steps of each from the same start should give identical parameters.
Output:
The difference is exactly zero, so every regularized run below differs from Chapter 9’s train_rl only by the regularizer it switches on.

​
Clip-higher on one batch

Whether the upper clip binds depends on how far the policy moves within a batch, and Chapter 9’s default (one epoch, two minibatches) clipped 0.13% of tokens, too few for ϵhigh\epsilon_{\text{high}} to matter. To see what the clip does when it binds, we take one batch of 64 episodes from the RL base agent and run 8 full-batch steps on it with ϵhigh\epsilon_{\text{high}} at 0.2, at DAPO’s 0.28, and effectively unbounded. After the 8 steps we count the positive-advantage policy tokens whose ratio exceeds 1+ϵhigh1 + \epsilon_{\text{high}}, and how many of them started below probability 0.5.
Output:
In both clipped settings, most of the positive-advantage tokens that ended above the upper clip had started below probability 0.5 (15 of 20 at ϵhigh=0.2\epsilon_{\text{high}} = 0.2, 14 of 15 at 0.28), although only 17 of the 121 positive-advantage tokens started there. That part of DAPO’s argument holds: the upper clip is a constraint on rare tokens. The entropy column does not follow the argument’s conclusion. After 8 steps, entropy fell least with the tightest clip (0.547 to 0.538) and most with no upper clip at all (to 0.508). Removing the cap frees every positive-advantage token, including the mid-probability ones whose growth sharpens the distribution, and on this batch that effect outweighs the extra room for rare tokens. DAPO reports that clip-higher raises entropy in long-reasoning training at 32B scale; on TinyAgent we do not see it, either here or in Chapter 9’s ablation, so the comparison below leaves clip-higher out and the claim stays with its source.

​
10.7 Comparing the regularizers on TinyAgent

We now train the default configuration for 40 steps (where Chapter 9’s run still had headroom) under nine settings: no regularizer; KL penalties in the loss with the correct gradient (k3+) at β\beta of 0.05, 0.3, and 1.0; the GRPO-style k3k_3 loss at 0.3; the KL in the reward at 0.3 (Section 10.4); entropy bonuses of 0.03 and 0.1; and a sampling temperature of 1.3. Each run starts from the same RL base agent with seed 0. Besides held-out pass@1 we measure diversity. For a task with cc passing attempts out of nn, the probability that kk attempts drawn without replacement contain a pass is 1−(n−ck)/(nk)1 - \binom{n-c}{k}/\binom{n}{k}; Chapter 17 derives this estimator and saves it, so here it is a local helper. We also count distinct action sequences among the 16 attempts at each task, and estimate the per-token KL from the starting policy with k3k_3.
Output:
Read the table by column, and remember that it is one seed on 10 held-out tasks with 16 attempts each, so pass@8 moves in steps of about 0.1 when a task is never solved.
  • Entropy. Without a regularizer the policy-token entropy fell from 0.52 to 0.12 in 40 steps. Every regularizer except the weakest KL slowed the fall, roughly in proportion to its strength: a KL coefficient of 0.3 held it at 0.28, 1.0 held it where it started (0.54), and the entropy bonus of 0.03 at 0.31. The bonus of 0.1 drove the policy to the uniform distribution: 5.707 nats against log⁡316=5.756\log 316 = 5.756, six malformed calls per episode, and zero pass rate.
  • pass@1 against pass@8. The unregularized run had a pass@1 of 0.725 and a pass@8 of only 0.80: two held-out tasks were never solved in 16 attempts, because the collapsed policy repeated the same wrong procedure on them. The weakest KL (0.05) left entropy unchanged at 0.124 and still reached pass@8 of 1.000, with the best pass@1 of the table, 0.80; at one seed we cannot separate that from noise. Stronger KL bought diversity with pass@1: at β=0.3\beta = 0.3 the distinct action sequences per task doubled (8.3 against 4.3) and pass@8 reached 0.99, while pass@1 fell to 0.59. At β=1.0\beta = 1.0, reading did not take over (0.53 reads per episode) and pass@1 was 0.23, close to the blind ceiling of 0.2, with pass@8 still 0.90 because the near-reference policy samples many different guesses.
  • Estimator. The k3k_3 loss and the k3+ loss at β=0.3\beta = 0.3 ended at nearly the same entropy (0.294 and 0.284), pass@1 (0.594 and 0.588), and distance from the start. At these per-token KLs (about 0.1), Section 10.3 puts the gradient difference near 25%, and one seed does not resolve its effect.
  • Placement. The KL in the reward at the same β=0.3\beta = 0.3 was much stronger than either loss placement: pass@1 0.34, reads 0.48, and the smallest distance from the start apart from β=1.0\beta = 1.0 (0.059). This is the scale difference of Section 10.4: the reward version sums the log-ratio over the episode’s 12 to 14 policy tokens instead of averaging it, and then enters an advantage normalized by each group’s spread.
  • Entropy bonus and temperature. Both kept entropy higher while moving the policy further from the start than no regularizer (KL 0.325 and 0.517 against 0.237), and both raised malformed calls (0.28 and 0.25 per episode against 0.04). The entropy bonus spreads mass onto tokens no policy should emit; the temperature run’s network sharpened its logits to compensate for T=1.3T = 1.3, so evaluated at T=1T = 1 it is further from the start policy than any setting except the collapsed entropy bonus. Its training-time entropy, measured at T=1.3T = 1.3, still fell from 0.78 to 0.47.
Figure 10.3: Policy-token entropy (top) and training batch reward (bottom) over 40 GRPO steps from the RL base agent, one seed. Left: KL penalties with the reverse-KL gradient (k3+) at three coefficients. Right: an entropy bonus, the k3 loss, and the KL in the reward. The unregularized run is the orange reference line in every panel. Figure 10.3 shows the dynamics behind the table. The left column orders the KL runs by coefficient: the stronger the penalty, the later and shallower the entropy fall, and the lower the reward curve. The right column shows the other placements. The entropy bonus tracks the unregularized reward curve closely while keeping entropy near 0.3; the k3k_3 loss follows nearly the same path as the k3+ loss at the same coefficient (the two green curves, one in each column); and the KL in the reward holds entropy highest and reward lowest. No regularizer improved the training reward at 40 steps on this gym, which has one correct procedure per task: what they buy is diversity in the samples, at a cost in pass@1 that grows with their strength.
Evidence. Measured here: one seed per setting, 40 steps, 64 episodes per step, evaluation on 10 held-out tasks with 16 attempts each. Differences in pass@1 smaller than about 0.08 are within the noise of this evaluation (Chapter 9 measured a spread of 0.71 to 0.81 across three seeds for the default run). The qualitative ordering by strength (entropy, distinct sequences, distance from the start) is the mechanism being demonstrated.

​
The case for dropping the KL

DAPO removes the KL term, arguing that when training long reasoning, “the model distribution can diverge significantly from the initial model, thus this restriction is not necessary” (DAPO, Section 2.3, 2025-03-18). The public coding-agent recipes in Chapter 20 (DeepSWE, SkyRL-Agent) also report no KL loss and no entropy loss. The argument from Section 10.1 supports this when the reward is a verifier: there is no learned reward to exploit, so the first reason for a reference disappears, and the remaining reasons (preserving unmeasured capabilities, a cumulative trust region) can be handled by evaluation on other tasks and by PPO’s per-step clip. The counter-position is ProRL (2505.24864, 2025-05-30), which trains for more than 2,000 steps and reports that “explicit regularization via a KL divergence penalty provides a stronger and more stable solution”, helping “maintain entropy”, while noting that over time “the KL term may increasingly dominate the loss, leading to diminishing policy updates”. Its answer is to reset the reference, which Section 10.8 reproduces. Both are reports from single-turn reasoning at large scale, not controlled evidence for agents.

​
10.8 Design problems in practice

Problem: entropy collapse early in training. Symptom: the policy-token entropy halves within the first tenth of the run while the reward is still low, the share of zero-advantage groups climbs, and pass@kk on held-out tasks approaches pass@1. Cause: the covariance of Section 10.6: in each group, the successful attempts are mostly the policy’s already-likely tokens, so most positive-advantage updates sharpen the distribution, and the step size (learning rate, epochs) sets how fast. Options: (a) lower the learning rate or the number of epochs, which slows learning too; (b) a KL penalty to the starting policy, which holds entropy near the reference’s at a cost in pass@1; (c) an entropy bonus, with a small coefficient and a watch on malformed outputs; (d) clip-higher, which helps only if the clip binds; (e) a higher sampling temperature, which delays collapse. Default: the book’s recommendation is to log entropy and pass@kk from the first step, and to treat collapse as a problem only when pass@kk falls or a known-good behavior disappears; try (b) with the k3+ gradient before (c). The histories show how fast the default run collapses and what (b) and (c) change.
Output:
Without a regularizer, entropy fell by a third in the first 10 steps (0.56 to 0.37) while the batch reward was still 0.41, and to 0.18 by step 20; Chapter 9’s 80-step run continued to 0.08. The KL penalty and the entropy bonus both slowed the fall to about 0.30 at step 20. The entropy bonus did so with the same batch reward at step 39 (0.86), and the KL at a cost (0.73). What the default trajectory cost on held-out tasks is in the table above: the two tasks it never solved.
Problem: a KL coefficient that is too strong or too weak. Symptom: too strong, the logged KL stays near zero, reward plateaus early, and a behavior the reference rarely shows (here, reading) never takes over; too weak, the KL grows without bound and the term does nothing measurable. Cause: at the optimum of Section 10.1, the policy tilts the reference by er/βe^{r/\beta}, so β\beta must be compared with reward differences, and the per-token KL must be compared with per-token advantages. The scale also depends on placement: a per-token mean in the loss and a per-episode sum in the reward differ by the number of policy tokens (about 13 here, thousands for agents). Options: (a) sweep β\beta on a log grid while logging the KL; (b) an adaptive controller that adjusts β\beta toward a target KL (verl’s AdaptiveKLController); (c) no KL, with a held-out regression suite. Default: (c) for verifiable rewards, following the public coding-agent recipes; (b) when a learned reward or judge is in the loop. The KL logs of the three k3+ runs show the regimes.
Output:
With β=0.05\beta = 0.05, the KL to the reference climbed to 0.20 by step 20, close to the unregularized run’s final distance from the start (0.237 on held-out episodes in the table), so the term changed little. With β=1.0\beta = 1.0, it held the policy within a per-token KL of 0.03 for the whole run, and reading, which the reference does about once in every four or five episodes (Chapter 9 measured 0.21 to 0.24 reads per episode), never took over (0.53 reads per episode, pass@1 0.23). β=0.3\beta = 0.3 is the intermediate regime: the KL plateaued near 0.1 while the reward kept rising slowly.
Problem: the memory and compute cost of the reference model. Symptom: enabling a KL term runs the trainer out of memory or adds a large share to step time. Cause: the reference is a second full model that must produce log-probabilities for every training token. It needs only its weights (no gradients or optimizer state), but it adds one forward pass to a step that otherwise consists of the trainer’s log-probability pass for πold\pi_{\text{old}} (Chapter 9 recomputes it) and one forward and backward pass. Options: (a) compute reference log-probs on the inference fleet during rollout and ship them with the batch; (b) offload or shard the reference separately; (c) with LoRA training, use the base model with adapters disabled as the reference, at no extra memory; (d) drop the KL. Default: (d) when the reward is verifiable, otherwise (c) or (a). The arithmetic for a few model sizes, assuming bf16 weights and gradients with fp32 master weights and Adam moments (16 bytes per trained parameter):
Output:
The ratio is the same 12.5% at every size, because it is two bytes against sixteen per parameter. The absolute numbers decide feasibility: for a 32B policy, the reference adds 60 GiB that must fit beside the sharded training state on every node that computes reference log-probs, and one more forward pass over every training token: counting a backward pass as about two forwards, a step without a reference costs about four forward-pass units (the πold\pi_{\text{old}} pass, the forward, the backward) and the reference adds a fifth, 25% more trainer compute. TinyAgent’s 157k parameters make the reference free here, which is why this chapter can afford one in every run; the decision at scale is not free.
Problem: resetting the reference. Symptom: with a KL penalty, the logged KL settles at a plateau while the reward stops improving; the penalty, not the task, has become the binding constraint. Cause: the reference is the starting policy, so the further training has to go from the start, the more of each update the KL gradient cancels. Options: (a) lower β\beta; (b) hard-reset the reference to the current policy (and, as ProRL does, the optimizer state), which keeps the KL as a local trust region rather than an anchor to the start; (c) drop the KL. Default: if a KL is used at all, reset on a schedule tied to a validation plateau, as ProRL reports doing. We continue the β=0.3\beta = 0.3 run for 20 steps twice, once anchored to the start and once with the reference reset to the current policy; both restart the optimizer.
Output:
Kept anchored to the start, the policy’s KL to the reference sat at 0.18 and its pass@1 rose to 0.731. With the reference reset to the current policy, the KL to the (new) reference was only 0.029, entropy fell to 0.135, and pass@1 rose further, to 0.787, with fewer distinct sequences (4.6 against 6.0). The reset does what ProRL describes: it removes the accumulated pull toward the start and lets the policy keep improving, and with it goes part of the diversity the KL had preserved. One seed and 160 evaluation episodes cannot rank 0.731 against 0.787; the direction of every column is the mechanism.

​
Summary

  • The KL-regularized objective has the closed-form optimum π∗∝πref er/β\pi^* \propto \pi_{\text{ref}}\, e^{r/\beta}: the reference tilted by the exponentiated reward, never leaving the reference’s support. An entropy bonus is the same objective with a uniform reference.
  • Of Schulman’s estimators, k1k_1 and k3k_3 are unbiased values and k3k_3 has by far the lowest spread at small KL. As losses, they give different gradients: k1k_1 gives zero in expectation, k2k_2 gives the reverse-KL gradient, and k3k_3 gives the forward-KL gradient, with an error against the reverse direction that grows with the distance. kl_estimate(..., 'k3+') logs k3k_3 and differentiates as k2k_2.
  • For sequences, a per-token KL loss ignores how a token changes the KL of the tokens after it; KL in the reward, through the policy gradient, gives the exact sequence gradient. On TinyAgent the reward placement at the same β\beta was a much stronger penalty, because it sums over tokens and enters the advantage.
  • Reverse KL is mode-seeking and forward KL mass-covering; a reverse-KL penalty does not by itself preserve the reference’s alternatives.
  • Entropy falls when updates raise already-likely tokens, ΔH≈−Cov(log⁡π,Δz)\Delta H \approx -\mathrm{Cov}(\log\pi, \Delta z). On TinyAgent the unregularized run went from 0.52 to 0.12 nats in 40 steps and never solved two of ten held-out tasks in 16 attempts; KL penalties and an entropy bonus slowed the collapse and raised pass@8 at a cost in pass@1, an entropy bonus of 0.1 destroyed the action format, and clip-higher had no measurable effect on entropy. These are one-seed mechanism demonstrations.
  • With verifiable rewards, dropping the KL (DAPO, DeepSWE) removes a reference model and a tuning knob; keeping it (ProRL) trades pass@1 for entropy and needs reference resets to stop anchoring training to its start.

​
Exercises

  1. Show that adding the missing score-function term to a k1k_1 loss, E[∇k1+sg(k1)∇log⁡πθ]\mathbb{E}[\nabla k_1 + \mathrm{sg}(k_1)\nabla\log\pi_\theta], gives the reverse-KL gradient, and that the same correction applied to k3k_3 also does. Check both with expected_loss_grad-style enumeration. Why does nobody implement the corrected k3k_3 instead of k3+?
  2. Expand Eπθ[k2]\mathbb{E}_{\pi_\theta}[k_2] to third order in ℓ\ell and predict the bias column of Section 10.2 for small KL. At what KL does your approximation fail?
  3. Change an assumption: rerun Section 10.7 for 80 steps instead of 40, with the unregularized and KL k3+ 0.05 settings only. Does the unregularized run’s pass@8 fall below the regularized one’s once its entropy reaches Chapter 9’s 0.08? Report the difference against the spread of three seeds.
  4. Extend the two-token model of Section 10.4 to three tokens and implement a loss placement that is exact: the per-token k2k_2 loss plus a REINFORCE term carrying each token’s future KL. Verify it against the exact gradient, then explain which part a GRPO trainer would have to add to its loss to implement it.
  5. Implement an adaptive KL coefficient (β←β(1+clip(KL/KLtarget−1,±0.2)/10)\beta \leftarrow \beta(1 + \mathrm{clip}(\mathrm{KL}/\mathrm{KL}_{\text{target}} - 1, \pm 0.2)/10) per step) in a copy of train_regularized. Which target KL lets TinyAgent learn to read while keeping pass@8 at least as high as the unregularized run?
  6. Restrict the entropy bonus to the first policy token after each tool observation (the positions ARPO identifies as high-entropy; Chapter 20). Compare malformed calls and pass@8 with the uniform bonus at a matched mean entropy.
  7. Design review: a team trains a SWE agent with verl’s defaults, use_kl_loss=True, kl_loss_type=low_var_kl, kl_loss_coef=0.001, on a test-execution reward. The logged kl_loss is 0.02 and policy entropy has fallen by 70% in 200 steps. Using this chapter, argue whether the KL term is doing anything, which gradient it computes, what you would log next, and what you would change if the reward were an LLM judge instead of tests.