Skip to main content
Chapter 8 made sample reuse safe with a clipped ratio, and it paid for its advantages with a critic that explained a small share of the variance of the returns and doubled the trainer’s memory. Chapter 7 had already shown the alternative: sample several attempts at the same task and let them serve as each other’s baseline. Group Relative Policy Optimization (GRPO; Shao et al., 2024) combines the two ideas, PPO’s clipped surrogate with a group baseline and no critic, and it and its descendants are the objectives of most public RL recipes for reasoning and coding models (Chapter 20). This chapter builds the book’s main training loop, train_rl, around GRPO, and expresses each published variant (Dr. GRPO, DAPO, RLOO, GSPO) as a set of configuration fields of that one loop rather than as a separate program. Chapters 10 to 19 all train with it. We then run it on the BugFix gym and meet a problem that no choice of objective fixes: the agent learns to pass training tasks without reading the code. Diagnosing that problem with behavior statistics and fixing it in the environment is the core of Section 9.4, and it decides the task setting the rest of the book uses. The chapter ends with small controlled ablations of the variants, labeled for what they are: mechanism demonstrations on a 158k-parameter model, not benchmark evidence.

​
9.1 GRPO as in DeepSeekMath

DeepSeekMath (Shao et al., 2024) defines GRPO for a question qq with GG sampled outputs o1,…,oGo_1, \dots, o_G from πold\pi_{\text{old}}: JGRPO(θ)=E[1G∑i=1G1∣oi∣∑t=1∣oi∣(min⁡(ρi,tA^i, clip(ρi,t,1−ϵ,1+ϵ)A^i)−β D^KL[πθ ∥ πref]i,t)],J_{\text{GRPO}}(\theta) = \mathbb{E}\Big[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\Big(\min\big(\rho_{i,t}\hat A_{i},\ \mathrm{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\hat A_{i}\big) - \beta\, \hat{\mathbb{D}}_{\text{KL}}\big[\pi_\theta \,\|\, \pi_{\text{ref}}\big]_{i,t}\Big)\Big], with outcome advantages A^i=(ri−mean(r))/std(r)\hat A_i = (r_i - \mathrm{mean}(\mathbf{r}))/\mathrm{std}(\mathbf{r}) shared by every token of output ii, and the per-token KL estimate πrefπθ−log⁡πrefπθ−1\frac{\pi_{\text{ref}}}{\pi_\theta} - \log\frac{\pi_{\text{ref}}}{\pi_\theta} - 1, the k3k_3 estimator. Read against the earlier chapters, it is four choices:
  1. Group sampling. GG attempts per task in the same step (DeepSeekMath reports G=64G = 64).
  2. A critic-free advantage. The group mean as the baseline and the group’s standard deviation as the scale: group_advantages of Section 7.5.
  3. PPO’s clipped per-token ratio (Section 8.2), aggregated by averaging each sequence’s tokens and then the sequences: seq_mean_token_mean of Section 6.6.
  4. A KL penalty to a frozen reference policy in the loss, with coefficient β\beta (reported as 0.04), rather than as a per-token reward as in RLHF-style PPO.
DeepSeekMath also reports that “the policy model only has a single update following each exploration stage”. With a single update, πθ=πold\pi_\theta = \pi_{\text{old}} at the moment of the gradient, every ratio is 1, and the clip is inactive; the objective is then a group-baseline policy gradient with a KL term. Many implementations instead split each batch into several minibatches, which makes all but the first slightly off-policy and gives the clip work to do. “GRPO” in practice therefore names a family of settings, and the book writes them down as fields. RLConfig collects every setting that train_rl reads. The defaults are the book’s GRPO configuration on the version of the gym that Section 9.4 arrives at. Fields that belong to a variant are introduced in Section 9.5.
Three defaults depart from DeepSeekMath and need a reason. token_mean replaces seq_mean_token_mean, for the length reasons of Section 6.6, revisited in Section 9.7. kl_coef is 0, as in DAPO and the public coding-agent recipes of Chapter 20; Chapter 10 studies the KL term. And each batch is split into 2 minibatches, so the clip is exercised but the update stays close to on-policy (Section 8.6).

​
9.2 The training loop

train_rl is assembled from five pieces, each saved separately so that later chapters can replace one: an environment factory and a group sampler, behavior statistics, advantages, the clipped update, and the loop itself.

​
Sampling groups

make_env builds one environment as the configuration describes it. EditBudgetEnv, the environment it builds by default, is defined in Section 9.4; with max_edits=None it returns Chapter 4’s BugFixEnv unchanged. sample_groups lays tasks out in consecutive groups of GG, which is the layout group_advantages expects, and runs all episodes in lockstep with Chapter 5’s rollout_batch. When cfg.rename is set, each episode’s task passes through randomize_names (Section 9.4) with a Python random stream seeded from the torch generator, so the whole step is reproducible from one seed. The env_factory argument is the hook for custom environments and rewards (Chapter 11 uses it).

​
What the agent did

Reward alone cannot distinguish an agent that reads the code from one that guesses well, and Section 9.4 depends on telling them apart. behavior_stats reports, per episode: the hidden-test pass rate, the reward, the number of read, test, and applied edit calls, whether the first applied edit was the task’s correct op, the number of turns, and the number of policy tokens. The correct op of each task comes from the family table of Chapter 4, so the renamed tasks of Section 9.4 are scored by their original identity.

​
Advantages and the KL term

compute_advantages turns a collated batch into one advantage per episode. Rewards may first be shaped by DAPO’s overlong penalty (Section 9.5); the three advantage modes are Chapter 7’s estimators. kl_penalty is the per-token estimate of KL(πθ ∥ πref)\mathrm{KL}(\pi_\theta \,\|\, \pi_{\text{ref}}) from tokens sampled by πθ\pi_\theta, with the log-ratio log⁡(πref/πθ)\log(\pi_{\text{ref}}/\pi_\theta) written as ℓ\ell: k1=−ℓk_1 = -\ell, k2=ℓ2/2k_2 = \ell^2/2, k3=eℓ−1−ℓk_3 = e^{\ell} - 1 - \ell. Chapter 10 derives their bias, variance, and the gradients they produce as losses.

​
The update

policy_loss dispatches on the ratio level: per-token ratios go to Chapter 8’s ppo_policy_loss, sequence-level ratios to gspo_policy_loss (Section 9.5). rl_update is Chapter 8’s PPO phase without the critic. It freezes πold\pi_{\text{old}} and, when a reference model is given, πref\pi_{\text{ref}}, by one no-grad forward pass over the batch (the trainer recomputes old_logp, as Section 8.6 recommends, at the sampling temperature), then runs cfg.epochs passes of cfg.minibatches clipped steps. The KL term is averaged over policy tokens. Entropy is computed from the logits the update already has, so logging it costs nothing.

​
The loop

sample_step draws prompts_per_step distinct training tasks and samples their groups. With dynamic sampling it keeps only groups whose rewards differ and samples more tasks until the batch is full or the round limit is reached; Section 9.5 explains the choice. It returns the kept episodes and everything sampled, because the cost of a step is what was sampled.
train_rl runs steps iterations of sample, advantage, update, and log. It trains model in place and returns a history with one dictionary per step: the behavior statistics of everything sampled, the update’s statistics (clip_frac, approx_kl, ratio range, entropy, grad_norm, and kl_ref when a reference is used), the number of episodes sampled, the fraction of sampled groups whose rewards were all equal (zero_adv_frac), and, every eval_every steps, the pass rate on eval_tasks with 4 episodes per task. When kl_coef > 0 and no reference is passed, the reference is a frozen copy of the starting model.
evaluate samples n_samples episodes per task under the same environment settings as training and returns the behavior statistics. The default is sampled pass@1 at temperature 1; Chapter 17 treats evaluation properly.

​
9.3 GRPO on the gym as built

We first train exactly the setup of Chapter 6: the base agent, the gym of Chapter 4 with its function names and unlimited edits. Only the update changes, to GRPO with groups of 8 on 8 tasks per step. rename=False and max_edits=None select the original gym.
Output:
Output:
The run converges quickly: the batch reward passes 0.9 by step 60, and the entropy of the policy tokens falls from 0.53 to 0.10. On the training tasks the trained policy passes 98% of episodes. On held-out tasks it passes 40%, against 15% for the base agent. A large gain on training tasks with a much smaller one on held-out tasks is the signature of memorization, and the behavior columns say what was memorized. The policy never reads and essentially never tests. Its first edit is the right op in 71% of training episodes and 13% of held-out ones, where a uniform guess among the five repair ops would score 20%. On the held-out tasks it falls back on stacking, 2.6 applied edits per episode. This is Chapter 6’s finding reproduced with a different estimator: what was learned is a property of the environment and the starting policy, not of REINFORCE.

​
9.4 Shortcut features in environment design

A shortcut feature is an input that predicts the reward on the training distribution without carrying the information the task is meant to require. Optimizers find shortcuts because they are cheaper than the intended skill. The behavior statistics point at the BugFix gym’s first one: the issue text.
Output:
The issue names the function, and within the training set each function name occurs in exactly one task, so the name identifies the task and therefore its fix op. A policy that maps the first word after issue : to an op needs no information from the environment at all. On held-out tasks the names are words it has never been trained on, and the map has nothing to say. Real benchmarks leak in the same way. SWE-Bench+ (Aleithan et al., 2024) reports that 32.67% of the successful patches it examined on SWE-bench had the solution “directly provided in the issue report”, and SWE-Bench Illusion (Liang et al., 2025) reports that models identify the buggy file path from the issue text alone with up to 76% accuracy, dropping to up to 53% on repositories outside the benchmark, which the authors read as memorization (Chapter 20). In both cases the issue carries information that a real engineer would have to obtain by reading code. The fix belongs in the environment: make the shortcut carry no information. randomize_names renames the function in a task’s issue and source to a name drawn at random from all 30 function names of the gym, excluding words already in the source. Every name stays in the Chapter 5 vocabulary, so no tokenizer change is needed. Applied independently to every episode, the name becomes noise: each name appears with every family equally often, and a policy can no longer predict the op from it.
The task keeps its name, so logs and FIX_OP still identify it. We rename the clamp task twice and check that every renamed issue and source of every task encodes without unknown tokens.
Output:
We retrain from the same base agent with renaming on and everything else unchanged.
Output:

​
The second shortcut

Renaming shrank the gap between training and held-out pass rates from 0.58 to 0.12 (0.59 against 0.48), and it did not produce reading: 0.01 reads per episode. The first edit is now right in 22% and 18% of episodes, the rate of a uniform guess, and the policy applies about 2.5 edits per episode instead. It has switched to the other strategy Chapter 6 found: stacking edits. A wrong op usually leaves the source unchanged or changes it harmlessly, so applying several ops in a fixed order repairs several families without looking. How far can that go? blind_pass_rate applies a fixed sequence of ops to every task and runs the hidden tests.
Output:
A policy that never reads can pass 60% of all tasks with three edits and a submit, well inside the six-turn budget. Renaming removed the information in the issue, and the edit menu still lets the agent enumerate. This is the second shortcut feature, and it is a property of the action space rather than of the observation: wrong actions are free. Real environments are rarely this forgiving. A patch that touches the wrong code usually breaks other tests, and benchmarks grade one final patch. The gym models that by limiting the number of edits an episode can apply. EditBudgetEnv refuses edits beyond max_edits: the call uses a turn and returns no change, and the source is untouched.
With one edit, the best blind policy is the best single op, 0.20 by the table above. Any pass rate above 0.2 now requires information from the environment: the read observation, or the test output before the edit. We retrain once more from the base agent, now with the book’s default configuration: renamed functions and one edit.
Output:

​
The exploration problem underneath

With both shortcuts closed, the policy did not learn to read either. Its pass rate stays at the blind ceiling, 0.19 on training tasks and 0.15 held-out, and its reads fall from 0.30 per episode at the first step to 0.02. The batch reward wanders between 0.14 and 0.34 as the policy shifts its single guess from one op to another. The environment now requires reading, and the policy gradient still cannot find it. The reason is in the base agent’s own samples. RL reweights what the policy already does: an action sequence gains probability only if it earns higher rewards than its group’s other attempts. We measure how often the base agent reads before its first edit, and how often that edit is then right.
Output:
The base agent reads before its first edit in only 7% of episodes, and that edit is then right 25% of the time, close to the 20% of a guess (on about 20 episodes), because its demonstrations came from random and blind play in which reading was never followed by an informed choice. Reading therefore earns the same reward as not reading, its advantage is zero in expectation, and noise and the faster-improving alternatives drive it out. This is an exploration failure, and closing shortcuts cannot fix it: an environment can make a skill necessary, but RL can only amplify a skill the policy already exhibits occasionally. A pretrained language model is in a better position than this base agent: it has seen code being read and used, and it sometimes does the right thing for the right reason. The book’s analogue is a base agent whose demonstrations include a small share of informed play. make_rl_base adds one demonstration set from Chapter 4’s oracle to every four random and four blind sets, so one demonstration in nine reads and then applies the right op.
Output:
The new starting point has the same pass rate as the old one, 0.17 on training tasks and 0.16 held-out, so it is not a stronger agent. The difference is in one conditional probability: it reads first in 12% of episodes, and its next edit is then right 41% of the time, twice the rate of a guess. That is enough signal for the group baseline to find. The main run of the chapter trains it with the default configuration and evaluates on held-out tasks every 20 steps.
Output:
This time reading spreads through the batch within 20 steps, and the policy reads in essentially every episode. Its first edit is right in 92% of training episodes and 84% of held-out ones, and the held-out pass rate rises with the training one, from 0.10 at step 0 to 0.85 at step 79 in the periodic evaluation and 0.84 in the 160-episode one. The 80 steps took about 10 to 15 seconds on the check host. The most common held-out episodes show the learned procedure.
Output:
Finally, a control: is the environment fix still needed once the base agent can explore? We train the RL base agent on the original gym, with function names and unlimited edits.
Output:
Figure 9.1: Held-out behavior after 80 GRPO steps in five settings: the base agent on the original gym, with renamed functions, with renaming and one edit; the RL base agent with renaming and one edit (the book's default) and on the original gym. Bars: held-out pass rate, reads per episode, and first-edit accuracy. Every frequent held-out episode is the oracle’s procedure: read, apply the op that the source calls for, test, submit. The test call is not needed with one edit, since nothing can be done with its result, and the policy keeps it because nothing penalizes it, a reminder of Chapter 7’s point that outcome advantages also credit useless steps. The control is informative in a way we did not expect when we designed it. The RL base agent learns to read on the original gym too, and reaches the same held-out pass rate, 0.838, with slightly more stacking (1.21 edits per held-out episode against 0.99). Once the starting policy explores, reading is the best strategy in either gym, and the shortcuts no longer win. Figure 9.1 collects the five runs. The two changes do different jobs. The starting policy decided whether the skill was learned: from the Chapter 6 base agent, no environment produced reading, and from the RL base agent, both did. The environment decides what a pass rate means. In the original gym, a pass rate of 0.6 can come from reading, from names, or from blind stacking, and only behavior statistics tell them apart. In the anonymized one-edit gym, anything above 0.2 must come from information gathered during the episode, so a later chapter whose change damages exploration (a strong KL penalty, an entropy collapse, a hackable reward) shows up as a drop toward 0.2 instead of a silent switch to a shortcut.
Evidence. Measured here, one seed per setting, 80 steps of 64 episodes. The qualitative pattern (memorize, then stack, then stall, then read) matched what we saw in exploratory runs with other learning rates and seeds while preparing the chapter; those runs are not reported. The numbers are tiny-model demonstrations of a mechanism. The general claims are about the mechanism: a policy gradient amplifies whatever features predict reward in the policy’s own samples, so shortcut features must be removed from the environment, and the intended skill must be present, however rarely, in the starting policy.
The default setting for the rest of the book. From here on, chapters train with RLConfig() defaults, which use the anonymized gym (rename=True) with one edit per episode (max_edits=1), starting from make_rl_base(tok, seed), and evaluate with evaluate(..., cfg=cfg) on heldout_tasks. With these settings, a pass rate above 0.2 can only come from information gathered in the episode, and the behavior statistics in every history make that checkable. Chapters that study a shortcut on purpose, such as reward hacking in Chapter 11, change the reward or the environment through RLConfig.reward or env_factory and keep everything else.

​
9.5 The variants as diffs

Each public variant of GRPO changes a few of the choices listed in Section 9.1. The table states each as a diff against DeepSeekMath’s GRPO and gives the RLConfig fields that implement it. The source column summarizes what each paper specifies, as recorded in Chapter 20; dates are first arXiv versions. Dr. GRPO removes the two normalizations whose biases Sections 6.6 and 7.5 measured: the division by each output’s length, which reduces the per-token penalty on long failures, and the division by the group’s standard deviation, which upweights tasks the policy almost always solves or almost always fails. DAPO makes four changes. Clip-higher raises ϵhigh\epsilon_{\text{high}} so that rare tokens with positive advantages can grow faster (Section 8.6 measured that the upper clip binds mostly on them). Dynamic sampling drops groups whose rewards are all equal, whose advantages are exactly zero, and samples more tasks until the batch is full, so every kept episode carries gradient. Token-level loss is token_mean. Overlong reward shaping adds a penalty that grows linearly from 0 to −1-1 over a buffer before the maximum length. In the gym the length that matters is the turn budget, so overlong_buffer=2 penalizes episodes that use the fifth or sixth turn by −0.5-0.5 and −1-1. DAPO also describes overlong filtering, masking truncated episodes out of the loss, which DeepSWE’s “compact filtering” generalizes to timeouts and step limits. RLOO uses the leave-one-out baseline of Section 7.5 and treats the whole generation as one action, so it is REINFORCE with a per-task baseline. Its advantage is GG−1\frac{G}{G-1} times Dr. GRPO’s, an overall scale that Adam largely absorbs. GSPO replaces the per-token ratio by one ratio per sequence, the geometric mean of its token ratios, si=exp⁡(1∣oi∣∑tlog⁡ρi,t)s_i = \exp\big(\frac{1}{|o_i|}\sum_t \log \rho_{i,t}\big), and clips sis_i. Every token of a sequence then shares one clip decision and one importance weight. Its authors motivate this by the mismatch between a sequence-level reward and token-level corrections, and report that it stabilized training of mixture-of-experts models. Because sis_i is an average of log-ratios, it moves much less than individual token ratios, which is why its reported clip ranges are three orders of magnitude smaller than PPO’s.
VARIANTS is a starting point for experiments, not a reproduction of any paper: learning rates, group sizes, batch sizes, and models differ from the papers’, and only the listed fields change. Each preset runs for 40 steps from the RL base agent.
Output:
Every preset learns to read within 40 steps; the differences in reward and held-out pass rate are single-seed numbers on 80 held-out episodes, about 0.04 standard error, and Section 9.6 measures the seed spread. Three columns say something about mechanisms rather than about which variant is best. DeepSeekMath’s GRPO is the only preset with a reference model, and its KL estimate from the starting policy reached 0.17 on average, which at β=0.04\beta = 0.04 is a small term next to the advantages; Chapter 10 studies when it is not. DAPO sampled about twice as many episodes as the others, the price of dynamic sampling. And GSPO’s clip fraction is 0.31 against about 0.002 for the token-level presets: with the paper’s ranges, chosen for outputs thousands of tokens long, the sequence ratio of a 12-token episode leaves the band after a single minibatch step, so most of every second minibatch is clipped away. RLOO, with one minibatch, never clips at all, as Section 9.1 predicted.

​
9.6 Controlled ablations at tiny scale

The presets change several fields at once. A controlled ablation changes one field of the default configuration and holds everything else fixed, including the seeds: each arm below runs 40 steps from the same RL base agent with seeds 0, 1, and 2, and is evaluated on the held-out tasks with 8 episodes per task. Five arms make 15 runs.
Output:
Read the table against its own noise. The three seeds of the default arm span 0.71 to 0.85 in late training reward and 0.76 to 0.81 held-out. Removing the standard-deviation normalization, switching to seq_mean_token_mean, and raising ϵhigh\epsilon_{\text{high}} to 0.28 all land inside that spread. Clip-higher could not have done anything else: the clip binds on 0.15% of tokens in this configuration, so the bound it relaxes is almost never reached. The two effects that stand out are mechanisms, not rankings. seq_mean_token_mean produced the longest episodes, 14.9 policy tokens against 12.9 (Section 9.7 follows this up). Dynamic sampling had the highest late reward and held-out pass rate in all three seeds, 0.81 to 0.85, and it sampled 123 episodes per step instead of 64 to get there. Figure 9.2: Training batch reward over 40 GRPO steps for each ablation arm against the default, with the band spanning three seeds (min to max) and the line their mean. RL base agent, anonymized one-edit gym, 64 episodes per step. Figure 9.2 shows the learning curves with their seed bands. The bands of the default and of each single-field change overlap over the whole run; the curves differ by less than seeds do. A real ablation of these choices needs more seeds than this chapter can afford, which Chapter 17 quantifies with paired comparisons. Figure 9.3: Dynamic sampling. Left: fraction of sampled groups whose rewards were all equal, per step, for the default arm (mean of three seeds). Right: training batch reward against the cumulative number of episodes sampled, for the default and dynamic-sampling arms. Figure 9.3 puts dynamic sampling on the axis that matters for its cost. As the policy improves, more groups pass in every attempt, so the share of zero-advantage groups rises and dynamic sampling has to sample more to fill its batch. Against the number of episodes sampled, its curve falls clearly behind the default’s: the default reaches a batch reward of 0.8 after about 2,000 episodes, and dynamic sampling needs about 3,500. It learns more per update and less per rollout. For agents, where a rollout costs far more than an update, rollouts are the axis to compare on.
Evidence. Measured here: three seeds, 40 steps, one 158k-parameter model on 20 training tasks. These are mechanism demonstrations. Differences smaller than the spread across seeds are not differences, and none of these results predicts the effect of the same change on a large model, a long-horizon agent, or a different reward. The reported benefits of these choices in the source papers come from math reasoning at much larger scale (Chapter 20).

​
9.7 Design problems in practice

Problem: group size versus number of prompts at a fixed sample budget. Symptom: at the same number of episodes per step, one configuration learns hard tasks and another stalls on them, or most of each batch carries zero advantage. Cause: a group of GG attempts at a task with pass rate pp has zero variance with probability pG+(1−p)Gp^G + (1 - p)^G. Small groups waste attempts on degenerate groups for tasks that are nearly always solved or failed, and they give noisier baselines. Large groups cover fewer distinct tasks per step, so each step’s gradient is about fewer tasks. Options: (a) large GG for hard, sparse-reward tasks, where a success is rare and must be sampled to be learned from. (b) Many prompts with small GG for broad task distributions with medium pass rates. (c) Adaptive GG per task from its running pass rate. Default: G=8G = 8 with 8 prompts at this scale; measure the zero-variance fraction and the number of distinct tasks per step before changing it. The block below fixes the budget at 64 episodes and measures both quantities for the RL base agent and the trained policy.
Output:
For the RL base agent, larger groups make more groups informative, from 62% at G=4G = 4 to all of them at G=16G = 16, at the price of covering a quarter as many tasks. For the trained policy, the measurement shows a different regime: its pass rate is above 0.9 and its entropy below 0.1, so its attempts at a given task almost always agree, and almost no group carries gradient whatever GG is. At that point, the lever is the task distribution (harder tasks, or dropping solved ones), not the group size.
Problem: zero-advantage groups and filtering. Symptom: as training succeeds, the share of all-pass groups rises, the effective batch shrinks, and progress slows; with token_mean, the zero-advantage episodes still count in the normalizer and dilute every other token’s weight. Cause: a zero-variance group has exactly zero advantage under every baseline of Chapter 7, so its episodes cost rollouts and contribute nothing but tokens to the denominator. Options: (a) dynamic sampling, which keeps the effective batch full and pays in extra rollouts. (b) Filter the zero-advantage groups without resampling, which keeps the cost and shrinks the batch. (c) A curriculum that drops tasks the policy always solves or never solves (Chapter 14). Default: log zero_adv_frac every step; filter without resampling when it is moderate, and add a curriculum when whole tasks saturate. Dynamic sampling pays off when rollouts are cheap relative to updates, which is the opposite of agent RL. The default run’s history shows the trend.
Output:
In the default run, the share of zero-advantage groups rises from 0.19 in the first ten steps to about 0.8 in the last twenty. By the end, four of every five groups sampled were all-pass or all-fail, and token_mean still counted their tokens in its denominator. Dynamic sampling refills the batch, and its cost rises with the same trend, from about 110 episodes per step early to about 150 late, more than twice the nominal 64.
Problem: length growth. Symptom: the mean number of policy tokens or turns per episode rises over training without a matching rise in pass rate, often in failing episodes. Cause: the aggregation sets each token’s weight (Section 6.6). Under seq_mean_token_mean, a failing episode’s total penalty is fixed, so each of its tokens is penalized less when it is longer, and extra calls in failures are cheap. A reward that does not cost turns adds nothing against it. Options: (a) a length-proportional aggregation (token_mean, constant, or seq_mean_token_sum). (b) Overlong shaping or a step cost, which change the task and must leave room for useful information gathering (Chapter 12). (c) Masking truncated episodes. Default: (a), with the lengths of successes and failures logged separately. The ablation histories compare the two aggregations directly.
Output:
Under seq_mean_token_mean, episodes had about 14 policy tokens from the first ten steps on, against 11.6 rising to 13.4 under token_mean, with more turns (4.36 against 4.07) and slightly more reads (1.06 against 1.00 per episode). The difference appears early because the base agent’s failures are long and per-sequence normalization charges each of their tokens less. In 40 steps on a six-turn budget the effect is small; in agent episodes with hundreds of turns, it is the mechanism Dr. GRPO identifies as response-length bias.
Problem: choosing the loss aggregation. Symptom: two runs that differ only in a “normalization detail” learn at different speeds or drift in length, and a change of batch composition changes the effective learning rate. Cause: token_mean couples the per-token weight to the batch’s total length; seq_mean_token_mean weights trajectories equally and tokens of long ones less; constant and seq_mean_token_sum keep per-token weights fixed and need the learning rate tuned for their scale. Options: the four modes of aggregate_loss. Default: token_mean for this book, because it is length-neutral within a batch, widely used (DAPO), and its batch-dependent scale is easy to monitor through the logged length; constant (Dr. GRPO) when lengths change a lot during training, since it removes even the batch coupling. Avoid seq_mean_token_mean for multi-turn agents, whose episode lengths vary by an order of magnitude and correlate with success; the previous block measured its effect on this gym.

​
Summary

  • GRPO is PPO’s clipped surrogate with a group baseline instead of a critic and, in DeepSeekMath’s form, a per-sequence token mean and a KL penalty in the loss. Its published variants are diffs in a handful of fields; RLConfig and train_rl implement them as options of one loop.
  • On the gym as built, GRPO reproduced Chapter 6’s memorization: 0.98 on training tasks, 0.40 held-out, zero reads, a first edit chosen from the function name.
  • Shortcut features live in the environment. Renaming functions per episode removed the name shortcut and most of the train/held-out gap, and the policy switched to stacking edits, which a blind sequence can push to 0.60. Limiting episodes to one edit removed that shortcut and exposed an exploration failure: the base agent almost never followed a read with an informed edit, so there was nothing to amplify.
  • A base agent with one informed demonstration in nine, at the same pass rate, learned read, edit, test, submit within 80 steps and reached 0.84 on held-out tasks, on the anonymized gym and, as a control, on the original one. The starting policy decides whether a skill is learned; the environment decides whether a pass rate certifies it. The anonymized one-edit gym with make_rl_base is the book’s default from here on.
  • At tiny scale, the single-field ablations of std normalization, aggregation, and clip-higher were within seed noise, and clip-higher had nothing to act on with a clip fraction of 0.15%. Two mechanisms were visible: seq_mean_token_mean lengthened episodes, and dynamic sampling learned more per update while sampling 1.9 times as many episodes, a cost that grew as zero-advantage groups rose to 80% of the batch. GSPO with its paper’s ranges clipped 31% of sequences on 12-token episodes. These are mechanism demonstrations, not benchmark evidence.

​
Exercises

  1. Show that with one minibatch and one epoch, GRPO’s update equals the REINFORCE update of Chapter 6 with a group-normalized advantage. Then show what changes with two minibatches. Which tokens of the second minibatch can be clipped?
  2. Change an assumption: rerun Section 9.4’s default with make_base_agent(tok, kinds=('random', 'blind') * 8 + ('oracle',)), one informed demonstration in 17. Does the policy still learn to read? Find the smallest informed share at which it does within 80 steps, and relate it to the measured read-then-right probability of the starting policy.
  3. The renaming draws names from all 30 gym functions, including the held-out ones. Restrict the pool to training-task names and re-measure the held-out pass rate of the default run. What does the difference tell you about the role of the held-out names’ embeddings?
  4. Implement CISPO (Chapter 20) as ratio_level='cispo': the loss −sg(min⁡(ρ,1+ϵhigh)) A^log⁡πθ-\mathrm{sg}(\min(\rho, 1 + \epsilon_{\text{high}}))\, \hat A \log \pi_\theta per token. Which tokens keep a gradient that PPO’s clip would remove? Compare its clip-related statistics with the default on the run of Section 9.6.
  5. Make the KL term matter: train the default configuration with kl_coef in {0.01, 0.1, 1.0} and log kl_ref and the pass rate. At what coefficient does the policy stop learning to read? Chapter 10 explains why the answer depends on kl_kind.
  6. Measure the GSPO preset’s clip fraction and the spread of its sequence ratios. Are the paper’s ranges, chosen for long math outputs, appropriate for 12-token episodes? Derive how the standard deviation of log⁡si\log s_i scales with the number of policy tokens if the token log-ratios were independent.
  7. Design review: a team’s SWE agent reaches 70% on its training repositories and 35% on a held-out set, and its trajectories rarely open files before editing. Using Section 9.4, list the shortcut features you would look for in the task data and the environment, the behavior statistics you would ask for, and the two kinds of fix you would consider.