Skip to main content
A BugFix episode has about four calls and seventy tokens. A SWE agent’s episode has tens to hundreds of calls, and its context fills with file contents, test logs, and stack traces that the policy did not write. Most of what changes between the two is not the objective. GRPO’s loss is the same; what changes is everything around it: which call deserves credit for the outcome, how many calls the agent is allowed and what each one costs, what the policy actually sees when the history no longer fits, and what happens when a call is malformed or an episode is cut off by a limit. This chapter studies those questions on the BugFix gym, where each of them can be changed with one argument and measured with the behavior statistics of Chapter 9. We first make the invariant that holds the multi-turn machinery together explicit: observation tokens are inputs, not actions. We then compare turn-level and token-level credit in training, vary the turn budget and the step penalty and watch information gathering change, break the single-sequence view of an episode with context management, and handle tool errors, early termination, and mixtures of single-turn and multi-turn data. The design problems at the end reproduce the four classic pathologies of agent RL: submitting immediately, never submitting, truncated episodes, and context overflow.

​
12.1 Observations are inputs, not actions

A multi-turn trajectory interleaves two kinds of tokens. Policy tokens were sampled by πθ\pi_\theta; they are the actions, and the policy gradient of Chapter 6 is a sum over them. Observation tokens (tool output, and the template tokens around it) were produced by the environment; the policy conditions on them but never chose them. The derivation of Section 6.1 makes the consequence exact: the environment’s transition probabilities do not depend on θ\theta, so their log-probabilities drop out of ∇θlog⁡pθ(τ)\nabla_\theta \log p_\theta(\tau), and a loss term on an observation token would be a gradient toward predicting the environment, which is a different objective. The action mask of Chapter 5 enforces this. We check it directly: compute a policy-gradient loss on a batch, differentiate with respect to the logits, and count which positions receive any gradient, by owner.
Output:
Only the policy positions receive gradient; every prompt, template, and tool position has an exactly zero gradient. The invariant matters more as episodes grow, because the share of the context that the policy wrote shrinks. We run the RL base agent with turn budgets of 6, 12, and 24 and measure that share, the context length, and how often an episode ends because the next call would not fit in TinyAgent’s 256-token context.
Output:
The RL base agent rarely uses a large budget, because it submits early: mean episode length grows only from 38 to 49 tokens when the budget goes from 6 to 24 turns, while the longest episode grows from 144 to 249 tokens, and at 24 turns 1% of episodes run out of context. In every setting only 30% of the predicted positions are the policy’s own; the other 70% are prompt, template, and tool tokens that carry no gradient. In a SWE agent whose observations are file contents and test logs, the policy’s share is a few percent, so a mask bug that trains on observations would put almost all of the loss on predicting the environment.

​
12.2 Credit assignment across turns

With an outcome reward and a group baseline, every policy token of an episode receives the same advantage (Section 7.6). The only way the decisive edit can earn more credit than a useless test is by comparison across episodes: the useless call appears in successes and failures alike, so on average its probability does not change. A per-turn reward makes the question sharper. With a cost cc per call, the return of an episode with KK calls that ends in verdict v∈{0,1}v \in \{0, 1\} is R(τ)=v−cK=∑k=0K−1rk,rk=−c+v [k=K−1],R(\tau) = v - cK = \sum_{k=0}^{K-1} r_k, \qquad r_k = -c + v\,[k = K-1], and the cost of a call is known the moment it is made. An outcome advantage A(τ)=R(τ)−bA(\tau) = R(\tau) - b still shares the total among all calls. A turn-level advantage AkA_k built from rkr_k and a value V(sk)V(s_k) at the start of each call can, in principle, charge each call for its own cost and credit it with the change in value it causes. per_call_rewards lays the rkr_k out on tokens: −c-c on the last token of every call and the verdict on the episode’s last policy token. masked_whiten normalizes advantages over policy tokens, as PPO implementations do for GAE advantages (Section 8.4).
We train three arms with the same reward, the hidden verdict minus 0.05 per call, from the same RL base agent: outcome GRPO (the book’s default loop), token-level GAE, and turn-level GAE, both with λ=0.95\lambda = 0.95 and a critic. The critic is copied from the policy and warmed up for 60 steps on 192 base-agent episodes, as Section 8.4 recommends, and then takes one regression step per training step on the batch’s returns-to-go.
train_credit is train_rl with the advantage computation swapped for a critic-based one: per-call rewards, critic values, GAE at the token or the turn level, whitening, the same clipped update, and one critic step.
Output:
All three arms learn to read. They differ in what they do with the cost. Outcome GRPO removed the useless test call almost entirely (0.01 per held-out episode), and it also learned something we did not want: it applies an edit in only 78% of episodes, submitting after a read without editing when it is unsure, because a failure with fewer calls costs less, and its held-out pass rate is 0.656. The two critic-based arms kept more of the cost-free habits: token-level GAE kept testing in nearly every episode (0.97 tests) and turn-level GAE in about a third (0.29), and both passed more held-out episodes, 0.787 and 0.769. The per-turn cost of 0.05 is small next to the noise of a critic whose explained variance is a few percent (Section 7.4), so the critic arms mostly ignored it, while the group comparison of outcome GRPO saw it directly, in episodes of the same task that differed in length.
Evidence. Measured here, one seed per arm, 40 steps. The arms’ differences in tests are not robust: in exploratory runs that differed only in floating-point summation order (thread count), token-level GAE dropped the test call too. What the comparison supports is the mechanism: a turn-level or token-level advantage can only charge a call for its own cost if the critic resolves that cost, and outcome advantages charge every call of a longer episode for the length. Figure 12.1 shows one episode’s turn-level credit.
Figure 12.1: Turn-level credit on one successful held-out episode of the turn-GAE policy: per-call reward, the critic's value at the start of each call, the TD error, and the turn-level advantage (unwhitened). The text below compares the spread of token-level GAE within each call. Figure 12.1 reads one episode of the turn-GAE policy (read, fix_comparison, test, submit) through its own critic. Every call costs 0.05, and the submit carries the verdict. The critic’s TD errors say that the read and the decisive edit lowered the value (−0.116 and −0.031) and that the useless test and the submit raised it (+0.070 and +0.305): after 40 steps the critic still assigns credit almost backward, as Chapter 7’s critic did. With λ=0.95\lambda = 0.95 the turn advantages chain the later TD errors back, so every call ends up positive, from +0.180 for the read to +0.361 for the test. Token-level GAE on the same episode also varies within each call, by up to 0.17 inside the edit, which is pure critic noise. This is the general situation: turn-level credit is exactly as good as the values at turn boundaries, and a value model trained on a few hundred binary outcomes does not supply good ones.

​
12.3 Turn budgets and step penalties

Two numbers define how much an agent may do: the turn budget (max_turns), a hard limit after which the episode ends unsubmitted, and the step penalty, a soft cost per call in reward units. Section 2.9 solved the repair MDP exactly and found that the optimal policy keeps its shape for costs up to 0.3 and gives up above 0.4. A learning agent does not face the optimal policy’s trade-off; it faces its own: a call is worth paying for only if the policy’s current use of it raises the expected verdict by more than the cost. We train the default configuration with a budget of 3 turns instead of 6, and with a step penalty of 0.3, and compare them with the default and with the 0.05 penalty of Section 12.2. Every arm uses the hidden-test verdict, the RL base agent, and 40 steps.
Output:
The budget and the penalty each produce a different agent from the same starting point. The default keeps an unneeded test in 80% of episodes, because it costs nothing. A penalty of 0.05 per call removes the test and some edits along with it (Section 12.2). A penalty of 0.3 produces the degenerate agent of Section 2.9: it submits on its first call in every held-out episode, with pass rate 0 and reward −0.30, the cost of one call. And a budget of 3 turns produces a blind guesser: no reads, one edit, a submit, and a pass rate of 0.200, exactly the best single blind op of Section 9.4. Read, edit, submit fits in 3 turns, but with no slack: the starting policy’s rare reading episodes almost always run out of turns before submitting, so reading earns less than guessing and is extinguished. (In an exploratory run that differed only in thread count, the 3-turn arm did learn to read and reached 0.41; a budget with no slack makes the outcome depend on early luck.) Figure 12.2: Behavior over training under four budgets and penalties: hidden-test pass rate, reads, tests, and turns per episode in the training batch (5-step moving average). RL base agent, 40 GRPO steps of 64 episodes, one seed. Figure 12.2 shows the dynamics. Under the 0.3 penalty, turns collapse to 1 within about 10 steps, before the pass rate has any chance to rise. Under the 3-turn budget, reads fall to 0 within the first 10 steps while the pass rate stays near the blind ceiling of 0.2. The default overshoots to about 1.4 reads per episode by step 12 and settles at 1. The 0.05 penalty learns to read more slowly, reaching one read per episode only around step 25, because every read is charged before its benefit is learned; its tests rise with the default’s at first and then collapse after step 27, once reading is established and the test is the only call left whose cost buys nothing. A cost shapes how a skill is executed once the skill exists, slows its discovery when it is small, and prevents it when it is large relative to the starting policy’s success rate.

​
12.4 Context management changes the policy’s input

A long episode eventually does not fit in the context window. Production agents manage the context in three main ways: truncation (drop the oldest turns), observation masking (keep every call but replace old tool outputs by a placeholder), and summarization (replace a span of history by a model-written summary). Lindenbauer et al. (2025) report that, for SWE-agent on SWE-bench Verified, simple observation masking halved cost relative to the raw agent and matched LLM summarization’s solve rate. Whatever the method, it changes what the policy conditions on, and therefore two things that the single-sequence view of Chapter 5 took for granted. First, the input distribution changes. A policy trained with full histories has never seen a history with a placeholder where the source code was. Second, an episode is no longer one sequence. With full context, the prefix of call kk is the prefix of call k+1k+1, so one forward pass over the whole episode scores every call. With masking, call k+1k+1 is generated from a prefix that differs from the one call kk saw, so each call is a separate (prompt, completion) sample, and the trainer must score each call under the exact prefix the sampler used, or it computes probabilities of tokens under contexts that never produced them, the retokenization-drift failure of Section 5.4 in another form. context_view implements observation masking on the token log of an Episode: before call kk, it drops the tool tokens of every turn older than the last keep_obs, keeping the <obs> </obs> template tokens as the placeholder. keep_obs=None is the full context.
context_rollout_batch is Chapter 5’s lockstep sampler with one change: each call is generated from context_view of the episode so far, and the (view, call) pair is recorded. It returns the full episodes, for statistics, and the per-call samples, each tagged with its episode’s index, for training.
Two checks pin down correctness. With keep_obs=None and the same generator, the sampler must reproduce rollout_batch exactly, and the per-call samples, scored separately, must give the same log-probabilities as the episodes scored as single sequences.
Output:
With the full context, the per-call sampler reproduces the single-sequence sampler token for token, and the 61 calls of the 16 episodes, scored as separate samples, give the same log-probabilities as the episodes scored whole, up to 2.1×10−62.1 \times 10^{-6}, float32 summation order. Under masking only the first check changes: the samples differ from any single sequence, and only the per-call scoring is correct. Training on per-call samples needs one more decision: the advantage. The verdict belongs to the episode, so we compute group advantages over episodes, as before, and give every call of an episode its episode’s advantage. With token_mean aggregation this is the same objective as the single-sequence update: the same tokens, the same advantages, the same normalizer. Only the contexts differ.
evaluate_context samples held-out episodes under a given context policy. We cross two training conditions, full context (the default run of Section 12.3) and observation masking with keep_obs=1, with three evaluation conditions: full, keep_obs=1, and keep_obs=0, which removes every tool output.
Output:
Read the table along the rows. The policy trained with full context and evaluated with masking keeps its pass rate (0.731 to 0.725) and changes its behavior: tests fall from 0.82 to 0.05 per episode, and reads rise from 1.01 to 1.27, because after the edit it sees a history in which the source was replaced by a placeholder, an input it never saw in training. The policy trained with masking and evaluated with full context loses most of its advantage: its pass rate falls from 0.794 to 0.500, and its episodes lengthen to 5.29 turns, as it meets the source still visible after its edit and does not know how to proceed. Each policy is best under the context it was trained with, and the mismatch is not symmetric. With every tool output removed (keep_obs=0) both fail completely, which confirms that the information comes from the observations. The default setting of context_rollout_batch is the full context, and Chapter 5’s sampler remains the book’s default; any context policy must be applied identically in training and evaluation, which Figure 12.3 summarizes. Figure 12.3: Held-out pass rate and tests per episode for each pair of training and evaluation context policies. Rows: how the policy was trained; columns: how it is evaluated.

​
12.5 Tool errors, malformed calls, and early termination

A call can fail in three ways: it does not parse (<call> edit </call> with a missing argument), it names a tool or argument that does not exist, or it is valid and the tool fails (a test run that crashes, a timeout). The gym answers the first two with an error observation and charges a turn, which is the design of most agent harnesses; the Model Context Protocol, for example, returns tool failures as results flagged isError so the model can see them and recover (Section 4.7). The alternative is early termination: end the episode on an error, as failed. It saves the compute of an episode that has gone wrong and gives a sharper signal against malformed output, and it removes any chance to learn recovery. EarlyStopEnv ends an episode unsubmitted after max_invalid errors.
How much is at stake depends on how often the policy errs and whether it recovers. We evaluate the RL base agent on the training tasks in both environments, and read the malformed-call rate of the default run from its environment log at the start and end of training.
Output:
The base agent makes an error in about 5% of episodes. With error observations, those episodes recover and pass at 0.143, about the base agent’s overall rate; with early termination they pass at 0 by construction. The overall pass rates (0.131 and 0.144) differ only by sampling noise, because the two environments produce different episodes after the first error, and early stopping saved about 6% of the policy tokens. The training log shows a less obvious fact: GRPO barely reduced malformed calls, from 1.9% to 1.4% of turns over 40 steps. A malformed call costs a turn, and the default budget of 6 has two turns of slack over the 4 the policy needs, so an error has almost no effect on the reward and almost no gradient against it. Format errors disappear under RL only when they cost reward: a tight budget, early termination, or make_reward(invalid_penalty=...).

​
12.6 Mixing single-turn and multi-turn data

Agent post-training rarely uses agent episodes alone. Single-turn data (a prompt with all the information and one answer, as in math RLVR or code generation from a specification) is cheaper to sample, has no environment latency, and teaches the same underlying skill. The BugFix version of a single-turn task gives the source in the prompt and grades the first call: single_turn appends the read projection of the source to the issue, and MixedTurnEnv grades such tasks after one call, as if submitted, while running every other task as a normal episode. The environment decides by inspecting the task, so one factory serves a mixed batch.
A policy trained only on multi-turn episodes meets a single-turn task as an unfamiliar input. We evaluate the default run’s policy on held-out tasks in both formats, then train one policy on a 50/50 mixture of the two formats over the training tasks and evaluate it the same way.
Output:
The policy trained only on multi-turn episodes passes none of the single-turn tasks: on the prompt that already contains the source it never reads (0.00 reads), and its first call, which ends the episode, is never the right edit, because a single call graded immediately is a situation its training never contained. The mixture-trained policy passes 25.6% of single-turn held-out tasks, above the 0.20 of a blind guess, and pays for it with a lower multi-turn pass rate, 0.719 against 0.762, after the same 40 steps with half as many multi-turn episodes. Reading the op off a source in the prompt and reading it off an observation are the same skill for a large pretrained model; for TinyAgent, whose two positions differ, they transfer only partly. The mixture also changes how much each episode weighs in the loss. Under token_mean, an episode’s weight is proportional to its number of policy tokens (Section 6.6), and a single-turn episode has one call while a multi-turn one has about four. The block samples a batch with 4 tasks of each format from the mixed policy and measures the share of the loss weight that the single-turn episodes receive under token_mean and under per-sequence averaging.
Output:
Half the episodes are single-turn, and they receive 19% of the token_mean loss weight, because each has one call against about four in a multi-turn episode. Per-sequence averaging would give them 50%. The mixture ratio that matters for the gradient is therefore the ratio of policy tokens, not of episodes, and it shifts as the multi-turn episodes get shorter or longer during training.

​
12.7 Design problems in practice

Problem: the agent learns to submit immediately. Symptom: turns per episode fall to 1, the pass rate falls to 0, and the reward settles at minus one step cost. Cause: with a step cost cc, an attempt that takes KK calls and passes with probability pp is worth p−cKp - cK, while submitting at once is worth −c-c. The policy compares these at its current pp and KK, not at the optimal ones, so a cost that the trained agent could afford (in Section 2.9’s MDP the optimal policy gives up only above 0.4) is ruinous for a starting policy with p≈0.17p \approx 0.17. Options: (a) a much smaller cost, or none until the pass rate is high. (b) A cost that ramps in over training. (c) Charge cost only on calls beyond a free allowance. (d) Report cost as a separate metric and select checkpoints on it instead of training on it. Default: (a) then (b), with the arithmetic below checked against the starting policy. The 0.3 penalty run of Section 12.3 is the reproduction.
Output:
For the starting policy, an attempt is worth 0.17−0.3×3.64=−0.920.17 - 0.3 \times 3.64 = -0.92 and submitting at once −0.30-0.30, so the group comparison favors submitting immediately in almost every group. Turns per episode fall from 1.82 in the first ten steps to 1.02 in the second ten and stay there; once no episode attempts a repair, no group contains a success and the policy can never find out that repairs pay.
Problem: the agent never submits. Symptom: the submit rate falls, episodes run to the turn limit, and the reward rises toward 0 while the pass rate does not. Cause: an episode stopped by the turn limit is scored by the reward function as unsubmitted, which is 0 under the hidden verdict. If a wrong submission is punished (here by −1-1), not submitting is safer than submitting at the current pass rate, and the policy learns to wait out the budget. The truncation outcome is part of the reward whether or not anyone designed it. Options: (a) make an unsubmitted episode at least as bad as a failed submission. (b) Exclude truncated episodes from the update (next problem). (c) Never punish failure below the truncation value. Default: (a): the outcome of running out of turns should be the worst outcome available. We train with wrong_submit=-1 and log the submit rate.
Output:
Within 10 steps the submit rate falls from 0.72 to 0.12, the episodes run to 5.78 turns of the 6 allowed, and the pass rate falls to 0.11, while the reward rises from −0.38 to +0.09: by not submitting, the policy avoids the −1. The run recovers here, because the policy’s reading skill is improving underneath (a submission that passes is worth +1, more than the 0 of waiting), and by step 39 it submits in 73% of episodes and passes 73%. On a harder task, where the pass rate stays low, nothing would pull it back.
Problem: timeout-truncated episodes. Symptom: runs with tight turn or token limits learn slowly, learn to rush, or (as above) learn to stall; a change of limit changes the learned behavior. Cause: a truncated episode has no verdict. Scoring it as a failure (the gym’s default) teaches the agent the limit; scoring it otherwise is a choice about what the limit means. Pardo et al. (2018) distinguish the two cases: if the limit is part of the task, the remaining time belongs in the agent’s input and truncation is a real terminal outcome; if the limit only exists to make training feasible, the agent should bootstrap from the value of the state where the episode was cut, R^=∑k<Krk+V(sK)\hat R = \sum_{k<K} r_k + V(s_K), which requires a critic. Options: (a) treat truncation as failure, and show the remaining budget in the observation. (b) Mask truncated episodes out of the loss, as DAPO’s overlong filtering and DeepSWE’s compact filtering do (Chapter 20). (c) Bootstrap with a critic. Default: (a) when the limit is the task’s (a coding agent with a real deadline); (b) only when the limit is an engineering cap that the policy cannot use to avoid a penalty; (c) only with a critic whose explained variance has been measured. The block retrains the never-submit setting with truncated episodes masked: their advantages are set to 0, and the group baselines use only the episodes that submitted.
Output:
Masking made it worse. The submit rate fell to 0.09 and stayed there, and the held-out pass rate is 0.075 against 0.706 when truncation is scored as 0. With truncated episodes masked, not submitting has no consequence: those episodes carry zero advantage, while every failed submission pushes the probability of submit down and nothing pushes it back up. Stalling became free. Masking a truncated episode removes the signal against truncation, which is what DAPO’s overlong filtering intends for responses that were cut off while still productive, and what a policy will exploit when truncation is a choice it can make. Use (b) only when the policy cannot reach truncation on purpose more cheaply than by working, or with a separate cost on truncation.
Problem: context overflow. Symptom: a few long episodes end with stop='truncated' before any verdict, memory spikes on the longest sequences, and long-horizon tasks fail more than their difficulty suggests. Cause: the context grows with every observation; tool output is unbounded in real environments (Section 4.8), and the window, here 256 tokens, is fixed. Section 12.1 measured how the risk grows with the turn budget. Options: (a) cap each observation (head and tail with a marker). (b) Observation masking or summarization (Section 12.4), which turns an episode into per-call samples and changes the input distribution; train with the same context policy you deploy. (c) A larger window, paid in memory and attention cost. (d) Treat overflow as truncation and apply the previous problem’s rule. Default: (a) always, (b) once episodes routinely exceed the window, and evaluation under exactly the training context policy, which Section 12.4 showed is not optional.

​
Summary

  • Observation tokens are inputs: a policy-gradient loss on a real batch put nonzero gradient on all 201 policy positions and on none of the 669 prompt, template, and tool positions. Only 30% of a BugFix episode’s tokens are the policy’s; in long SWE episodes it is a few percent.
  • With a per-call cost, outcome GRPO removed a useless test call by comparing episodes of different lengths and also learned to skip edits when unsure (held-out 0.656); critic-based token- and turn-level GAE kept more of the useless calls and passed more (0.787, 0.769). A turn-level advantage can charge a call for its own cost only if the critic resolves it.
  • Costs and budgets select different agents from one starting policy: a 0.3 step penalty made the agent submit immediately (pass 0), and a 3-turn budget with no slack extinguished reading (pass 0.200, the blind ceiling). Compare costs with the starting policy’s success rate, not the optimal policy’s.
  • Context management makes each call a separate sample that must be scored under the exact context the sampler used (checked to 2×10−62 \times 10^{-6}), and it changes the input distribution: a masked-context policy evaluated with full context fell from 0.794 to 0.500. Train and evaluate under the same context policy.
  • Malformed calls cost no reward when the budget has slack, so RL barely removed them (1.9% to 1.4% of turns). Mixing single-turn and multi-turn tasks taught the single-turn format (0.256 vs 0.000) at some multi-turn cost, and under token_mean single-turn episodes got 19% of the weight at 50% of the episodes.
  • Truncation is part of the reward: with a −1 for failed submissions the agent stopped submitting for a while, and masking truncated episodes out of the loss made stalling free (held-out 0.075). All results are single-seed mechanism demonstrations on a 158k-parameter model.

​
Exercises

  1. Show that, with per-call costs and γ=λ=1\gamma = \lambda = 1, turn-level GAE gives call kk the advantage ∑j≥krj−V(sk)\sum_{j \ge k} r_j - V(s_k). Under what condition on the critic does it charge a useless test call exactly its own cost?
  2. Change an assumption: rerun the max_turns 3 arm of Section 12.3 starting from a policy first trained for 20 steps with max_turns=6. Does a policy that already reads keep reading under the tight budget? What does this say about curricula over budgets?
  3. Implement summarization for the context policy of Section 12.4: replace the source observation by the name of the op that arl.infer_op would choose. Train with it and evaluate under full context. Which context policy transfers better, masking or summarization, and why?
  4. Add the remaining budget to the observation (“turns left 2”) in a copy of EditBudgetEnv, as Pardo et al. recommend when the limit is part of the task, and repeat the wrong_submit=-1 run. Does the policy still stall?
  5. In the mixed run of Section 12.6, compare loss_mode='token_mean' with 'seq_mean_token_mean'. Which format improves faster under each, and how does that follow from the weight shares measured above?
  6. Bootstrapping: using the critic of Section 12.2, compute V(sK)V(s_K) for budget-truncated episodes of the never-submit run and train with R^=∑krk+V(sK)\hat R = \sum_k r_k + V(s_K) for truncated episodes. Compare with the two treatments of Section 12.7.
  7. Design review: a SWE agent trained with a 50-turn budget and full context is deployed with a 100-turn budget and observation masking after 30 turns, and its solve rate falls. List the distribution shifts involved, the measurements that would separate them, and the training changes you would propose.