Skip to main content
Every chapter so far has trained on one reward: 1 if the agent submitted and every hidden test passed, 0 otherwise. Chapters 6 to 10 took it as given and asked how to turn it into a gradient. This chapter asks where it comes from and what it certifies. A policy gradient increases the probability of whatever earned reward in the policy’s own samples (Section 9.4), so the reward is the specification of the task. Wherever the reward and the intended behavior disagree, a strong enough optimizer finds the disagreement. For coding agents the best-known disagreement is special-casing the tests instead of fixing the code, and the BugFix gym was built with that exploit, hardcode_tests, on its edit menu. We first write the reward as a function of the episode’s terminal record and compare the verifiers a coding environment can offer. We then reproduce reward hacking: train on the public tests the agent can see, watch the training reward climb while the hidden pass rate does not, and read the dashboard signature. Three fixes follow, together with the rule behind them: the reward used for training and the measurement used for evaluation must not share a weakness. The second half covers rewards that are learned or judged rather than executed, process rewards, and potential-based shaping, the one form of reward shaping that provably leaves the optimal policy unchanged.

​
11.1 Verifiable rewards

A verifiable reward is computed by a program that checks the outcome, here by executing tests, rather than by a model that predicts quality. Its value is that it is cheap to trust: if the tests are right and the agent cannot see or change them, a pass means the code behaves correctly on those inputs. Its two limits are coverage, since a test suite checks finitely many inputs, and exposure, since anything the agent can observe it can learn to satisfy without solving the task. BugFixEnv calls its reward hook once, at the end of an episode, with the terminal info dictionary of Section 4.5. Writing the reward as a function of that record separates the verifier, which scores the final state, from reward terms that score the episode’s conduct. VERIFIERS holds four verifiers: all hidden tests (the book’s default), all public tests (the ones the test tool shows), and the fraction of each suite that passes. make_reward adds three optional terms: a penalty when a known exploit appears in the edit log, a format penalty per malformed call, and a cost per turn, which Chapter 12 studies. A fourth argument sets the reward of a submission that fails its verifier.
The returned function plugs into RLConfig(reward=...) or any BugFixEnv. We score the scripted policies of Section 4.6 under the four verifiers, plus one more policy that submits immediately without touching the code. Each policy plays 5 episodes on each of the 20 training tasks.
Output:
Read the table by columns. The hidden verifier ranks the policies as intended: the oracle at 1, guessing near 0.2, and the hacker at 0. The public verifier gives the hacker the oracle’s score, 1.000, because hardcode_tests makes exactly the visible tests pass. The two fractional verifiers give credit to code that was never fixed: the buggy source already passes part of each suite, so submitting immediately earns 0.397 of the hidden suite without a single edit, and the blind guesser earns more partial credit (0.538) than its binary pass rate (0.200) by a factor of 2.7. The hacker’s fractional hidden score equals the do-nothing score, because hardcode_tests only adds lines for the public inputs. The random policy shows the remaining distinction: its public pass rate (0.310) is twice its hidden one (0.160), because one of its nine calls is the exploit. Each verifier defines a different task, and only the first one is the task we want.

​
11.2 Dense and sparse rewards

A binary reward is sparse: most episodes of a weak policy earn exactly 0, and with a group baseline a group whose attempts all fail has zero advantage (Section 7.5). A dense reward assigns different values to more of the episodes, so more groups carry gradient. Partial credit is the obvious way to densify a test-based reward. We sample one batch of 8 groups of 8 from the RL base agent and score the same episodes with the binary and the partial hidden-test verifiers.
Output:
The base agent’s mean binary reward is 0.125, and 75% of its groups contain at least one success and one failure. The partial verifier takes 7 distinct values instead of 2 and makes 95% of the groups informative. That is the case for density: more of the batch carries gradient. Its mean reward, 0.373, is three times the binary one, and almost all of the difference is credit for tests that the unfixed code passes anyway, as the table of Section 11.1 showed. Density has a price that the group normalization of GRPO makes visible. Dividing by the group’s standard deviation rescales any nonzero spread to unit size, so a group whose attempts all fail, but by slightly different fractions of the hidden tests, receives advantages as large as a group with one real success. The block finds such a group in the batch and prints its partial rewards and GRPO advantages.
Output:
In this group no attempt fixed the bug. Six attempts submitted unfixed code and scored 0.62, the share of hidden tests that code passes; two did not submit and scored 0. Std-normalized, the six submissions receive +0.54+0.54 each and the two non-submissions −1.62-1.62: the gradient pushes hard toward submitting unfixed code. Under the binary verifier this group has zero advantage and teaches nothing, which is the right answer for a group without a fix. Does partial credit help or hurt learning here? Answering requires training, so we write the two helpers that every experiment in this chapter uses: run trains a copy of the RL base agent for 40 steps of the default configuration with a given reward, evaluating on held-out tasks every 10 steps, and held_out measures the hidden-test pass rate of a policy on the held-out tasks. Section 11.7 runs the partial-credit comparison; the binary hidden-test run of Section 11.4 is the reference for the whole chapter.
run refers to LoggedEnv, which Section 11.3 defines before it is first called with a log.

​
11.3 Reproducing reward hacking

To watch hacking happen we need the quantities the training reward does not show: the hidden pass rate, and how often the exploit is used. LoggedEnv wraps any environment and appends each finished episode’s terminal record, with its reward, to a list. log_by_step cuts that list into one slice per training step, using the episode counts in the history and skipping the evaluation episodes that train_rl runs through the same factory. reward_dashboard turns a slice into the three numbers an engineer should plot for any coding reward: the rate at which the training verifier passes, the rate at which an independent verifier passes, and the share of episodes with a known exploit.
The hacking run is the book’s default configuration with one change: the reward is the public-test verifier. Everything else, including the anonymized one-edit gym and the starting policy, is as in Chapter 9. We evaluate on held-out tasks every 10 steps; eval_pass is always the hidden-test pass rate, whatever the training reward is.
Output:
Within 10 steps the exploit is in 78% of the batch, and by step 15 in 97%. The training reward rises from 0.234 to 1.000, and the public pass rate tracks it exactly, because it is the reward. The hidden pass rate rises at first, to 0.297 at step 5, while the policy still reads and fixes some bugs alongside the exploit, then falls to 0 as the exploit crowds everything else out. Reading falls from 0.31 calls per episode to 0. On held-out tasks, which the policy never trained on, the hidden pass rate is 0.175 at step 0 and 0 from step 20 on: the exploit is not specific to the training tasks, because hardcode_tests works on any task. Compared with the 0.20 that a blind guess gets, the hacking run ends below the policy it started from. Figure 11.1: Reward hacking on the BugFix gym. Top: public-test pass rate (the training reward) and hidden-test pass rate of the training batch per step, under the public-test reward. Bottom: share of episodes that apply hardcode_tests, under the public-test reward and under the same reward with an exploit penalty. RL base agent, default configuration, 40 steps of 64 episodes. Figure 11.1 is the dashboard signature of reward hacking, and every element of it generalizes. The training reward is a smooth, rising, low-variance curve; nothing about it looks wrong. The independent measurement (hidden tests) moves in the opposite direction after a brief rise. A behavior statistic, here the share of a named exploit, rises in step with the reward. On a real dashboard the second and third curves exist only if someone built them: the hidden-test pass rate requires a second verifier, and the exploit share requires knowing what to look for.

​
11.4 Fixing a hacked reward

There are three families of fixes, and they work at different places. The first changes the verifier: grade with tests the agent cannot observe. The second adds a detector: recognize known exploits in the episode and penalize or discard them. The third changes monitoring: measure an independent signal during training, so that hacking is seen even when the reward cannot be fixed. We train the first two and compare them with the hacking run; the hidden-test run is the book’s default configuration.
Output:
Grading with hidden tests removes the exploit’s payoff, and the policy essentially never uses it (a share of 0.00 over its last 640 episodes); its held-out pass rate is 0.762, the book’s default result at 40 steps. The penalty run keeps the hackable public verifier and subtracts 1 whenever hardcode_tests appears, which makes the exploit worth 0. It learns to read and fix just as well, 0.719 held-out, a difference within the single-seed noise of Chapter 9’s ablations. Its last-10-step public and hidden pass rates are almost equal (0.811 and 0.809), which is the dashboard telling us that, once the exploit is gone, the public tests in this gym agree with the hidden ones. The penalty works here because the gym has exactly one exploit and the detector recognizes it perfectly. Neither holds for a real agent. Detectors are written for exploits someone has already seen, such as edits to test files, sys.exit(0) before the assertions, or literals copied from test inputs, and a policy under pressure from a penalty is pushed toward the exploits the detector does not recognize. Baker et al. (2025) report this pattern with a learned monitor: adding a chain-of-thought monitor to the training reward of a reasoning model in agentic coding environments helped “at low optimization levels”, and with more optimization the agents learned “obfuscated reward hacking, hiding their intent within the CoT”. A detector is therefore most valuable as a monitor whose output is logged and audited, and least valuable as a reward term that the policy can learn to evade.

​
The train/eval reward split

The three runs differ in one place: whether the reward that drives the update shares a weakness with the measurement that decides whether training worked. In the hacking run, the training reward checked the public tests, and so did nothing else; the only reason we saw the hack is that eval_pass uses the hidden tests. In general:
  • The training reward must be cheap, dense enough to learn from, and as hard to exploit as possible. It is optimized, so every weakness will be found.
  • The evaluation measurement must be independent of the training reward: different tests, different tasks, and, for any learned or judged component, a different model. It is not optimized, so it stays honest only if no gradient ever flows from it.
  • Held-out monitoring during training is what connects them. Plot the gap between the training reward and the independent measurement every few steps. A gap that grows while the training reward climbs is the signature of Figure 11.1, and it is visible within the first 10 steps of the hacking run above.
There is a hierarchy of tests in a real coding environment. Public tests (those in the repository the agent can read and run) are observable. Hidden tests (graded after the episode, outside the sandbox) are not observable, but they are used for the reward, so they are optimized against: a policy can overfit their style, for example by learning which behaviors those suites never check. A third set, never used for any reward, is the only one that measures generalization. SWE-bench’s FAIL_TO_PASS and PASS_TO_PASS tests (Section 4.7) are hidden tests in this sense when used as a training reward, and a benchmark built from them stops being an independent measurement for a model trained on the same harness.

​
11.5 Learned rewards and judges

Tests do not exist for every task. When they do not, the reward comes from a model: a reward model trained on human or automated preferences, or an LLM judge prompted with a rubric. Both trade verifiability for coverage.

​
A Bradley–Terry reward model

The Bradley–Terry model assumes that a preference between two outcomes is a logistic function of the difference of their latent scores, P(τw≻τl)=σ(rϕ(τw)−rϕ(τl))P(\tau_w \succ \tau_l) = \sigma\big(r_\phi(\tau_w) - r_\phi(\tau_l)\big). Fitting rϕr_\phi by maximum likelihood on labeled pairs gives the reward-model loss used in RLHF since Christiano et al. (2017), L(ϕ)=−E(τw,τl)[log⁡σ(rϕ(τw)−rϕ(τl))].\mathcal{L}(\phi) = -\mathbb{E}_{(\tau_w, \tau_l)}\big[\log \sigma\big(r_\phi(\tau_w) - r_\phi(\tau_l)\big)\big]. The loss sees only differences, so rϕr_\phi is identified up to an additive constant, and its scale is set by how separable the pairs are. We build pairs from the RL base agent’s own episodes: within each group, every passing episode is preferred to every failing one, with the hidden tests as the labeler. The reward model is a copy of TinyAgent with a one-unit head, make_critic from Section 7.4, read at the last token of the episode.
We sample 4 batches of groups on the training tasks for training pairs and one batch on the held-out tasks for test pairs, then fit the Bradley–Terry loss for 150 steps.
Output:
The reward model separates 99.3% of its 597 training pairs and 87.5% of 40 held-out pairs, on functions and constants it never saw. With so few held-out pairs that figure has a standard error of about 0.05, but it is far above chance: the model has learned something general about what successful episodes look like. A reward model is used out of distribution as soon as the policy changes: RL moves the policy toward whatever the model scores highly, including behaviors that were rare or absent in its training pairs. We score scripted episodes the reward model never saw: the oracle, blind guesses, and the hacker, plus a reading hacker that reads the file, applies hardcode_tests, tests, and submits.
Output:
On scripted episodes, the reward model ranks the oracle highest (+4.31) and blind guessing lowest (−2.46), as a good proxy should. The hacker that never reads scores +0.41, between the two. The reading hacker scores +3.08, much closer to the oracle than to failure, although none of its episodes passes a hidden test. What the model learned is what successful episodes look like in its training data: a read, followed by an edit, an all-pass test output, and a submit. The reading hacker reproduces every one of those features and fixes nothing. An RL run against this reward model would push probability toward the reading hacker, because its score is high and its behavior is easy to learn. Gao et al. (2022) measured the consequence at scale: as a policy is optimized against a proxy reward model, a gold-standard reward first rises and then falls, and the turning point depends on the reward model’s size and data. The proxy is accurate on the distribution it was trained on and is exploited off it, which is exactly what RL produces.

​
LLM judges

An LLM judge reads the transcript, or the final patch, and returns a score, often against a rubric. It needs no training and covers tasks without tests. It is also a reward computed from what the policy produced, so the policy can optimize the judge’s inputs. Its failure modes are those of the reward model plus its own: surface features (“One Token to Fool LLM-as-a-Judge” (2025) reports that strings such as a lone colon or “Let’s solve this problem step by step.” elicit false positives), sensitivity to length and confident tone, and run-to-run noise. A judge also sees only what the transcript shows. We stand in for a judge with a rubric applied to the decoded transcript: did the agent read the code, run the tests, see them pass at the end, and submit? This is a caricature of an LLM judge, not a model of one; what it shares with a real judge is that it scores evidence visible in the transcript.
Output:
The rubric judge gives the reading hacker 1.00, the oracle’s score, and the plain hacker 0.75: the exploit’s transcript shows a read (for the reading hacker), a test run, all tests passing, and a submission, which is everything the rubric asks for. A real LLM judge reads more than four features, and an exploit must work harder to fool it, but the structure of the failure is the same: the judge scores evidence that the policy produces, and a policy optimized against it learns to produce the evidence.

​
11.6 Process rewards and potential-based shaping

An outcome reward arrives once, at the end. A process reward scores intermediate steps: a process reward model (PRM) trained on step labels, as in Let’s Verify Step by Step (2023) and Math-Shepherd (2023), or a heuristic such as “a test went from failing to passing”. It gives credit where it is earned, which is what turn-level advantages need (Section 7.6). It also changes the task: a policy maximizes the sum of rewards, and a bonus that can be collected repeatedly, or without making progress, becomes a goal of its own. Reward shaping adds a term F(s,a,s′)F(s, a, s') to the environment’s reward. Ng, Harada, and Russell (1999) showed that shaping of the form F(s,a,s′)=γ Φ(s′)−Φ(s),F(s, a, s') = \gamma\,\Phi(s') - \Phi(s), for any function Φ\Phi of the state, leaves the optimal policy unchanged, and that, without further assumptions on the MDP, no other form of FF guarantees this. The sufficiency half has a short proof. Along any trajectory, the shaping terms telescope: ∑t=0T−1γt(γΦ(st+1)−Φ(st))=γTΦ(sT)−Φ(s0).\sum_{t=0}^{T-1} \gamma^t \big(\gamma\Phi(s_{t+1}) - \Phi(s_t)\big) = \gamma^{T}\Phi(s_T) - \Phi(s_0). If Φ\Phi is 0 at terminal states (or γT→0\gamma^T \to 0), the shaped return of every trajectory from s0s_0 is its original return minus Φ(s0)\Phi(s_0). Taking expectations, Q′π(s,a)=Qπ(s,a)−Φ(s)Q'^{\pi}(s, a) = Q^{\pi}(s, a) - \Phi(s) for every policy π\pi, so in every state all actions shift by the same constant, and the argmax, the optimal policy, does not move. A bonus that is not a difference of potentials has no such guarantee. We check this on the repair MDP of Chapter 2. shape adds the expected shaping term γ E[Φ(s′)]−Φ(s)\gamma\,\mathbb{E}[\Phi(s')] - \Phi(s) to every state-action reward. The potential Φ\Phi is a progress score, 0 for unread, larger for read, larger still for a correct edit, and 0 for done. The non-potential alternative pays 0.2 every time the agent reads, a “reward the good habit” process bonus.
Output:
With γ=0.9\gamma = 0.9, the potential-shaped MDP has the original optimal policy in every state: read, edit, submit when the edit is correct, test when it is not, and edit again after a failed test. Its optimal value at unread is the original 0.773 because Φ(unread)=0\Phi(\text{unread}) = 0, and in every state V′∗=V∗−ΦV'^{*} = V^{*} - \Phi to within 3×10−133 \times 10^{-13}, as the proof predicts. The read bonus, a process reward of 0.2 per read, makes reading forever optimal from every state, with value 0.2/(1−0.9)=2.00.2/(1-0.9) = 2.0, more than twice what solving the task is worth. In an LLM agent the same mechanism produces an agent that inspects files indefinitely; the turn budget hides it as “never submits” (Chapter 12). The proof also says what potential-based shaping cannot do for the book’s training loop. With γ=1\gamma = 1 and Φ=0\Phi = 0 at the end of the episode, the shaped return of every episode of a task is its original return minus Φ(s0)\Phi(s_0), a constant for that task. A group baseline subtracts it. Under outcome-level GRPO, a potential-based shaping term has no effect at all on the gradient; its value appears only when credit is assigned per turn, by a critic or a turn-level advantage that sees the individual terms (Chapter 12). And a shaping term that looks potential-based but uses a potential the agent can manipulate, or one that is not 0 at the end, is simply a new reward. The block makes this concrete in the gym: Φ\Phi = the fraction of public tests that pass in the current source, computed after every call of an oracle episode and a hacker episode.
Output:
The oracle’s potential rises from 1/3 (the buggy code passes one public test) to 1 at its edit; the hacker’s rises at its edit just the same. Both collect +0.67 of shaping, and with Φ=0\Phi = 0 imposed at the end the sum would be −Φ(s0)=−0.33-\Phi(s_0) = -0.33 for both, a constant. Without that terminal condition, the shaping return is Φ(sT)−Φ(s0)\Phi(s_T) - \Phi(s_0), the public pass fraction of the final code minus a constant: the public-test reward of Section 11.3, hackable in exactly the same way. Process signals computed from what the agent can observe are as exploitable as the outcome signals they are computed from. Figure 11.3 collects the reward types of this chapter on four axes. The top four rows are measured on the gym: signal density is the share of the RL base agent’s groups with nonzero reward variance (Section 11.2), and hackability is the hacker’s reward divided by the oracle’s (Section 11.1). The bottom four rows are the book’s qualitative judgments, labeled as such. No row is light in every column. The verifiers that are hardest to hack (hidden tests) are the least dense; every way of adding density (fractions, learned models, judges, process rewards) adds a way to be exploited, except potential-based shaping, which under an outcome-level advantage adds nothing at all. Figure 11.3: Reward design matrix. Rows: reward types; columns: signal density, hackability, cost, and noise. Top four rows measured on the BugFix gym; bottom four rows are qualitative judgments. Darker means more.

​
11.7 Design problems in practice

Problem: sparse binary rewards. Symptom: most groups have zero advantage, early learning is slow, and hard tasks are never learned. Cause: a binary verifier gives no signal until a success is sampled; a group of GG attempts at a task with pass rate pp is uninformative with probability pG+(1−p)Gp^G + (1-p)^G (Section 7.5). Options: (a) larger groups or dynamic sampling, paid in rollouts (Chapter 9). (b) A curriculum that keeps tasks of medium pass rate (Chapter 14). (c) Partial credit or shaping, which densify the signal and change the task (next problem). (d) A better starting policy (SFT or distillation, Chapters 15 and 16). Default: keep the binary verifier and fix density with (a), (b), and (d); add partial credit only with a measurement of what it rewards. The hidden-test run’s history shows how sparse the binary reward is over training.
Output:
As the hidden-test run learns, the binary reward becomes less informative, not more: the share of zero-advantage groups rises from 0.21 in the first ten steps to 0.71 in the last ten, because most groups now pass every attempt. Early in training, sparsity was not the bottleneck (Section 11.2 measured 75% informative groups for the RL base agent); late in training, saturation is.
Problem: partial-credit gaming. Symptom: the reward rises while the binary pass rate does not, or the policy settles on submitting quickly. Cause: a fractional verifier pays for tests that the unfixed code already passes, so doing nothing has a positive reward, and any behavior that moves the fraction a little, including breaking code in ways that happen to pass more cases, earns credit. With GRPO’s standard-deviation normalization, those small differences get full-sized advantages (Section 11.2). Options: (a) binary reward on the task’s FAIL_TO_PASS tests, with partial credit only on tests that the original code fails. (b) Partial credit added to a binary term that dominates it. (c) Partial credit without std normalization (Dr. GRPO), so that small differences give small advantages. Default: (a). The partial-credit run of Section 11.2 measures what it bought here.
Output:
The partial-credit run reaches a higher training reward, 0.901 against 0.711, and a lower held-out pass rate, 0.588 against 0.762. Its reward includes the credit that unfixed submissions collect, so a reward of 0.9 does not mean nine fixes in ten. Both policies read about once per episode, so partial credit did not prevent the skill from being learned at this scale; it made the reward a worse measurement of it and the policy a little worse at it (one seed).
Problem: judge exploitation. Symptom: judge scores rise steadily while executed tests or human review do not; transcripts grow more confident and formulaic. Cause: the judge scores the transcript, which the policy writes, and is therefore an optimization target with the blind spots of Section 11.5. Options: (a) use a judge only where no executable check exists, and combine it with one that does. (b) Rubrics with instance-specific, checkable items (Rubrics as Rewards, 2025). (c) A judge that is not the model being trained and is refreshed or ensembled. (d) Monitor judge score against an independent measurement, as in Section 11.4. Default: (a) and (d); never let a judge be the only reward for a capability that tests can check. The rubric judge above gave the reading hacker the oracle’s score.
Problem: reward noise from flaky tests. Symptom: a correct policy’s reward varies between identical runs, groups that should be uniform have nonzero advantages, and learning slows without a visible cause. Cause: if a correct solution fails with probability ϵ−\epsilon_- (false negatives) and an incorrect submission passes with probability ϵ+\epsilon_+ (false positives), the expected reward of a policy with true pass rate pp is (1−ϵ−−ϵ+) p+ϵ+⋅s(1 - \epsilon_- - \epsilon_+)\,p + \epsilon_+\cdot s, where ss is its submit rate. The noise rescales the signal by 1−ϵ−−ϵ+1 - \epsilon_- - \epsilon_+ and adds variance; false positives also pay for submitting anything. Options: (a) rerun failing tests and take the majority, multiplying grading cost. (b) Validate every task by running its reference fix repeatedly and drop flaky tasks (Section 4.8). (c) Larger groups to average the noise. Default: (b) before training, then (a) for the remaining tests that flake. flaky_reward wraps any reward with both kinds of noise; we train with each at a rate of 0.3.
Output:
Both noisy runs still learn to read and fix: their true pass rates over the last ten steps, 0.766 and 0.764, match the clean run’s 0.711 within noise, and their held-out pass rates, 0.706 and 0.675, are a little below the clean 0.762. What the noise does change is the reported reward. With 30% false negatives, the training reward (0.544) understates the true pass rate by about 30%; with 30% false positives it overstates it (0.841), because failed submissions are paid at random. Figure 11.2 shows the two curves for each run. At this scale, with GRPO’s group comparison, symmetric noise of this size costs little learning; the cost appears in what the dashboard says, and in the slower learning that the expected-advantage scaling 1−ϵ−−ϵ+1 - \epsilon_- - \epsilon_+ predicts for harder tasks with fewer successes per group. Figure 11.2: Flaky verifiers. Left: the training reward as the noisy verifier reports it. Right: the hidden-test pass rate of the same batches. Hidden-test reward, clean or with 30% false negatives or 30% false positives; RL base agent, 40 steps, one seed.
Problem: reward scale and normalization. Symptom: changing a reward’s units, or adding a small penalty term, changes the learning speed or the learned behavior far more than expected. Cause: with std-normalized group advantages, the scale of the reward cancels within a group, so multiplying it by 10 changes nothing; without std normalization (adv_mode='mean', Dr. GRPO) the advantage scales with the reward and acts like a learning-rate change. A penalty term, in turn, competes with the verifier in reward units: a format penalty of 0.5 per malformed call can outweigh a success. Options: (a) keep the verifier in [0, 1] and set every term’s size relative to it, in units of “how many failures is this worth”. (b) Std normalization, with its difficulty bias (Section 7.5). (c) Batch-level scaling of group-centered advantages. Default: (a) with (c) or std normalization, and every term logged separately. The block computes advantages for the same batch at two reward scales.
Output:
With std normalization, multiplying every reward by 10 leaves the advantages unchanged (mean ∣A∣|A| 0.512 at both scales). Without it, the advantages grow tenfold, from 0.203 to 2.031, and so does the gradient: a reward-scale change is a learning-rate change. The same arithmetic governs penalty terms: under adv_mode='mean', a 0.05 step cost moves advantages by 0.05 per call whatever the verifier’s scale; under std normalization, its effect depends on how much the verifier varies within the group.
Problem: leaking tests. Symptom: training reward on the “hidden” verifier climbs faster than a fresh held-out evaluation, or solutions contain literal values from the grader. Cause: the hidden tests reach the agent through some channel: a test tool that runs them, a file in the repository, an error message, or test data copied into the training tasks. A leaked hidden test is a public test, and every exploit of Section 11.3 applies to it. Options: (a) grade outside the sandbox after the episode, and audit observations for grader content (Section 4.8). (b) Keep a third test set that no reward ever uses. (c) Canary strings in grader files and a scan of transcripts for them. Default: (a) and (b). The block simulates a leak: the test tool, and therefore hardcode_tests, sees the hidden tests too.
Output:
With the hidden tests visible to the test tool, hardcode_tests special-cases them too, and the hacker’s hidden reward becomes 1.00. A leak converts the training reward that Section 11.4 trusted into the one that Section 11.3 broke, and nothing in the training curves distinguishes the two. Only a third test set that never reached the agent would show it.

​
Summary

  • A reward is the specification of the task. Writing it as a function of the episode’s terminal record separates the verifier (what counts as success) from reward terms (what the episode’s conduct costs); make_reward and flaky_reward implement both. Public tests, hidden tests, and test fractions define different tasks: the hacker scores 1.000 on the public verifier and 0 on the hidden one.
  • Training on the public tests reproduced reward hacking in about 10 steps: the exploit spread to 97% of episodes, the training reward reached 1.000, and the hidden pass rate on held-out tasks fell from 0.175 to 0. Its dashboard signature is a smooth rising reward, a falling independent measurement, and a rising behavior statistic. Grading with hidden tests (0.762 held-out) or penalizing the known exploit (0.719) removed it.
  • The training reward and the evaluation measurement must not share a weakness: hidden tests are still optimized against, leaked tests are public tests, and only a set that no reward ever uses measures generalization. Monitor the gap between them during training.
  • Learned and judged rewards are proxies that RL takes out of distribution: a Bradley–Terry reward model with 87.5% held-out pairwise accuracy scored a reading hacker at +3.08 against the oracle’s +4.31, and a rubric judge scored it equal to the oracle. Partial credit and std normalization paid submissions of unfixed code; partial credit raised the training reward and lowered the held-out pass rate.
  • Potential-based shaping γΦ(s′)−Φ(s)\gamma\Phi(s') - \Phi(s) provably leaves the optimal policy unchanged (checked to 3×10−133 \times 10^{-13} on the repair MDP), and a read bonus made reading forever optimal. Under outcome-level GRPO a potential-based term with Φ=0\Phi = 0 at the end has no effect at all; a potential computed from public tests is a public-test reward. All measurements here are single-seed mechanism demonstrations on a 158k-parameter model.

​
Exercises

  1. Prove the necessity half of Ng et al.’s result in a two-state example: construct an MDP and a shaping term F(s,a,s′)F(s, a, s') that is not a difference of potentials and changes the optimal policy, then show that no Φ\Phi reproduces your FF.
  2. Change an assumption: rerun the hacking run of Section 11.3 starting from the Chapter 6 base agent (arl.make_base_agent(tok)) instead of make_rl_base. Does the exploit spread faster or slower? Relate the answer to how often each starting policy applies hardcode_tests before training.
  3. Write a second exploit op for the gym, for example one that wraps the function body in try/except and returns the expected value of the first public test on any error, and add it to EDIT_OPS in a copy of the environment. Train with make_reward('public', exploit_penalty=1.0). Does the policy move to the undetected exploit? What does this say about detectors as reward terms?
  4. Train the default configuration against the Bradley–Terry reward model of Section 11.5 instead of the hidden tests: sample with arl.sample_step, replace batch['rewards'] by rm_scores, and update with arl.rl_update. Plot the RM score and the hidden pass rate over 40 steps. Do you see the rise-then-fall of Gao et al.?
  5. Derive the expected GRPO advantage of an episode under the flaky reward of Section 11.7 as a function of ϵ−\epsilon_-, ϵ+\epsilon_+, and the group’s true pass rate. At what value of ϵ−+ϵ+\epsilon_- + \epsilon_+ does the expected gradient vanish?
  6. Implement the potential Φ\Phi = fraction of hidden tests passing as a per-turn shaping term with turn_advantages and a critic (Section 7.6). Does it speed up learning compared with outcome-only GRPO? Explain why the same term under outcome-only GRPO cannot.
  7. Design review: your team’s SWE agent is trained with a reward of “all FAIL_TO_PASS tests pass”, and an LLM judge was added to penalize “test special-casing”. After 200 steps the judge’s flag rate fell to 0.5% and SWE-bench Verified rose by 2 points, while a fresh internal benchmark fell by 1 point. Write down the hypotheses, the measurements that would distinguish them, and the changes to the reward and the evaluation that you would propose.