Skip to main content
Chapter 17 made a pass rate into a measurement with an error bar. This chapter asks what to do when the measurement is bad, or, more dangerously, good for the wrong reason. An RL run is a loop of five programs (sampler, environment, reward, advantage, update) that feed each other, and a fault in any of them produces a run that still executes, still logs, and often still learns something. A supervised loss that is wrong usually announces itself as a loss that does not fall. An RL bug can raise the reward: a mask that trains on the wrong tokens, a stale probability ratio, and a broken gradient-accumulation normalizer all produced rising reward curves in this chapter’s experiments. We do three things. First, we decide what to log and give each number a healthy range measured on TinyAgent. Second, we build a compact instrumented trainer, check that it reproduces train_rl exactly, and then break it on purpose, one fault at a time: an off-by-one mask, a stale old_logp, entropy collapse, length bias, a sign error, reward hacking, a learning rate that is too high, a gradient-accumulation bug, a temperature mismatch, and a reference model that is secretly the live policy. Each fault gets its dashboard signature next to a healthy run. Third, we turn the signatures into invariants that a codebase can assert before step 1, and into a triage procedure from symptom to test. Every run here is a 158k-parameter mechanism demonstration on 20 training tasks; the signatures transfer, the magnitudes do not.

​
18.1 What to log, and why

A metric earns a place on the dashboard when some failure moves it before the reward does, or moves it when the reward does not. That rule sorts the useful numbers into four groups.
  1. Outcome. The training reward, the hidden-test pass rate of the training batch, and the pass rate on held-out tasks. The training reward is what the optimizer sees, so it is the number a fault can corrupt without anyone noticing; the held-out pass rate under an independent verifier is the check on it (Chapter 11, Chapter 17).
  2. Behavior. What the agent did: reads, tests, and edits per episode, malformed calls, submissions, the share of episodes that use a known exploit (hardcode_tests), and the number of policy tokens per episode, split into passing and failing episodes. These say how a reward was earned, which Chapter 9 showed is the difference between a skill and a shortcut.
  3. Optimizer health. The policy entropy on action tokens, the approximate KL between the policy before and after the update (approx_kl), the clip fraction, the extremes of the ratio ρ\rho, the gradient norm before clipping, the KL to the reference policy when one is used, and the share of groups whose rewards are all equal (zero_adv_frac), which carry no gradient. With a critic, add the explained variance of its values (Section 7.4); this chapter’s runs are critic-free.
  4. Consistency. Quantities that should be a fixed value in a correct implementation, such as the gap between the log-probabilities the sampler recorded and the ones the trainer recomputes. A healthy run puts them at float-precision noise; any visible value is a bug. These are the cheapest metrics to log and the most diagnostic.
log_metrics builds the rollout half of one dashboard row from the episodes a step sampled. It reuses Chapter 9’s behavior_stats and adds the metrics that chapter did not need: malformed calls, the exploit share, submissions, and the lengths of passing and failing episodes. The update half comes from the trainer below and is merged in through update.

​
18.2 An instrumented trainer with injectable faults

To show what a fault looks like, we need a trainer in which each fault is one switch. The trainer below is Chapter 9’s rl_update and train_rl, split at the places where real bugs live, with a bug argument that turns one fault on. BUGS lists the code faults; the other failures in the catalogue (a high learning rate, reward hacking, length bias) are configuration choices and need no switch. prepare_batch produces the three tensors the loss consumes: the action mask, old_logp, and the advantages. Three faults live here. The off-by-one mask is built from the token owners aligned to ids instead of ids[1:], which moves every policy span one position to the right: each call loses its first token, <call>, and gains the token after its last one, the <obs> delimiter that the environment writes. The temperature fault takes old_logp from a sampler that ran at temperature 0.7, as a system does when it trusts the inference engine’s log-probabilities. The sign error negates the advantages. make_ref builds the KL reference, and under ref_is_policy it returns the live model itself, a one-character bug in a real codebase (ref = policy instead of a deep copy).
minibatch_backward computes the loss of one minibatch and backpropagates it, without stepping the optimizer. Two more faults live here. Under accum_local, the minibatch is split into four microbatches sorted by length, as a token-budget scheduler would split it, and each microbatch’s token_mean loss is divided by four: the normalization bug of Section 14.4. Under entropy_sign, an entropy bonus is added to the loss instead of subtracted, which turns it into an entropy penalty. The KL to the reference is computed inside the minibatch, from the reference’s forward pass on the same rows, so a reference that is the live policy gives exactly zero.
debug_update is rl_update assembled from the two pieces, with the stale-old_logp fault: before every minibatch it recomputes “old” log-probabilities from the current weights, which is what happens when the no-grad forward pass is moved inside the minibatch loop. It also logs the consistency metric of Section 18.1, logp_gap, the mean absolute difference between the sampler’s recorded log-probabilities and the trainer’s own recomputation on the tokens the loss trains on.
train_logged is train_rl with log_metrics rows and the sampler configuration that the temperature fault needs. It evaluates on held-out tasks every eval_every steps with the same protocol as train_rl: 4 episodes per task, hidden-test pass rate, in the training environment.
A debugging harness is only useful if, with no fault injected, it is the trainer it claims to be. We train two copies of the RL base agent for 5 steps, one with train_rl and one with train_logged, and compare every parameter and every statistic both histories record.
Output:
The healthy run is the book’s default configuration for 24 steps, with a held-out evaluation every 8 steps. run stores each history under a name, and show prints a few metrics of each run as the mean of its first four steps, an arrow, the mean of its last four, and the final held-out pass rate.
Output:
With no fault, the 24-step run raises the training reward from 0.17 to 0.66 and the held-out pass rate from 0.10 to 0.70. The columns give the healthy range of every metric in this setup, and every failure below is read against them. Three groups deserve a closer look. The consistency metric logp_gap stays between 7×10−97 \times 10^{-9} and 5×10−85 \times 10^{-8}: the sampler and the trainer run the same fp32 model on the same tokens and agree to rounding error. The update is small. approx_kl is 0.003 at step 0 and about 3×10−43 \times 10^{-4} afterwards, the largest ratio is at most 1.27, and the clip fraction is 0 in at least half of the steps, because the first of the two minibatches is exactly on-policy and the second moves little. And between 0 and 63% of groups per step carry no gradient (median 25%), from the first step on, because the base agent fails all eight attempts at some tasks. The table condenses Section 18.1 into a reference card. The healthy column is measured above; the failure column refers to the catalogue that follows, numbered as in the opening of the chapter.

​
18.3 Faults in what the loss sees

The first three faults corrupt the inputs of the loss: which tokens it trains on, and which probabilities it treats as πold\pi_{\text{old}}. If Section 18.1’s grouping is right, their signatures should be in the consistency metrics, before any curve has had time to move.
Output:
The off-by-one mask did not stop learning. Its training reward rose from 0.19 to 0.52, and a reward curve alone would pass it as a slower seed of the healthy run. The consistency metric disagrees from step 0: logp_gap averages 3.75 nats per trained token over the first four steps, against 4×10−84 \times 10^{-8}. The sampler records log-probabilities only for tokens it sampled. On the <obs> delimiter that the shifted mask now covers, its record is the placeholder 0, while the trainer’s model gives the delimiter a log-probability far below zero, because no demonstration ever trained the policy to predict it. The same tokens drive the optimizer metrics. The gradient of log⁡π\log \pi with respect to a token’s logit is 1−π1 - \pi, near 1 for a token the model considers nearly impossible, so these few tokens dominate the update: the gradient norm is 0.48 at the start and 1.5 at the end, against about 0.2. Their ratios move far as well (largest ratio 3.0, clip fraction 0.044). The fault cost 23 points of held-out pass rate, 0.47 against 0.70. The stale-old_logp run is the opposite case. Its outcome cannot be told apart from the healthy one (training reward 0.72 against 0.66, held-out 0.68 against 0.70), and its only signature is that two optimizer metrics sit exactly at their trivial values for the whole run: clip fraction 0 and largest ratio 1. When old_logp is recomputed from the current weights, ρ≡1\rho \equiv 1, the clip can never act, and PPO silently becomes a sequence of plain policy-gradient steps. Here that is harmless, because each batch is used for two minibatch steps and the second moves little. With more epochs or minibatches the clip is what stops the policy from moving too far on one batch (Section 8.5 measured clip fraction and KL growing over four epochs), and this bug removes it without a trace in the reward. A metric that is exactly 0 or exactly 1 for a whole run is a warning, not a reassurance. The temperature fault is the most destructive of the three. The sampler ran at temperature 0.7, and the trainer, at temperature 1, used the sampler’s log-probabilities as πold\pi_{\text{old}}. The training reward fell from 0.23 to 0.16 and the held-out pass rate ended at 0.15, below the blind single guess of 0.20. logp_gap is 0.075 nats at the start, six orders of magnitude above healthy noise, and the largest ratio at step 0 is already 2.65, where a correct pipeline has ρ=1\rho = 1 exactly before the first update. The behavior metrics show what happened next.
Output:
Reads fell from 0.09 per episode to 0.004, and by the end every group had equal rewards (zero_adv_frac 1.0) and the gradient norm was 0: the policy stopped exploring and the run died. The mask fault, by contrast, raised entropy (0.58 to 0.80) and malformed calls (0.04 to 0.12), side effects of training a token the policy never emits and no longer training the <call> token that opens each call. Why would a temperature mismatch kill exploration in particular? Lowering the temperature moves probability from rare tokens to common ones, so for a rare token the ratio πT=1/μT=0.7\pi_{T=1}/\mu_{T=0.7} starts above 1, and for a common token below 1. PPO’s clip removes the gradient of a positive-advantage token whose ratio exceeds 1+ϵ1 + \epsilon. We measure this on one batch of the base agent sampled at temperature 0.7, before any update. probe samples 64 episodes the way a given (possibly faulty) pipeline samples them; Section 18.6 reuses it.
Output:
Among the tokens the update should reinforce, the rare ones (sampler probability below 0.5) start with a mean ratio of 1.17, and 12% of them are clipped before the policy has changed at all; the common ones start at 0.95, and none is clipped. The clip, which exists to keep the policy from moving too far, here withholds gradient from the rare good actions, and rare good actions are what exploration consists of. Two effects therefore act together: the cooler sampler draws reads less often (0.09 per episode at step 0, against 0.47 at temperature 1), and when a read is rewarded, part of its update is clipped away. The rare-token sample is small (17 tokens), and the chapter does not run the control with a consistent temperature of 0.7 on both sides; in an exploratory run not reported here, consistent sampling and training at a lower temperature learned normally, which points at the mismatch rather than the temperature. Exercise 1 asks you to run the control. The fix does not depend on the decomposition: the trainer must compute probabilities under the distribution the sampler actually used (Chapter 3), or correct for the difference with importance weights (Chapter 13).

​
18.4 Faults in the gradient

The next three faults leave the loss’s inputs intact and corrupt the gradient: its sign, its size, and its normalization. The sign and learning-rate runs stop at step 8, as an engineer would stop them once the failure is unambiguous.
Output:
With its advantages negated, the policy performs gradient descent on the reward. The training reward fell from 0.13 to 0.03 in eight steps, the held-out pass rate to 0.03, and episodes lengthened from 4.2 to 5.5 turns, because using the budget without submitting is a reliable way to earn 0. The optimizer metrics are unremarkable: approx_kl, entropy, and malformed calls are all in the healthy range. Only the outcome says the run is wrong, which is why Section 18.6 tests the direction of an update directly. A learning rate of 10−210^{-2}, ten times the default, shows first in the optimizer and format metrics. Malformed calls rose from 0.76 to 2.89 per episode, entropy from 0.42 to 1.08 nats, every late episode used all 6 turns, and the reward fell from 0.04 to 0. The last two lines of the output show the order of events in the per-step KL. The first update at learning rate 10−210^{-2} moved the policy by an approximate KL of 0.31 nats per token, about 100 times the healthy first step’s 0.0032, and the second by 0.064. After that, the per-step KL fell back toward healthy values. The run had not recovered: its reward had already collapsed, the groups were flat, and there was little gradient left to apply. This is the general shape of an instability: one or two steps with a large approx_kl or grad_norm, then a quiet, broken run. A dashboard that shows only smoothed curves can hide the cause and show only the aftermath, so log the per-step maximum alongside the mean. The accumulation fault has no signature at all. Its training reward (0.67) and held-out pass rate (0.65) match the healthy run within the noise of a 40-episode evaluation (a binomial standard error of about 0.07 at a pass rate of 0.7), and so does every optimizer metric; malformed calls end slightly higher (0.16 against 0.07). Yet the gradient is wrong: the invariant check of Section 18.6 measures a relative error of 0.54 against the one-pass gradient (Figure 18.2), because tokens in short microbatches receive several times the weight of tokens in long ones (Section 14.4). In this gym the episodes are short and similar in length, the error acts like a noisy reweighting of tokens, and the run survives. With long-tailed agent episodes, a length-sorted microbatch can hold a single very long trajectory, whose tokens then receive a small fraction of their intended weight. Whether that shows on a dashboard depends on the run; whether it is present does not, and only a test of the gradient can say.

​
18.5 Faults in the objective

The last four failures come from the objective rather than from a defect in the update: a regularizer, a loss aggregation, a reward, and a reference model. We expect their signatures in the behavior statistics.
Output:
We force an entropy collapse with an entropy bonus whose sign is flipped, which makes it a penalty of 0.3 on the mean entropy of action tokens. In real runs, collapse usually has slower causes, such as a low sampling temperature, a reward that favors one mode, or many epochs per batch (Chapter 10), but its signature is the same. Entropy fell from 0.45 to 0.027 nats, and reads disappeared with it, from 0.19 per episode to 0.008. The reward stayed at 0.18 and the held-out pass rate ended at 0.20, the blind single-guess ceiling of Section 9.4: the policy collapsed onto “edit, submit” before it discovered that reading pays. zero_adv_frac separates this from success. A converged policy and a collapsed one both have low entropy and many flat groups, but here 84% of groups are flat at a reward of 0.18, so almost all of them are all-fail groups, while a converged policy’s flat groups are all-pass. Log the all-pass and all-fail shares separately, and treat “entropy falling faster than the reward rises” as the early warning.
Output:
Length bias needs room to appear, so both runs here have a 12-turn budget and run 16 steps; they differ only in the loss aggregation. Under token_mean, failing episodes got shorter, from 16 to 13 policy tokens. Under seq_mean_token_mean they grew from 21 to 34 tokens, and episodes used 6.8 turns against 4.0, with 2.7 reads per episode where one is enough. Passing episodes stayed at about 13 tokens in both runs, and neither the training reward (0.51 against 0.49) nor the held-out pass rate (0.55 in both) separated them. This is the mechanism of Section 6.6: under per-sequence normalization each failing episode carries a fixed total penalty, so each of its tokens is penalized less when the episode is longer, and the cheapest way to fail is to fail slowly. The signature is length split by outcome; the overall mean length mixes the effect with the changing share of failures. For a coding agent, this failure costs rollout time and context long before it shows in the reward.
Output:
Reward hacking reproduces Chapter 11’s run in 24 steps. The training reward is the public-test verifier, and it rose from 0.27 to 0.97, the best curve in this chapter. On the same episodes, the hidden-test pass rate fell from 0.18 to 0.008, the share that applies hardcode_tests rose from 0.13 to 0.97, reads fell to 0.04 per episode, and the held-out pass rate ended at 0. The exploit share is the most direct metric because it names the behavior, but it exists only because someone already knew the exploit. The hidden-test pass rate needs an independent verifier, and it also catches exploits that nobody anticipated.
Output:
The last fault is a reference model that is the live policy. With β=0.1\beta = 0.1 and a frozen copy as the reference, the KL to the reference grows from 0.014 to 0.057 nats per token in eight steps as the policy moves away from its start. With the live policy as the reference, it is exactly 0 at every step, and the run is bit-identical to the healthy run without any KL term: the k3k_3 estimate and its gradient both vanish when πref=πθ\pi_{\text{ref}} = \pi_\theta. Every other metric is healthy, because this run is the healthy run. The failure is that the regularization the configuration asks for does not exist, and it surfaces only when someone needs it, for example when a long run drifts. A KL that is exactly zero while its coefficient is not is always a bug. Figure 18.1: The dashboard of ten failures. Each failure has two panels: the training batch reward (top) and the metric that exposes it (bottom), in red, against its matched healthy run in blue. Healthy runs: the default configuration, except a 12-turn budget for the length-bias control and a KL coefficient of 0.1 for the reference fault. 24 steps of 64 episodes; the sign-error and learning-rate runs stopped at 8 steps, and the length-bias pair ran 16 steps. Figure 18.1 collects the ten failures as a dashboard shows them. Read the top panel of each pair first. In four of the ten (the stale ratio, the length bias, the accumulation bug, the live reference) the reward curve is inside the healthy run’s step-to-step noise; in one (the mask) it rises more slowly but still rises; and in one (reward hacking) it is the best curve on the page. Then read the bottom panels, which show why Section 18.1 sorted metrics into groups. Consistency metrics flag the code faults (1), (2), (9), and (10) from the first step: a log-prob gap of 100.510^{0.5} instead of 10−810^{-8}, a clip fraction pinned at 0, ratios with single steps above 20, a KL pinned at 0. The format metric flags the learning rate (7) within two steps. Behavior metrics flag the objective failures (3), (4), and (6), whose training reward either never separates from the healthy run or moves the wrong way. And one fault, (8), flags nothing.

​
18.6 Invariants every RL codebase should assert

Every fault of Sections 18.3–18.5 that lives in code violates an identity that holds for a correct trainer at step 0, before any training. Checking the identities costs a few forward and backward passes on one batch, and a failed check names the fault far more precisely than a curve does. The book’s list:
  1. Mask = policy tokens. The loss mask covers exactly the positions whose target token the policy sampled.
  2. Sampler = trainer log-prob. On those positions, the trainer’s recomputed log-probability equals the sampler’s record up to numerical noise: 10−410^{-4} here, where both are fp32; for bf16 engines, a threshold set from the measured mismatch distribution (Chapter 13, Section 13.6).
  3. Ratio = 1 before any step. In the first minibatch of a step, ρ=πθ/πold\rho = \pi_\theta / \pi_{\text{old}} is exactly 1.
  4. Group advantage mean = 0 for group baselines (Section 7.5), which also checks that groups are laid out as the advantage function expects.
  5. Loss invariant to padding. Appending pad columns to the batch changes nothing.
  6. Accumulated = one-pass gradient. Gradient accumulation reproduces the gradient of one backward pass over the whole batch.
  7. old_logp frozen in the update. After the first minibatch step, later minibatches see ρ≠1\rho \neq 1, so approx_kl over the update is positive.
  8. The update follows the sign of the advantage. After one update, log⁡π(τ)\log \pi(\tau) rises for episodes with A>0A > 0 and falls for A<0A < 0: the change and the advantage are positively correlated.
  9. KL to the reference is 0 at init and positive after a step, when a reference is used. The first half holds by construction in this trainer; the second catches a live reference.
pad_batch and flat_grad are small helpers: one adds right padding, the other runs the trainer’s backward on a whole batch and returns its gradient as one vector. step_checks takes one real update on a copy of the model and tests invariants 7 to 9.
Output:
sanity_checks builds the batch exactly as the trainer does and runs the six static checks and the three one-step checks, returning each check’s measured value and verdict. The trajectories carry their token owners (Section 5.6), which is what makes the mask check independent of the mask under test. With no fault, all nine checks pass, and the values show the margins. The mask matches the owners exactly. The largest sampler–trainer gap over 64 episodes is 1.3×10−61.3 \times 10^{-6}: lockstep sampling runs the episodes in different batch shapes than the trainer, so fp32 rounding differs slightly, which is why the threshold is not zero. Padding changes the loss by 7×10−97 \times 10^{-9}. After one step, the change in episode log-probability correlates with the advantage at 0.51; a small clipped step over two minibatches is not expected to give a correlation near 1, and its sign is what matters. Now the same checks with each code fault switched on, one row per fault and one column per check, numbered as in the list above.
Output:
Every code fault fails at least one check, and the failing checks identify it: the stale ratio fails only check 7, the sign error only check 8, the accumulation bug only check 6, the live reference only check 9. Figure 18.2 shows the matrix with the measured values. Three results deserve comment. The off-by-one mask fails three checks, one of them padding invariance, which we did not design for it: shifting a mask to the right drops the last column of an unpadded batch but moves it into a pad column of a padded one, so the loss depends on the padding. The temperature fault fails both log-probability checks, with step-0 ratios as large as 3 where they should all be 1. And the entropy penalty passes all nine: it is a plausible-looking term with the wrong sign, the update still follows the sign of the advantage (correlation 0.37 in Figure 18.2), and only the dashboard of Section 18.5 shows the collapse. Invariants catch implementation faults; objective faults need the dashboard. Neither catches reward hacking, the learning rate, or length bias, which are configuration choices running correct code, and need the held-out evaluation and the healthy reference. Figure 18.2: Which invariant catches which fault. Rows: the instrumented trainer with no fault and with each code fault of BUGS; columns: the nine checks of sanity_checks, run once on 64 episodes of the RL base agent before any training. Red cells fail.

​
18.7 Triage: from symptom to test

A dashboard says that something is wrong. Triage is the procedure that turns a symptom into the cheapest test that decides between its causes. The book’s procedure has four steps, in order of cost.
  1. Run the invariants. sanity_checks costs a few forward and backward passes on one batch, and it identifies or rules out every code fault of this chapter. Run it at the start of every run, and again after every change to the trainer, the template, the sampler, or the batching code.
  2. Read the consistency metrics. A logp_gap above noise, a ratio or clip fraction stuck at its trivial value, or a KL that is exactly 0 is a bug, not a dynamic, and needs no comparison run.
  3. Compare against a healthy reference. Optimizer and behavior metrics have no fixed correct value. Their meaning comes from the healthy range of the same setup (the table of Section 18.2) and from the seed spread (Chapter 17). Change one thing at a time against the reference.
  4. Check the outcome against an independent signal. Held-out tasks under a verifier the policy cannot see are the only defense against a reward that measures the wrong thing.
Figure 18.3 maps each symptom of this chapter to its first test. Figure 18.3: Triage map. Left: a symptom on the dashboard. Middle: the first test to run. Right: the faults it points to, numbered as in this chapter's catalogue. Read a row left to right; several symptoms share a test, and the consistency checks of Section 18.6 come first because they are cheap and decisive. Two notes on using the map. Symptoms share causes: a zero_adv_frac rising toward 1 at a low reward appeared under the entropy collapse, the temperature mismatch, the sign error, and the high learning rate, so the test decides, not the symptom. And the order matters: a consistency fault can imitate the dynamic failures downstream of it. The mask fault raised the gradient norm, the clip fraction, entropy, and malformed calls, which together look like a learning rate that is too high; lowering the learning rate, the natural response, does nothing about the cause. Fix the cheap invariants before tuning anything.
In production. verl at the project’s pinned commit 00094bd (source read on the check host, 2026-10-13) logs most of these groups: score, reward, advantage, and return statistics; response_length/mean, max, min, and clip_ratio (the share of responses cut at the length limit); actor/entropy; actor/pg_clipfrac and actor/pg_clipfrac_lower (the dual clip); actor/ppo_kl, a masked mean of the log-ratio (the k1k_1 estimate); the gradient norm; and critic/vf_explained_var when a critic is used. When the rollout’s log-probabilities are recorded in the batch, it also computes training/rollout_probs_diff_mean, _max, and _std, absolute differences of token probabilities between the rollout engine and the trainer on the response mask, and their Pearson correlation: the consistency metric of this chapter, on probabilities rather than log-probabilities. Task-specific behavior statistics, such as tool calls per episode or an exploit share, are the user’s to add.

​
18.8 Design problems in practice

Problem: a run that looks healthy on reward but is broken. Symptom: the training reward rises, perhaps a little more slowly than in earlier runs, and nobody looks further. Cause: a policy gradient is robust to many implementation faults, because the gradient on the correct tokens is still present. A fault that adds wrong terms, drops some right ones, or removes a safety mechanism slows or biases learning without stopping it; the mask fault of Section 18.3 is the canonical case. Options: (a) gate every run on sanity_checks at step 0; (b) log the consistency metrics (logp_gap, ratios and KLs that should be exactly trivial) and alert on any value above noise; (c) compare reward curves against a pinned healthy reference run. Default: (a) and (b), which are cheap and decisive; (c) alone cannot tell a slower bug from a slower seed. The two reward curves, every four steps:
Output:
Through step 12, the faulty run’s rewards (0.17, 0.27, 0.39, 0.41) track the healthy run’s (0.17, 0.34, 0.36, 0.44) within their step-to-step noise. The faulty run falls behind only later, ending at 0.50 against 0.72 at step 20, a gap that someone looking only at the reward could attribute to a seed. The step-0 numbers on the same lines are unambiguous: a log-prob gap of 3.5 against 3×10−83 \times 10^{-8}, and a gradient norm 2.4 times the healthy one.
Problem: a silent failure seen only in held-out evaluation. Symptom: every training-side metric looks good, or better than usual: rising reward, falling entropy, small KL steps, normal gradient norms. Cause: the training signal and the goal have separated. Reward hacking is the clean case (Chapter 11); memorized shortcuts (Chapter 9) and a reward that measures a proxy are others. No training-side metric can see it, because they all measure the policy’s relation to the reward it was given. Options: (a) held-out evaluation under an independent verifier at a fixed cadence; (b) behavior statistics for every exploit found so far; (c) periodic human or model review of sampled trajectories. Default: (a) always, with (b) for known exploits, and (c) when (a) and (b) disagree with what the trajectories suggest. The hacking run of Section 18.5:
Output:
Read the step-16 row without its last column: reward 0.97, entropy 0.12, a gradient norm of 0.16, and an approx_kl of 1.4×10−41.4 \times 10^{-4}, the profile of a run converging cleanly. Its held-out hidden-test pass rate was 0.025, on the way to 0 at step 23, from 0.175 at the start. The same logic applies to checkpoint selection: a checkpoint chosen by training reward would be the most hacked one. Choose checkpoints on a held-out validation split (Section 17.8) and keep the test split for the final report.
Problem: a spike that does not reproduce. Symptom: a run shows a spike in grad_norm, approx_kl, or reward, and a rerun of the same configuration does not, so the cause cannot be inspected. Cause: either the rerun did not draw the same samples, or numerical nondeterminism changed them. Options: (a) make every step replayable: log the seeds and sampler state, and save the batch (token ids, sampler log-probs, rewards) with a model checkpoint at every step, or at every step whose metrics cross a threshold, then replay the update deterministically and inspect the batch; (b) estimate the seed spread of the metric (Chapter 17) before calling a spike anomalous; (c) batch-invariant kernels, which remove the numerical source at a throughput cost (the Thinking Machines report in Chapter 20). Default: (a) and (b): a replayed spike is a bug report with evidence, and a spike inside the seed spread is noise. The block measures how far three kinds of rerun move the first six steps: an exact rerun, a rerun from weights multiplied by 1+10−51 + 10^{-5} noise (a stand-in for nondeterministic reductions), and a rerun with a different seed.
Output:
The same seed reproduced six steps exactly, even though the reference run interleaved held-out evaluations with training: evaluation uses its own generator, so it cannot shift the training stream. Weights perturbed by 10−510^{-5} changed the gradient norms by at most 8×10−68 \times 10^{-6} and no reward. No sampled episode changed in six steps, because a categorical sample flips only when its uniform draw lands within the perturbation of a probability boundary. A different seed changed the reward at every step and the gradient norm by up to 0.17, more than the healthy run’s whole range of gradient norms (0.16 to 0.29). On this model, then, an irreproducible spike is a question about samples, not about numerics. At scale the two mix: bf16 kernels that are not batch-invariant change log-probabilities by about 0.03 nats per token with the batch composition (Section 13.6), enough to flip sampled tokens early, after which two runs share nothing but their configuration.
Problem: bisecting a regression between two code versions. Symptom: the current code trains worse than last week’s, and twenty commits lie between them. Cause: any of them. RL regressions rarely sit where the diff looks dangerous; a refactor of a helper or a changed default can flip a sign or shift a mask. Options: (a) read the diff; (b) bisect: test the midpoint commit, keep the half that contains the transition, and repeat, which needs ⌈log⁡2n⌉\lceil \log_2 n \rceil tests for nn commits; (c) bisect with the cheapest decisive test, the invariants when one fails, and otherwise a short run with matched seeds, which costs a run per probe and works only when the regression exceeds the seed spread (Chapter 17). Default: (c). The block simulates a history of eight commits in which commit 3 introduced the sign error; the search sees only whether the invariants pass at each commit.
Output:
Three tests located commit 3 among eight. With invariants as the test, bisection is fast and its answer is unambiguous. With a training run as the test, each probe costs a run, and a regression smaller than the seed spread cannot be bisected reliably. That is a reason to add an invariant once a bug is found, so that the next regression of its kind costs one batch to find. git bisect run automates the same search over a real history.

​
Summary

  • A useful dashboard has four groups of metrics: outcome, behavior, optimizer health, and consistency. Consistency metrics (the sampler–trainer log-prob gap, ratios and KLs that must be exactly 1 or 0) are the cheapest and most decisive; on TinyAgent the healthy gap is below 10−710^{-7}.
  • An instrumented copy of the trainer reproduced train_rl bit for bit, and ten failures were reproduced on it. The training reward rose under the mask, stale-ratio, accumulation, and hacking faults; the mask fault cost 23 points of held-out pass rate while its reward rose, and reward hacking produced the best reward curve and a held-out pass rate of 0.
  • Each failure has a signature in one metric group: consistency for the mask, stale-ratio, temperature, and live-reference faults; optimizer and format metrics for the learning rate; behavior for entropy collapse, length bias, and hacking. The accumulation bug had none.
  • Nine step-0 invariants, saved as sanity_checks, caught every code fault, each through the checks that name it. They cannot catch objective or configuration failures, which need the dashboard, a healthy reference, and held-out evaluation.
  • Triage runs the invariants first, then the consistency metrics, then compares with a healthy reference and the seed spread, and always checks the outcome against an independent verifier. Make steps replayable and bisect regressions with an invariant when one exists. All runs here are tiny-scale mechanism demonstrations: the signatures transfer, the magnitudes do not.

​
Exercises

  1. Change an assumption: run the control that Section 18.3 leaves open. Train with RLConfig(temperature=0.7) and no fault, so that the sampler and trainer agree at 0.7. Does the policy learn to read? Then rerun the fault with SAMPLER_TEMP = 0.9 and 0.5. How do the clipped share of rare positive-advantage tokens and the final held-out pass rate change with the gap?
  2. The stale old_logp was harmless with one epoch and two minibatches. Rerun it and the healthy run with epochs=4, minibatches=4, and log approx_kl per minibatch. When does the missing clip begin to matter?
  3. Write an invariant that catches the entropy-penalty fault in one step. One candidate: with every advantage set to 0, a correct update leaves the policy unchanged unless a regularizer is intended. Implement it, and list the configurations in which it would raise a false alarm.
  4. Inject a fault the nine checks miss, for example advantages computed after the batch is shuffled out of its group layout. Which check comes closest, and what new invariant catches it?
  5. The logp_gap threshold of 10−410^{-4} assumes fp32 on both sides. Using Section 13.6’s bf16 measurements, propose a threshold and a statistic (mean, p99, or max) that separates numerical mismatch from an off-by-one mask in a bf16 engine–trainer pair.
  6. Make a spike replayable: extend train_logged to save the batch and the model state whenever grad_norm exceeds three times its running median, and write replay(step) that reproduces the logged statistics of that step exactly.
  7. Design review: your SWE agent’s training reward has risen steadily for 300 steps, its entropy is falling smoothly, and its held-out pass rate has been flat for the last 100 steps. A colleague proposes doubling the learning rate. Write the triage you would run first, in order, with the metric or test each step reads and the decision it supports.