Skip to main content
Every chapter so far has ended with a number: a held-out pass rate of 0.84, an ablation arm at 0.71 against a default at 0.76, a variant “within seed noise”. Chapter 9 could only say that last phrase informally. This chapter makes evaluation a measurement with a definition, an error bar, and a protocol. It asks five questions that a post-training team answers every week. What quantity does a pass rate estimate, and with what estimator? How uncertain is it, given that ten tasks and thirty-two samples are not 320 independent coin flips? Is checkpoint B better than checkpoint A, and how many tasks or seeds would it take to know? Which choices of the harness define the number, so that changing one of them changes the result without changing the model? And which checkpoint may we report without having tuned on the test set? The chapter answers them on TinyAgent with GRPO checkpoints from the default setting of Chapter 9, and it uses the same machinery for the most debated measurement in RL for reasoning models: whether RL improves pass@kk for large kk or only sharpens the distribution the base model already had. Every number here is a mechanism demonstration on a 158k-parameter model with ten held-out tasks. The ten tasks are the point: they make the statistical problems large enough to see.
The checkpoints come from one default GRPO run of 80 steps. train_snapshots calls arl.train_rl in chunks of ten steps and keeps a frozen copy after each chunk. Each call restarts the Adam optimizer and uses its own seed, a small departure from one uninterrupted 80-step run that does not matter for what this chapter measures.

​
17.1 What a pass rate estimates

An evaluation estimates a property of a policy under a harness. The harness HH is everything except the weights: the task set, the prompt template, the environment and its budgets (turns, edits, context length), the sampling temperature, and the success criterion. For a task jj, write pj(π,H)=Pr⁡[an episode of π on task j under H succeeds].p_j(\pi, H) = \Pr\big[\text{an episode of } \pi \text{ on task } j \text{ under } H \text{ succeeds}\big]. The benchmark’s pass@1 is the task average pˉ=1N∑j=1Npj\bar p = \frac{1}{N}\sum_{j=1}^{N} p_j. It is not “the fraction of episodes that passed” unless every task gets the same number of episodes, and it is not a property of the weights alone: change HH and every pjp_j changes. The success criterion here is the one Chapter 4 fixed for evaluation, independent of any training reward: the agent submitted and every hidden test passed. The raw data of an evaluation are therefore per-task success counts: draw nn episodes on each of NN tasks and record cjc_j, the number that succeeded. Everything else in this chapter (pass@1, pass@kk, intervals, paired tests) is a function of the vector cc. sample_counts produces it with the training harness cfg, through the same sample_groups that training uses, so the template, renaming, edit budget, turn budget, and temperature cannot silently differ.
We draw 32 episodes on each of the ten held-out tasks for the RL base agent and for the checkpoint after 80 steps.
Output:
The two pass rates, 0.144 and 0.819, are task averages of very different vectors. The base agent’s successes are spread almost evenly, 1 to 8 of 32 on every task: it guesses, and every task is about equally guessable. The trained agent solves eight tasks in nearly every episode, under_cap in 17 of 32, and speed in 1 of 32. A single number hides that one task is essentially unsolved, and the statistics below will show that this heterogeneity, not the number of episodes, sets the uncertainty of the trained agent’s score.

​
17.2 pass@kk and its unbiased estimator

Coding systems are often allowed several attempts: sample kk patches and keep one that passes, given a verifier. pass@kk is the probability that at least one of kk independent attempts succeeds, averaged over tasks: pass@k=1N∑j=1N(1−(1−pj)k).\text{pass@}k = \frac{1}{N}\sum_{j=1}^{N} \big(1 - (1 - p_j)^k\big). It needs an oracle to pick the passing attempt, so it measures coverage, an upper bound on what any selector can achieve, not a deployable success rate (Chapter 20 distinguishes it from Best@kk). The plug-in estimator is biased. The obvious estimate replaces pjp_j by p^j=cj/n\hat p_j = c_j/n. For k≥2k \ge 2 the map f(p)=1−(1−p)kf(p) = 1 - (1-p)^k is strictly concave on (0,1)(0, 1), so by Jensen’s inequality E[f(p^j)]<f(E[p^j])=f(pj)\mathbb{E}[f(\hat p_j)] < f(\mathbb{E}[\hat p_j]) = f(p_j) whenever 0<pj<10 < p_j < 1: the plug-in systematically underestimates pass@kk, most for small nn and large kk. The unbiased estimator. Chen et al. (2021) draw n≥kn \ge k samples and ask a different question of them: if we picked kk of the nn samples uniformly at random without replacement, what is the probability that all kk fail? With cc successes, it is (n−ck)/(nk)\binom{n-c}{k}\big/\binom{n}{k}, so pass@k^j=1−(n−cjk)(nk).\widehat{\text{pass@}k}_j = 1 - \frac{\binom{n-c_j}{k}}{\binom{n}{k}}. Unbiasedness takes one line. For any fixed set SS of kk sample indices, the event “every sample in SS fails” has probability (1−pj)k(1-p_j)^k, because samples are independent. The estimator’s failure term is the average of that event’s indicator over all (nk)\binom{n}{k} sets SS. Expectation is linear, so its expected value is (1−pj)k(1-p_j)^k exactly. (It is a U-statistic, the minimum-variance unbiased estimator built from the sample.) Unbiasedness holds per task, and averaging over tasks keeps it. A stable form. The binomial coefficients overflow floating point long before nn is large: (20001000)\binom{2000}{1000} has 600 digits. Writing both coefficients as factorials and cancelling, (n−ck)(nk)=(n−c)! (n−k)!n! (n−c−k)!=∏i=n−c+1ni−ki=∏i=n−c+1n(1−ki),\frac{\binom{n-c}{k}}{\binom{n}{k}} = \frac{(n-c)!\,(n-k)!}{n!\,(n-c-k)!} = \prod_{i=n-c+1}^{n} \frac{i-k}{i} = \prod_{i=n-c+1}^{n}\Big(1 - \frac{k}{i}\Big), a product of cc factors in (0,1)(0, 1) when n−c≥kn - c \ge k. When n−c<kn - c < k, every kk-subset contains a success and the estimate is exactly 1.
The check below draws 100,000 evaluations of n=10n = 10 samples for one task at several true pass rates and compares the average of each estimator of pass@5 with the truth.
Output:
The plug-in estimate is low at every pp, by 0.04 to 0.08, which is larger than most of the differences Chapter 9 tried to read. The unbiased estimator matches the truth to within Monte Carlo error (about 0.001 here). The bias is largest at intermediate pp, where ff is most curved over the spread of p^\hat p. The product form agrees with exact rational arithmetic, and it still works where the direct ratio of binomials cannot be represented.
Output:
For n=10,c=3,k=5n = 10, c = 3, k = 5 the product form gives 1−21/252=0.91671 - 21/252 = 0.9167, the value worked out in Appendix B, Problem 21. For n=2000n = 2000, (20001000)\binom{2000}{1000} has more than 600 digits and cannot be converted to a float, while the product of five factors near 1/21/2 gives 0.9689≈1−2−50.9689 \approx 1 - 2^{-5}. Chen et al. use n=200n = 200 and k≤100k \le 100 and note the same instability. Choose nn well above the largest kk you report: the estimate at kk close to nn is nearly a single indicator per task and has high variance.

​
17.3 Confidence intervals

A pass rate without an interval cannot be compared with anything. Three intervals cover the cases this book meets.

​
Binomial intervals for one task

For nn independent episodes with success probability pp, p^=c/n\hat p = c/n has standard error p(1−p)/n\sqrt{p(1-p)/n}. The Wald interval p^±zp^(1−p^)/n\hat p \pm z\sqrt{\hat p(1-\hat p)/n} plugs in p^\hat p, and it fails exactly where agent evaluations live: at p^=0\hat p = 0 or 11 its width is zero, and near the edges it undercovers. The Wilson interval inverts the test instead of plugging in. It collects every pp that the score test would not reject, ∣p^−p∣≤zp(1−p)/n|\hat p - p| \le z\sqrt{p(1-p)/n}. Squaring gives a quadratic in pp whose roots are p^+z22n1+z2n±z1+z2np^(1−p^)n+z24n2.\frac{\hat p + \frac{z^2}{2n}}{1 + \frac{z^2}{n}} \pm \frac{z}{1 + \frac{z^2}{n}}\sqrt{\frac{\hat p(1-\hat p)}{n} + \frac{z^2}{4n^2}}. The center is pulled toward 1/2 and the width never collapses, which is what a small nn needs.
We measure the coverage of both intervals over 20,000 simulated evaluations of 40 episodes each.
Output:
At p=0.03p = 0.03 and p=0.97p = 0.97, the Wald interval covers the truth in only 70% of evaluations, because about 30% of 40-episode evaluations see zero failures (or zero successes) and report an interval of width zero. Wilson stays between 0.93 and 0.97 at every pp. Use Wilson, or an exact interval, for one task or for a small number of episodes; the Wald interval is acceptable only when np^n\hat p and n(1−p^)n(1-\hat p) are both comfortably large.

​
The bootstrap, and which unit to resample

The bootstrap estimates the sampling distribution of a statistic by recomputing it on data resampled with replacement from the observed data. The percentile interval takes the α/2\alpha/2 and 1−α/21-\alpha/2 quantiles of the recomputed values. The one assumption that matters is that the resampled units are independent draws from the population of interest. bootstrap_ci takes one value per unit; paired_bootstrap_diff resamples the same units for two systems.
In an evaluation of NN tasks with nn episodes each, the episodes are not independent: episodes of the same task share its difficulty. Treat the tasks as drawn from a population with mean μ\mu and between-task variance σtask2=Var(pj)\sigma^2_{\text{task}} = \mathrm{Var}(p_j). By the law of total variance, the estimate μ^=1N∑jcj/n\hat\mu = \frac{1}{N}\sum_j c_j/n has Var(μ^)=1N(σtask2+E[pj(1−pj)]n),\mathrm{Var}(\hat\mu) = \frac{1}{N}\Big(\sigma^2_{\text{task}} + \frac{\mathbb{E}[p_j(1-p_j)]}{n}\Big), whereas treating all NnNn episodes as independent assumes μ(1−μ)/(Nn)\mu(1-\mu)/(Nn). Since μ(1−μ)=σtask2+E[pj(1−pj)]\mu(1-\mu) = \sigma^2_{\text{task}} + \mathbb{E}[p_j(1-p_j)], the ratio of the true variance to the naive one is the design effect 1+(n−1) ρ1 + (n-1)\,\rho, with the intra-task correlation ρ=σtask2/(μ(1−μ))\rho = \sigma^2_{\text{task}}/(\mu(1-\mu)). Two consequences follow. When tasks differ (ρ>0\rho > 0), the naive interval is too narrow by a factor 1+(n−1)ρ\sqrt{1 + (n-1)\rho}. And as n→∞n \to \infty the variance does not go to zero but to σtask2/N\sigma^2_{\text{task}}/N: more samples per task cannot compensate for too few tasks. The fix is the cluster bootstrap: resample tasks, carrying each task’s episodes along, which is bootstrap_ci applied to per-task rates. Which interval is right depends on the question, as Miller (2024) emphasizes for language-model evals. “How well does this checkpoint do on these ten tasks?” conditions on the tasks, and only episode sampling is random. “How well will it do on tasks like these?” treats the tasks as a sample. The second is almost always the question. A simulation with a known population separates the two: task pass rates drawn from a U-shaped Beta(0.4, 0.3) distribution (most tasks nearly always or nearly never solved, as in agent benchmarks), ten tasks, 32 episodes each.
Output:
Resampling episodes covers the population mean in 38% of evaluations: its intervals are so narrow that they usually miss. They cover the mean of these ten tasks in 99.8% of evaluations, which is the question they answer (and they are conservative even for that, because the binomial variance μ(1−μ)\mu(1-\mu) overstates the within-task variance). Resampling tasks covers the population mean in 93% of evaluations, close to nominal; the remaining shortfall is the percentile bootstrap’s known undercoverage at N=10N = 10. Figure 17.1 repeats the experiment for nn from 1 to 64 episodes per task. With one episode per task the two intervals coincide; as nn grows, the episode-level interval shrinks toward zero width while the true uncertainty stays at σtask2/N\sigma^2_{\text{task}}/N, and its coverage falls below 0.3. Figure 17.1: Coverage of nominal 95% bootstrap intervals for the population pass rate, resampling episodes versus resampling tasks, as the number of episodes per task grows. Simulation with 10 tasks whose pass rates are drawn from Beta(0.4, 0.3), 300 evaluations per point. The same comparison on the measured counts of Section 17.1, with the moment estimate of the intra-task correlation:
Output:
For the base agent, the two intervals agree (widths 0.075 and 0.081): its tasks are equally hard, ρ≈0.01\rho \approx 0.01, and its episodes behave like independent coin flips. For the trained agent, ρ=0.63\rho = 0.63 and the design effect is 20: the task-level interval, [0.60,0.96][0.60, 0.96], is 4.4 times as wide as the episode-level one. With ten tasks and one of them nearly unsolved, the honest statement is that the trained agent’s held-out pass rate on tasks like these is somewhere between about 0.6 and 0.96. More episodes per task would not narrow that; more tasks would.

​
17.4 Comparing two runs

Two kinds of randomness separate two numbers. Evaluation noise comes from sampling episodes from a fixed checkpoint, and more episodes shrink it. Training noise comes from the seed of the run (task order, sampled groups, minibatch order), and it is a property of the recipe: two runs of the same recipe end at different checkpoints. Whether training noise belongs in the error bar depends on the claim. “Checkpoint B is better than checkpoint A” is a claim about two fixed checkpoints. “Recipe B is better than recipe A” is a claim about distributions of checkpoints, and its unit of replication is the seed. We train three more seeds of the default recipe for 30 steps, a point at which the runs have not saturated, and evaluate each of the four step-30 checkpoints twice, with independent episodes (320 per evaluation). The duplicate evaluations measure the evaluation noise; the spread across seeds contains both kinds.
Output:
The two evaluations of the same checkpoint differ by up to 0.028, and the standard deviation of one 320-episode evaluation is 0.016, close to the binomial 0.71⋅0.29/320=0.025\sqrt{0.71 \cdot 0.29/320} = 0.025 (it estimates the conditional, these-ten-tasks noise, which Section 17.3 showed is smaller). The four seeds’ means, 0.706 to 0.720, spread less than two evaluations of one seed, and the estimated training-noise standard deviation is zero. In this setting, at step 30, the default recipe is highly reproducible across seeds, and almost all of the visible spread is evaluation noise. This also re-reads Chapter 9’s ablation table: its default arm’s three held-out values, 0.76 to 0.81, came from 80 episodes each, whose evaluation standard error is about 0.045, so that spread was mostly evaluation noise too. Neither result says seeds do not matter in general. Seed variance depends on the recipe, the scale, and the point in training, and it has to be measured with independent evaluations, as here, before it can be ignored.

​
Paired comparisons

When two systems run on the same tasks, compare them task by task. The per-task difference dj=p^jA−p^jBd_j = \hat p^{A}_j - \hat p^{B}_j removes the task’s difficulty, which both systems share, so Var(dˉ)=1N(Var(p^jA)+Var(p^jB)−2 Cov(p^jA,p^jB)),\mathrm{Var}(\bar d) = \frac{1}{N}\big(\mathrm{Var}(\hat p^A_j) + \mathrm{Var}(\hat p^B_j) - 2\,\mathrm{Cov}(\hat p^A_j, \hat p^B_j)\big), and the covariance term, large when tasks differ in difficulty, is what an unpaired comparison throws away. We compare the checkpoints at steps 60 and 80 on the same ten tasks, and we also give both the same random generator. Two similar policies driven by the same random numbers tend to sample the same episodes, which correlates their outcomes further. This is the common random numbers technique from simulation; it is legitimate for a paired comparison, and it is why the same generator must not be reused when the goal is to measure evaluation noise, as in the block above.
Output:
The step-80 checkpoint is 0.041 better than the step-60 one, with a paired interval of [+0.006,+0.087][+0.006, +0.087] that excludes zero. Treated as two unpaired samples of ten tasks, the same difference has a standard error of 0.132, six times the paired 0.023, and would be invisible. The per-task rates of the two checkpoints correlate at 0.97: the tasks that are hard for one are hard for the other, and pairing subtracts that shared difficulty. The paired interval answers a question about these tasks and these two checkpoints; it says nothing about the recipe, and with ten tasks it is itself a rough interval.

​
How many tasks or seeds?

A power calculation turns the comparison around: given an effect δ\delta we want to detect, how many units do we need? For a two-sided test at level α\alpha with power 1−β1-\beta, the mean of NN paired differences with per-unit standard deviation σd\sigma_d must satisfy δ/(σd/N)≥z1−α/2+z1−β\delta/(\sigma_d/\sqrt N) \ge z_{1-\alpha/2} + z_{1-\beta}, so N≥((z1−α/2+z1−β) σdδ)2.N \ge \Big(\frac{(z_{1-\alpha/2} + z_{1-\beta})\,\sigma_d}{\delta}\Big)^2. For two independent groups (unpaired tasks, or two recipes with separate seeds), the difference of two means has per-unit standard deviation 2 σ\sqrt2\,\sigma, and the formula gives the size of each group.
We plug in the measured spreads: the per-task difference between two checkpoints, the per-task spread of one checkpoint (for an unpaired design), and the spread across seeds.
Output:
With paired tasks, detecting a 2-point difference between two checkpoints of this kind needs about 103 tasks, and a 5-point difference about 17. An unpaired design (for example, comparing a number from a paper with a number from your harness on different task samples) needs about 30 times as many: 533 tasks per arm for 5 points. The seed column is small because the measured spread across seeds was small; the formula’s answer of one seed is a statement about power, not about practice. At least three seeds per arm are needed just to estimate σseed\sigma_{\text{seed}}, and a recipe comparison should never rest on fewer.
In production. The same arithmetic sets the size of real benchmarks. On SWE-bench Verified’s 500 tasks, the binomial standard error of a pass rate near 0.4 is about 0.022 for one attempt per task, so the difference of two independent single-attempt runs has a standard error of about 0.031, and a 2-point gap is well inside it. Per-task pairing, several attempts per task (as DeepSWE’s average of 16 runs does, Chapter 20), and multiple training seeds are the levers. This is arithmetic from the binomial formula, not a measurement.

​
17.5 The harness defines the number

A harness choice is any setting outside the weights that changes pj(π,H)p_j(\pi, H). If training and evaluation, or two evaluations being compared, differ in one, the comparison measures the harness. We evaluate the same two checkpoints under the training harness and under seven one-field changes.
Output:
Changing one field of the harness moves the trained agent’s number by up to 0.13. A temperature of 1.3 instead of the training temperature 1.0 costs 0.10; lowering it to 0.7, or decoding greedily, adds 0.03 to 0.04, because a sharper distribution samples the policy’s mode, which is right on most tasks. A three-turn budget costs 0.13: the learned procedure is read, edit, test, submit, so episodes that test before submitting run out of turns, and only the episodes that skip the test still pass. The settings that the agent does not use leave its number unchanged: it never stacks edits, so unlimited edits change nothing, and it ignores names, so renaming changes nothing. For the base agent the same settings move the number between 0.116 and 0.200, but that agent guesses, so a harness that allows more attempts or more turns pays it more. Figure 17.2 shows these numbers with task-level intervals. Each harness change is a few lines of configuration in a real system; each is larger than most effects a team tries to measure. Figure 17.2: Held-out pass@1 of the RL base agent and the 80-step GRPO checkpoint under the training harness and seven one-field changes, with task-bootstrap 95% intervals. Dashed: the 0.2 pass rate of a blind single guess.

​
Greedy versus sampled evaluation

Greedy decoding (temperature 0) evaluates the mode of the policy; sampled evaluation at the training temperature evaluates the distribution that training optimized and that a best-of-kk system would draw from. They answer different questions, and the per-task counts show how.
Output:
Greedy decoding is deterministic given the prompt, so its per-task counts are 0 or 32 except where the prompt itself varies. Here it does vary: renaming draws a random function name for every episode, which is why the base agent’s greedy counts are 10 to 16 on four tasks and 0 on the other six. Its greedy behavior is essentially one fixed guess that is right only on the compare and zero tasks, and only for some names, while its sampled behavior spreads over all five repair ops. For the trained agent, greedy decoding removes the residual mistakes on most tasks (under_cap rises from 25 to 30) and turns speed, where the trained agent occasionally succeeds, into a hard zero. Greedy pass@1 is the right number for a deployment that decodes greedily. It is the wrong number for a sampled deployment or a best-of-kk system, and it hides exactly the low-probability successes that the pass@kk debate of Section 17.7 is about. The book’s default is sampled evaluation at the training temperature, with the temperature reported.

​
17.6 Held-out splits and leakage

A held-out set answers a generalization question, and the question depends on what is held out. The default split of Chapter 4 holds out variants: two of the six variants of every bug family, with different function names, constants, and tests. It measures whether the agent generalizes across surface forms of bugs it was trained on. Holding out a family asks whether it generalizes to a kind of bug it has never been rewarded for fixing. We train for 30 steps on the four families other than zero (missing zero guard) and evaluate on all six zero tasks and on the eight held-out variants of the other families.
Output:
The two splits give opposite verdicts on the same checkpoint. On held-out variants of trained families, the agent reaches 0.930 from 0.148. On the unseen family it falls to 0.031, below the base agent’s 0.125. It still reads (1.41 calls per episode) but its first edit is right in 3% of episodes: training made it confident in the four ops it was rewarded for and removed the guessing that gave the base agent its 0.125. A variant split would have reported a strong agent; a family split reports one that has learned four mappings and nothing about missing zero guards. Real coding benchmarks face the same choice at larger scale: held-out issues from training repositories, held-out repositories, or held-out languages measure different things, and a report should name the split. Contamination and leakage. A held-out number is only held out if nothing about the held-out tasks reached training. In the gym, three channels exist, and it is worth checking each. The tokenizer is built from all 30 tasks (build_tokenizer(make_tasks('all'))), so held-out words have embeddings; a word that never occurs in training keeps its random initialization, which is harmless for the answer but is a leak of vocabulary. The base agent was behavior-cloned on training tasks only. The third channel is the one Chapter 9’s Exercise 3 points out: randomize_names draws from all 30 function names, so held-out names appear in training episodes as random names. The block measures how often, and then tests whether it matters by evaluating with the renaming pool restricted to training names or to held-out names.
Output:
About a third of training episodes (0.34, ten of the 29 candidate names) carry a held-out function name. Restricting the held-out evaluation to training names or to held-out names gives 0.844 and 0.863, a difference within the evaluation noise of Section 17.4 (a standard deviation of about 0.016 per evaluation). The trained policy does not use the name, because renaming made the name uninformative during training, so this leak is real but inert for this metric. That is the general shape of a leakage audit: list the channels, then measure whether each one can move the number. The leaks that move numbers on real benchmarks carry answers, not names: an issue text that contains the fix, a benchmark repository in the pretraining or RL data, or test files visible to the agent (Chapter 9 and Chapter 20 cite measured cases). Decontaminate training tasks against every evaluation set by repository, issue, and patch, and keep a fresh evaluation set built after the training data was frozen.

​
17.7 Does RL narrow the distribution? pass@kk on TinyAgent

Chapter 20 summarizes a public debate. Yue et al. (2025) report that on math benchmarks RLVR-trained models win at small kk but “base models achieve a higher pass@k score when k is large”, and read this as RL sharpening the base model’s distribution rather than adding capability; ProRL (2025) reports the opposite after longer training with reference resets. Both are reported by their authors on math with large models, not reproduced here. The measurement itself is cheap with the tools of this chapter: unbiased pass@kk on held-out tasks for the base agent and for RL checkpoints, from the same 64 episodes per task. We add a reference curve, a uniform guess among the five repair ops, which passes with probability 0.2 per attempt.
Output:
The curves cross. At k=1k = 1 the 80-step checkpoint leads the base agent by 0.70; at k=16k = 16 it still leads, 0.958 against 0.941; at k=32k = 32 the base agent is ahead, 0.998 against 0.988. The 20-step checkpoint is at or above both at every kk. The per-task counts locate the crossing in one task, speed. The base agent solves it in 10 of 64 episodes, the 20-step checkpoint in 13, and the 80-step checkpoint in 3. Its pass@32 on speed is therefore about 0.88 against essentially 1 for the base agent, and that one task produces the whole deficit at large kk. What did training do on that task?
Output:
speed contains both a multiplication and an integer division; its bug is the missing zero guard. The base agent’s first edit on it is spread over all six edit ops, add_zero_guard in 13 of 64 episodes. The trained agent applies fix_comparison in 59 of 64 episodes and the right op in 4. Somewhere between steps 20 and 80, the policy’s mapping from source to op became confident and wrong for this task, and the probability of the right op fell from about 0.2 to 0.06. This is narrowing in the precise sense of the debate: RL concentrated probability on the behavior that its other experiences rewarded and removed most of the mass from an alternative that occasionally worked. Figure 17.3 shows the mean curves and the speed task. Figure 17.3: Left: unbiased pass@k on the ten held-out tasks for the RL base agent and the GRPO checkpoints after 20 and 80 steps, from 64 episodes per task, with task-bootstrap bands for the base agent and the 80-step checkpoint; dashed, a uniform guess among five ops. Right: the same curves for zero/speed alone. So, does RL narrow the distribution on TinyAgent? Measured here, yes on one task of ten, and the narrowing appeared late: the 20-step checkpoint dominated the base agent at every kk. Two caveats bound the reading. First, the base agent’s large-kk coverage is not a capability: its curve sits just below the uniform-guess curve, because in a five-op action space 32 guesses almost always include the right op. pass@kk at large kk measures how many attempts a guesser needs whenever the answer space is small, which is also a standing criticism of large-kk pass rates on benchmarks with short answers. Second, the crossing rests on a single task and one training run; with ten tasks, the task-level band of the 80-step curve spans the base agent’s curve at k≥16k \ge 16. The mechanism is nevertheless the one the debate is about, and the per-task view is how to find it: look for tasks whose success count fell during training, and inspect what the policy now does instead.
Evidence. Measured here: one 80-step GRPO run of a 158k-parameter model, ten held-out tasks, 64 episodes per task per checkpoint. Reported by sources: Yue et al. and ProRL (math, large models; Chapter 20). The tiny-scale result illustrates the measurement and one mechanism of narrowing; it is not evidence for either side of the debate at scale.

​
17.8 Choosing a checkpoint without touching the test set

Every choice made by looking at a number turns that number into a training signal. Choosing the checkpoint with the highest test score, then reporting that score, reports the maximum of several noisy estimates, which is biased upward: the winner’s curse. With mm checkpoints of equal true skill and evaluation noise σ\sigma, the expected maximum is about μ+σ E[max⁡i≤mZi]\mu + \sigma\,\mathbb{E}[\max_{i \le m} Z_i], which is μ+1.42σ\mu + 1.42\sigma for m=8m = 8. The fix is the one used for hyperparameters: split the held-out tasks into a validation set used for every decision and a test set evaluated once, for the final report. With ten held-out tasks we can afford only five and five, one variant of each family in each.
Output:
The validation and test columns disagree in level more than in shape. Validation reaches 0.975 at step 80 while test stays near 0.74 from step 40 on, because the test half contains speed and the validation half does not: with five tasks per split, the composition of the split decides the level. Choosing by validation picks step 80, whose test score, 0.744, is an unbiased estimate for the chosen checkpoint because the test set played no part in the choice. Choosing by test picks step 70 and reports 0.762. Whether that particular number is inflated cannot be read from one evaluation. The winner’s curse is a statement about the expected value of the procedure, so we measure it by repeating the procedure: ten small test evaluations (8 episodes per task), each used to pick a checkpoint, and each pick scored on the nine evaluations that did not choose it.
Output:
Different small evaluations pick different checkpoints (steps 30 to 80, most often 50), which already shows that the ranking among the later checkpoints is noise. The pick scores 0.795 on the evaluation that chose it and 0.766 on the evaluations that did not: selecting on the test set inflated the reported number by 0.03, about half a standard error of one of these 40-episode evaluations. The simulation with a 160-episode test set and eight equally good checkpoints gives the same order, 0.04 above the truth. The inflation grows with the number of checkpoints, configurations, and prompts tried against the same test set, which is why a test set that a team evaluates against every week stops being a test set. Figure 17.4 shows the checkpoint table with task-level intervals, and the four seeds of Section 17.4 with both kinds of interval. Figure 17.4: Left: validation and test pass@1 of the checkpoints of one GRPO run (5 tasks each, 32 episodes per task) with task-bootstrap 95% intervals. Right: the four seeds at step 30, each with an episode-level and a task-level bootstrap interval over all ten held-out tasks. The intervals in the left panel overlap almost entirely from step 40 on. With five test tasks, no ranking of those checkpoints is supported, and the defensible report is the validation-chosen checkpoint’s test score with its interval. In a larger evaluation, the same protocol scales: decide on validation (including early stopping, checkpoint averaging, and decoding settings), evaluate on test once per reported model, and report how many models were evaluated on that test set.

​
17.9 Design problems in practice

Problem: evaluation noise larger than the effect. Symptom: an ablation table where arms differ by a few points, a periodic eval curve that zigzags, or a “win” that disappears on rerun. Cause: the evaluation’s standard error, computed at the right unit (Section 17.3), is as large as the effect, and the comparison is unpaired or unreplicated. train_rl’s periodic eval_pass uses 4 episodes per task, 40 on the held-out set; the block re-evaluates one checkpoint eight times at that size. Options: (a) more episodes per task, which helps until the task term σtask2/N\sigma^2_{\text{task}}/N dominates; (b) more tasks, the only fix for that term; (c) paired comparisons on the same tasks, with common random numbers when comparing checkpoints; (d) more training seeds when the claim is about a recipe. Default: before running an ablation, compute the paired standard deviation from a pilot evaluation and use required_units to size it; report intervals at the task level.
Output:
The same checkpoint scores between 0.80 and 0.90 across eight 40-episode evaluations, a range of 0.10. Chapter 9’s preset and ablation tables compared arms evaluated with 80 episodes each, at a comparable noise level, and most of their differences were below this range; that is why Chapter 9 could not rank its variants.
Problem: harness drift. Symptom: a checkpoint’s number changes between two dashboards, a reproduction misses a paper’s number by several points, or an improvement appears exactly when the eval code changed. Cause: pj(π,H)p_j(\pi, H) depends on the harness, and some harness field differs: temperature, budgets, template, the environment’s rules, or the success criterion (Section 17.5). Drift is most dangerous when the changed field is one a weak policy can exploit. The block runs a skill-free policy that applies three fixed ops and submits, under the training harness and under the plain BugFixEnv that allows unlimited edits. Chapter 15’s pass_rate helper evaluates in the plain environment, which matches Chapter 15’s own training setup but not the default setting of Chapter 9 onward, so numbers from the two are not directly comparable. Options: (a) build evaluation from the training configuration object, as sample_counts(..., cfg) does, so that every field is shared by construction; (b) version the harness and record it with every number; (c) run a fixed reference policy (scripted or a frozen checkpoint) through every harness change and require its number to stay put. Default: (a) and (c). A canary policy whose score is known catches drift that no amount of looking at the evaluated model will.
Output:
The same skill-free policy passes 0.20 of held-out tasks under the training harness and 0.60 under the plain environment, because the plain environment lets it stack three guesses (Section 9.4). A model compared across those two harnesses would be credited with a 40-point improvement it did not make.
Problem: an evaluation that reuses the training reward. Symptom: the training reward and the evaluation number rise together, and the behavior behind them is not the intended skill. Cause: when the evaluation computes success with the same function the policy was optimized against, it inherits every flaw the optimizer found: public tests that can be special-cased, a judge that can be flattered, a format check that can be satisfied without solving anything (Chapter 11). The block runs Chapter 4’s scripted hacker, which special-cases the public test inputs, through the gym with the public-test reward and the hidden-test reward. Options: (a) evaluate with a stronger, held-out verifier than the training reward (hidden tests, more tests, a different test generator); (b) audit samples of passing trajectories by hand or with behavior statistics; (c) track the gap between the training reward and the held-out verifier as a dashboard metric. Default: (a) always, and (c) as an alarm. sample_counts scores with the hidden tests whatever cfg.reward says, so an RL run trained on a hackable reward is still evaluated honestly.
Output:
Under its own training reward, the hacker scores a perfect 1.00; under the hidden tests, 0.00. An evaluation that reused the public-test reward would have reported a perfect agent.
Problem: choosing the checkpoint by the test set. Symptom: the reported test score is never matched again, by a rerun, by a fresh test set, or by users. Cause: every decision made by looking at test numbers (the checkpoint, the decoding temperature, the prompt, which of several runs to report) selects on noise, and the reported maximum is biased upward by roughly σ E[max⁡iZi]\sigma\,\mathbb{E}[\max_i Z_i] (Section 17.8, where it was 0.03 to 0.04). Options: (a) a validation split for all decisions and a test split used once; (b) when tasks are too few to split, report the pre-registered final checkpoint rather than the best one; (c) report the number of models evaluated on the test set, so readers can discount. Default: (a), with the selection rule written down before the test evaluation. For long-lived benchmarks, refresh the test set periodically, since a team’s accumulated decisions leak into it even when each one is careful.

​
Summary

  • An evaluation estimates pj(π,H)p_j(\pi, H), a per-task success probability under a harness, and its raw data are per-task success counts. sample_counts produces them with the training configuration, scoring with hidden tests whatever the training reward was.
  • pass@kk needs the unbiased estimator 1−(n−ck)/(nk)1 - \binom{n-c}{k}/\binom{n}{k}, computed as a product; the plug-in 1−(1−p^)k1 - (1-\hat p)^k underestimated pass@5 by 0.04 to 0.08 at n=10n = 10.
  • Episodes of one task are correlated. With the trained agent’s heterogeneous tasks the design effect was 20, and the task-level interval was 4.4 times as wide as the episode-level one; resampling episodes covered the population mean in 38% of simulated evaluations. Use Wilson for single tasks and the task-level bootstrap for benchmarks.
  • Compare on the same tasks: pairing cut the standard error of a checkpoint difference from 0.132 to 0.023. Size comparisons with a power calculation (17 paired tasks for 5 points, 533 per arm unpaired) and measure seed variance with independent evaluations; at step 30 it was indistinguishable from evaluation noise here.
  • One harness field moved the trained agent’s held-out pass rate by up to 0.13, and a looser environment tripled a skill-free policy’s score. Held-out variants (0.93) and a held-out family (0.03) gave opposite verdicts on one checkpoint.
  • The base agent overtook the 80-step checkpoint at k=32k = 32 (0.998 against 0.988) because RL drove the right op on one task from 13 to 4 of 64 episodes; at kk large enough, a guesser in a five-op space covers everything. Selecting on the test set inflated a reported score by about 0.03; decide on validation and report test once.

​
Exercises

  1. Prove that 1−(n−ck)/(nk)1 - \binom{n-c}{k}/\binom{n}{k} is the minimum-variance unbiased estimator of 1−(1−p)k1-(1-p)^k from nn Bernoulli samples, using the fact that cc is a complete sufficient statistic. Then compute its variance for n=64n = 64, k=32k = 32, p=0.05p = 0.05 by simulation and compare it with k=4k = 4.
  2. Change an assumption: rerun Section 17.7 with n=256n = 256 episodes per task and kk up to 128, and with the 40-step checkpoint added. Does the crossing move? Does any task other than speed lose coverage? Plot per-task success counts across all checkpoints and find the step at which speed starts to fall.
  3. Implement a Bayesian alternative to the task-level bootstrap: a Beta prior per task and a hierarchical model for task pass rates, sampled with simple Gibbs or importance sampling in torch. Compare its interval with bootstrap_ci on the counts of Section 17.1.
  4. Repeat the held-out family experiment of Section 17.6 for each of the five families. Is the collapse below the base agent’s guessing rate universal? Relate the result to the entropy of the first edit at the start of an unseen-family episode (Chapter 10).
  5. Write a harness canary: a function that runs a fixed set of scripted policies (oracle, blind, the stacker of Section 17.9) through make_env(cfg) and asserts their pass rates. Which fields of RLConfig does it detect a change in, and which can it miss?
  6. Common random numbers: compare the paired standard error of the step-60 versus step-80 difference with the same generator for both checkpoints and with different generators. Explain the difference with the variance formula of Section 17.4.
  7. Design review: a teammate reports that a new advantage estimator improves SWE-bench Verified pass@1 from 41.0% to 43.4%, from one training run per arm and one sample per task, and that the checkpoint was “the best of the last five”. Using this chapter, list what you would ask for before accepting the claim, estimate the standard error of the reported difference under an unpaired and a paired analysis, and say how many seeds and samples you would request.