> ## Documentation Index
> Fetch the complete documentation index at: https://agent-rl.tacetai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Sources and implementation ledger

Reviewed: 2026-10-13. This ledger identifies what was actually inspected. It is not a comprehensive bibliography.

| Source | Evidence used | Boundary |
| - | - | - |
| [D2L](https://github.com/d2l-ai/d2l-zh) and [softmax source](https://github.com/d2l-ai/d2l-zh/blob/master/chapter_linear-networks/softmax-regression-scratch.md) | Teaching structure and indexing explanation | Reference, no copied prose/code |
| [DeepSWE release report](https://www.together.ai/blog/deepswe) | Coding-agent system motivation | Public report; no benchmark reproduction |
| [Pinned rLLM SWE example](https://github.com/agentica-project/rllm/blob/7b47687f6a9ef1bf5cbd56dd1af61fff08c4b0e4/examples/swe/README.md) | Tool/environment integration reference; README inspected | SHA resolved 2026-10-13; no installation or training run |
| [R2E-Gym](https://github.com/agentica-project/R2E-Gym) | Public repository reference | HEAD resolved to `353348a0ff690f2592025eff41b3fef4201a4d8b`; code not audited |
| [PyTorch gather](https://pytorch.org/docs/stable/generated/torch.gather.html) | Tensor indexing semantics | Educational implementation uses installed PyTorch 2.9.1 |
| [PPO](https://arxiv.org/abs/1707.06347), [GAE](https://arxiv.org/abs/1506.02438) | Sampled-action surrogate and value-based estimator | Tiny implementations, no full critic trainer |
| [DeepSeekMath](https://arxiv.org/abs/2402.03300) | GRPO origin | Does not specify all present-day variants |
| [DAPO](https://arxiv.org/abs/2503.14476), [Dr. GRPO](https://arxiv.org/abs/2503.20783) | Public design alternatives | Not reproduced at benchmark scale |
| [DPO](https://arxiv.org/abs/2305.18290) | Preference objective | Artificial sequence likelihood pairs |
| [On-policy distillation](https://arxiv.org/abs/2306.13649) | Student-prefix teacher supervision | Does not imply one mandatory divergence |
| [Tail-corrected top-k](https://arxiv.org/html/2609.34447v1) | Fixed-prefix gradient proposal; full text inspected | September 2026 preprint, not reproduced |
| [SparseOPD](https://arxiv.org/abs/2609.34386) | Vocabulary selection research direction | Abstract inspected; no implementation audit |
| [DIAL-OPD](https://arxiv.org/html/2610.11659v1) | Timestep supervision allocation; full text inspected | October 2026 preprint, author-reported results |
| [Control-variate OPD](https://arxiv.org/abs/2605.07865) | Variance-reduction proposal | Abstract inspected; no implementation audit |
| [Entropy-Aware OPD](https://proceedings.mlr.press/v306/jin26e.html) | Divergence selection research direction | Abstract inspected; no implementation audit |
| [verl torch kernels](https://github.com/verl-project/verl/blob/00094bd9cd3fef9cf8903daf60ea4a2bcf832efc/verl/utils/torch_functional.py) | Selected log-probability paths inspected locally | Pinned SHA; hardware branches differ |
| [verl core algorithms](https://github.com/verl-project/verl/blob/00094bd9cd3fef9cf8903daf60ea4a2bcf832efc/verl/trainer/ppo/core_algos.py) | Group IDs, standard deviation, response-shaped advantages | Actual helper differs from our explicit population-std teaching option |
| [Mintlify settings](https://www.mintlify.com/docs/organize/settings), [CLI](https://www.mintlify.com/docs/cli/install) | docs.json and MDX reading interface | Locally installed CLI may predate newest commands |
| [rLLM](https://github.com/rllm-org/rllm) | Ch20: framework behind DeepSWE; README inspected 2026-10-13 | HEAD `4e828c81680e5a85b638ceedeaf09d1e90109ca8` (old `agentica-project/rllm` archived at `7b47687`); code not audited |
| [R2E-Gym paper](https://arxiv.org/abs/2504.07164) | Ch20: environment curation, SFT training, hybrid verifiers; abstract and full text inspected 2026-10-13 | Author-reported 34.4% pass\@1 and 51.0% Best\@26; repo HEAD `0d94c4eb9431cd195c55a7ea3abd54006c9a1735` |
| [SWE-Gym](https://arxiv.org/abs/2412.21139) | Ch20: 2,438 tasks, RFT and verifier results; abstract and full text inspected 2026-10-13 | Not RL; author-reported numbers |
| [SWE-RL](https://arxiv.org/abs/2502.18449) | Ch20: single-turn GRPO with patch-similarity reward; abstract and full text inspected 2026-10-13 | 41.0% uses 500 samples plus reranking |
| [SkyRL v0 README](https://github.com/NovaSky-AI/SkyRL/tree/a0d50c482436af7fac8caffa4533616a78431d66), [SkyRL-Agent](https://arxiv.org/abs/2511.16108) | Ch20: multi-turn RL results and recipe; README at v0 commit and paper full text inspected 2026-10-13 | SkyRL-v0 Notion blog did not render; repo HEAD `a1e34dbf21003f66bd7921487fd47db6c154aba2` |
| [Golubev et al. 2508.03501](https://arxiv.org/abs/2508.03501) | Ch20: RFT then DAPO for SWE agents | Abstract only, 2026-10-13 |
| [Kimi-Dev](https://arxiv.org/abs/2509.23045) | Ch20: workflow RL, test-writer reward, selection vs pass\@1 | Abstract and full text, 2026-10-13; 60.4% uses test-time selection |
| [LEGO-RL](https://arxiv.org/abs/2608.17393) | Ch20: harness-native GSPO for coding agents | Abstract only, 2026-10-13 |
| [SWE-smith](https://arxiv.org/abs/2504.21798), [SWE-rebench](https://arxiv.org/abs/2505.20411), [SWE-rebench V2](https://arxiv.org/abs/2602.23866), [Skywork-SWE](https://arxiv.org/abs/2506.19290), [SWE-Factory](https://arxiv.org/abs/2506.10954) | Ch20: task mining and synthesis | Abstracts only, 2026-10-13 |
| [Absolute Zero](https://arxiv.org/abs/2505.03335), [Self-play SWE-RL](https://arxiv.org/abs/2512.18552) | Ch20: self-play rewards (learnability; injector/solver) | Abstract and full text, 2026-10-13; SSR absolute scores not extracted |
| [GSPO](https://arxiv.org/abs/2507.18071), [MiniMax-M1 / CISPO](https://arxiv.org/abs/2506.13585), [RLOO](https://arxiv.org/abs/2402.14740) | Ch20: objective definitions, clip ranges, MoE and FP32-head claims | Full text, 2026-10-13; not reproduced |
| [VAPO](https://arxiv.org/abs/2504.05118), [DeepSeek-R1](https://arxiv.org/abs/2501.12948) | Ch20: value-based variant; R1 context | Abstracts only, 2026-10-13 |
| [Entropy Mechanism](https://arxiv.org/abs/2505.22617), [80/20 high-entropy tokens](https://arxiv.org/abs/2506.01939), [Reasoning with Exploration](https://arxiv.org/abs/2506.14758), [ARPO](https://arxiv.org/abs/2507.19849) | Ch20: entropy dynamics and exploration | Entropy Mechanism full text; others abstract only, 2026-10-13 |
| [Yue et al.](https://arxiv.org/abs/2504.13837), [ProRL](https://arxiv.org/abs/2505.24864), [Invisible Leash](https://arxiv.org/abs/2507.14843), [CoT-Pass@K](https://arxiv.org/abs/2506.14245), [Pass@k Training](https://arxiv.org/abs/2508.10751), [RL Squeezes](https://arxiv.org/abs/2509.21128), [Spurious Rewards](https://arxiv.org/abs/2506.10947), [One-shot RLVR](https://arxiv.org/abs/2504.20571) | Ch20: pass\@k capability debate | Yue and ProRL full text; others abstract only, 2026-10-13; all math-domain |
| [ArCHer](https://arxiv.org/abs/2402.19446), [RAGEN](https://arxiv.org/abs/2504.20073), [GiGPO](https://arxiv.org/abs/2505.10978), [Wei et al. 2505.11821](https://arxiv.org/abs/2505.11821), [SWEET-RL](https://arxiv.org/abs/2503.15478), [TRACE](https://arxiv.org/abs/2607.13988), [SWE-MILE](https://arxiv.org/abs/2609.32631) | Ch20: multi-turn credit assignment | RAGEN and GiGPO full text; others abstract only, 2026-10-13 |
| [MEM1](https://arxiv.org/abs/2506.15841), [SUPO](https://arxiv.org/abs/2510.06727), [Context-Folding](https://arxiv.org/abs/2510.11967), [ReSum](https://arxiv.org/abs/2509.13313), [FoldAct](https://arxiv.org/abs/2512.22733), [CAMG-RL](https://arxiv.org/abs/2609.34422) | Ch20: context management in agent RL | Abstracts only, 2026-10-13; table numbers not used |
| [Let's Verify Step by Step](https://arxiv.org/abs/2305.20050), [Math-Shepherd](https://arxiv.org/abs/2312.08935), [PRIME](https://arxiv.org/abs/2502.01456), [SWE-RM](https://arxiv.org/abs/2512.21919), [Agent-as-a-Judge](https://arxiv.org/abs/2410.10934), [Rubrics as Rewards](https://arxiv.org/abs/2507.17746), [One Token to Fool](https://arxiv.org/abs/2507.08794) | Ch20: process rewards, verifiers, judges | Abstracts only, 2026-10-13 |
| [AReaL](https://arxiv.org/abs/2505.24298), [PipelineRL](https://arxiv.org/abs/2509.19128), [Magistral](https://arxiv.org/abs/2506.10910), [Kimi k1.5](https://arxiv.org/abs/2501.12599), [Ring-1T / IcePop](https://arxiv.org/abs/2510.18855) | Ch20: async designs, staleness, partial rollouts, mismatch masking | Full text, 2026-10-13; AReaL speedup stated as 2.77x (abstract) and 2.57x (intro) |
| [LlamaRL](https://arxiv.org/abs/2505.24034), [ROLL Flash](https://arxiv.org/abs/2510.11345), [Trust Region Masking](https://arxiv.org/abs/2512.23075), [HybridFlow](https://arxiv.org/abs/2409.19256), [OpenRLHF paper](https://arxiv.org/abs/2405.11143) | Ch20: async systems and sequence-level masking | Abstracts only, 2026-10-13 |
| [TIS blog](https://fengyao.notion.site/off-policy-rl), [When Speed Kills Stability](https://yingru.notion.site/When-Speed-Kills-Stability-271211a558b7808d8b12d403fd15edda), [Defeating Nondeterminism](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/) | Ch20: training-inference mismatch and corrections | TIS and nondeterminism posts full text; Speed Kills partially read, 2026-10-13 |
| [verl docs](https://github.com/volcengine/verl/tree/76e837e0bfcb18b380a66c0ba56ef9f0ac840f09) (one\_step\_off, fully\_async, rollout\_corr) | Ch20: async modes and rollout-correction presets | Docs at `76e837e` read 2026-10-13; code not audited |
| [slime](https://github.com/THUDM/slime/tree/05e3b017fed9e02de418c2fbac9d75602a534be5), [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF/tree/dc2a7ad326f619fc5a47737ea47d1920ce2c0b53) | Ch20: framework architecture and async options | READMEs at pinned SHAs, 2026-10-13 |
| [MiniLLM](https://arxiv.org/abs/2306.08543), [Qwen3 report](https://arxiv.org/abs/2505.09388), [OPD blog](https://thinkingmachines.ai/blog/on-policy-distillation/), [TCOD](https://arxiv.org/abs/2604.24005), [ReOPD](https://arxiv.org/abs/2607.04763) | Ch20: on-policy distillation evidence | Qwen3 Table 21 and OPD blog full text; others abstract only, 2026-10-13 |
| [SWE-bench](https://arxiv.org/abs/2310.06770), [SWE-bench Verified card](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified), [Multimodal](https://arxiv.org/abs/2410.03859), [Multi-SWE-bench](https://arxiv.org/abs/2504.02605), [SWE-bench-Live](https://arxiv.org/abs/2505.23419), [Terminal-Bench](https://arxiv.org/abs/2601.11868), [LiveCodeBench](https://arxiv.org/abs/2403.07974) | Ch20: benchmark definitions | Abstracts and dataset card, 2026-10-13; OpenAI Verified announcement returned 403 and is not cited |
| [SWE-Bench Pro](https://arxiv.org/abs/2509.16941), [SWE-Bench Pro Verified](https://arxiv.org/abs/2609.08149) | Ch20: contamination-resistant design; git-object leak | Full text (Pro), partial full text (Pro Verified), 2026-10-13 |
| [SWE-Bench+](https://arxiv.org/abs/2410.06992), [SWE-Bench Illusion](https://arxiv.org/abs/2506.12286), [UTBoost](https://arxiv.org/abs/2506.09289), [SWE-bench issue #465](https://github.com/SWE-bench/SWE-bench/issues/465), [METR note](https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/) | Ch20: benchmark validity problems | Abstracts; issue page and METR note read in full, 2026-10-13 |
| [DeepSWE release report](https://www.together.ai/blog/deepswe) (2025-07-02), re-checked 2026-10-13 for Chapter 1 | Base model Qwen3-32B; rLLM; 4.5K R2E-Gym problems with SWE-bench Verified repositories filtered out; GRPO++ modifications (clip-higher, no KL loss, no reward std normalization, length normalization, leave-one-out, compact filtering, no entropy loss); reported 42.2% pass\@1 (avg of 16), 59% hybrid test-time scaling, 64 H100 GPUs for six days | Blog report inspected, not reproduced; the report gives both 59% and 59.2% for the hybrid result |
| [SWE-bench](https://arxiv.org/abs/2310.06770) (Jimenez et al., v1 2023-10-10), checked 2026-10-13 | 2,294 task instances from real GitHub issues and pull requests in 12 Python repositories; best evaluated model in the paper resolved 1.96% | Abstract page inspected; harness and SWE-bench Verified construction not inspected |
| [SWE-RL](https://arxiv.org/abs/2502.18449) (Wei et al., v1 2025-02-25), checked 2026-10-13 | RL on Llama 3 with software-evolution data; rule-based reward from similarity to the ground-truth solution; reported 41.0% on SWE-bench Verified (70B) | Abstract page inspected; exact similarity function not checked |
| [R2E-Gym](https://arxiv.org/abs/2504.07164) (Jain et al., v1 2025-04-09), checked 2026-10-13 | Procedural environments from commits with test generation and back-translation; more than 8.7K tasks; reported 34.4% pass\@1 (32B) and 51% with hybrid execution-based and execution-free verifiers on SWE-bench Verified | Abstract page inspected; code not audited |
| [Approximating KL Divergence](http://joschu.net/blog/kl-approx.html) (Schulman, posted 2020-03-07), checked 2026-10-13 | Definitions of the k1, k2, k3 estimators with r = p(x)/q(x), x \~ q; k1 and k3 unbiased, k2 biased with low variance | Blog post inspected; used in Appendix A and Chapter 10 |
| [Qwen3-0.6B model card](https://huggingface.co/Qwen/Qwen3-0.6B) | Recommended sampling (thinking: T=0.6, top-p 0.95, top-k 20; non-thinking: T=0.7, top-p 0.8, top-k 20; no greedy); history should keep only final output, not thinking (Ch. 3, 5) | Inspected 2026-10-13; tokenizer, template, and weights used from the check host's local HF cache (no `generation_config.json` cached) |
| [rLLM `BaseEnv` at pinned SHA](https://github.com/agentica-project/rllm/blob/7b47687f6a9ef1bf5cbd56dd1af61fff08c4b0e4/rllm/environments/base/base_env.py) | `reset() -> tuple[dict, dict]`, `step(action) -> tuple[Any, float, bool, dict]`, `from_dict` (Ch. 4) | Inspected 2026-10-13; signatures only, not run |
| [R2E-Gym README at pinned SHA](https://github.com/agentica-project/R2E-Gym/blob/353348a0ff690f2592025eff41b3fef4201a4d8b/README.md) | 8.1K+ problems across 13 repos, `RepoEnv`, `apply_patch`, Docker images of \~300-500 MB per instance (Ch. 4) | Inspected 2026-10-13; README only, code not audited |
| [SWE-bench dataset card](https://huggingface.co/datasets/princeton-nlp/SWE-bench) | Field definitions of `FAIL_TO_PASS`, `PASS_TO_PASS`, `test_patch`, `base_commit` (Ch. 4) | Inspected 2026-10-13 |
| [SWE-bench evaluation guide](https://www.swebench.com/SWE-bench/guides/evaluation/) | Patches applied and tests run in Docker; image cache levels `base`/`env`/`instance` (Ch. 4) | Inspected 2026-10-13; harness not run |
| [MCP tools specification 2025-06-18](https://modelcontextprotocol.io/specification/2025-06-18/server/tools) | Tool definition (`name`, `description`, `inputSchema`), `isError` results, server input-validation requirements (Ch. 4) | Inspected 2026-10-13 |
| [Williams (1992)](https://doi.org/10.1007/BF00992696), "Simple statistical gradient-following algorithms for connectionist reinforcement learning", *Machine Learning* 8 | Origin of the REINFORCE name and estimator (Ch. 6) | Bibliographic citation only, 2026-10-13; derivation in Ch. 6 is self-contained |
| [GAE](https://arxiv.org/abs/1506.02438) (Schulman et al., v1 2015-06-08), checked 2026-10-13 | Exponentially weighted advantage estimator analogous to TD(λ); value functions reduce variance "at the cost of some bias" (Ch. 7) | Abstract page inspected; Ch. 7 derivation and λ measurements are the book's own |
| [DeepSeekMath](https://arxiv.org/abs/2402.03300) (Shao et al., v1 2024-02-05), checked 2026-10-13 | GRPO introduced as a PPO variant (Ch. 7 attribution of the group-mean baseline) | Abstract page inspected; the abstract does not state the advantage formula, which Ch. 7 states as the common implementation |
| [Back to Basics / RLOO](https://arxiv.org/abs/2402.14740) (Ahmadian et al., v1 2024-02-22), checked 2026-10-13 | REINFORCE-style optimization outperforming PPO in RLHF; leave-one-out baseline for LLMs (Ch. 7) | Abstract page inspected; the leave-one-out identity in Ch. 7 is derived, not quoted |
| [Kool, van Hoof, Welling (2019)](https://openreview.net/forum?id=r1lgTGL5DE), "Buy 4 REINFORCE Samples, Get a Baseline for Free!" (ICLR 2019 workshop) | Leave-one-out baseline over multiple samples (Ch. 7) | Bibliographic citation only, 2026-10-13 |
| [Dr. GRPO](https://arxiv.org/abs/2503.20783) (Liu et al., v1 2025-03-26), checked 2026-10-13 | Sec. 3.1: response-length bias from 1/\|o\_i\| and question-level difficulty bias from dividing by std; Sec. 3.2: remove both, use a constant normalizer such as the generation budget (Ch. 6 aggregation, Ch. 7 std normalization) | HTML full text sections 3.1-3.2 read; experiments not reproduced; TinyAgent numbers in Ch. 6-7 are the book's own |
| [STaR](https://arxiv.org/abs/2203.14465) (Zelikman et al., v1 2022-03-28), checked 2026-10-13 | Ch15: generate rationales, keep those leading to correct answers, fine-tune, repeat; rationalization given the correct answer | Abstract page inspected; not reproduced |
| [RFT for math reasoning](https://arxiv.org/abs/2308.01825) (Yuan et al., v1 2023-08-03), checked 2026-10-13 | Ch15: rejection-sampling fine-tuning; more distinct reasoning paths give larger gains; helps weaker models more | Abstract page inspected; numbers not used |
| [DPO](https://arxiv.org/abs/2305.18290) (Rafailov et al., v1 2023-05-29), checked 2026-10-13 | Ch15: closed-form reparameterization of the KL-regularized objective into a classification loss without sampling | Abstract page inspected; derivation re-done in the chapter |
| [Iterative RPO](https://arxiv.org/abs/2404.19733) (Pang et al., v1 2024-04-30), checked 2026-10-13 | Ch15: DPO loss with an added NLL term, which the abstract calls "crucial" | Abstract page inspected; not reproduced |
| [DAgger / no-regret imitation](https://arxiv.org/abs/1011.0686) (Ross, Gordon, Bagnell, v1 2010-11-02), checked 2026-10-13 | Ch16: sequential prediction violates i.i.d. assumptions; iterative learner-state data collection | Abstract page inspected; the $\epsilon H^2$ vs $\epsilon H$ bound is cited from the paper's known analysis, not re-derived from the full text |
| [GKD](https://arxiv.org/abs/2306.13649) (Agarwal et al., v1 2023-06-23), checked 2026-10-13 for Ch16 | Train/inference mismatch of KD; student trained on self-generated outputs with teacher feedback; alternative divergences; combinable with RL fine-tuning | Abstract page inspected |
| [MiniLLM](https://arxiv.org/abs/2306.08543) (Gu et al., v1 2023-06-14), checked 2026-10-13 for Ch16 | Reverse KL instead of forward KL, optimized on-policy; reported lower exposure bias, 120M-13B | Abstract page inspected |
| [Qwen3 Technical Report](https://arxiv.org/abs/2505.09388) (v1 2025-05-14), checked 2026-10-13 for Ch16 | Sec. 4.5 strong-to-weak distillation: off-policy then on-policy, logits aligned by minimizing KL; Table 21 (Qwen3-8B, from the same off-policy-distilled checkpoint at AIME'24 55.0): RL 67.6 at 17,920 GPU hours, on-policy distillation 74.4 at 1,800 GPU hours | HTML full text, Section 4.5 and Table 21 only |
| [On-Policy Distillation](https://thinkingmachines.ai/blog/on-policy-distillation/) (Kevin Lu, Thinking Machines Lab, 2025-10-27), checked 2026-10-13 for Ch16 | Per-token advantage = negative reverse KL on sampled tokens; discount factor zero; no top-k logit distillation; Qwen3-8B-Base after SFT (60% AIME'24) to 70% in about 150 steps; 9-30x compute reduction vs extrapolated SFT depending on accounting | Full post; a June 2026 update notes the cookbook models changed |
| [PPO](https://arxiv.org/abs/1707.06347) (Schulman et al., v1 2017-07-20), checked 2026-10-13 for Ch8 | Clipped surrogate objective; multiple epochs of minibatch updates per batch | Abstract page inspected; Ch8 derivations and TinyAgent measurements are the book's own |
| [TRPO](https://arxiv.org/abs/1502.05477) (Schulman et al., v1 2015-02-19), checked 2026-10-13 for Ch8 | Surrogate lower bound with a KL penalty; KL-constrained natural-gradient step with line search | Bibliographic and method summary only; bound stated in Ch8 in qualitative form |
| Kakade and Langford (2002), "Approximately Optimal Approximate Reinforcement Learning", ICML | Performance-difference identity used for the per-token surrogate (Ch8) | Bibliographic citation only, 2026-10-13; derivation in Ch8 is self-contained |
| [Dual-clip PPO](https://arxiv.org/abs/1912.09729) (Ye et al., v1 2019-12-20), checked 2026-10-13 for Ch8 | Abstract lists "dual-clip PPO" among the method's strategies | Abstract page inspected; the c = 3 value in Ch8 is a common choice, not checked against the paper |
| [What Matters In On-Policy RL?](https://arxiv.org/abs/2006.05990) (Andrychowicz et al., v1 2020-06-10), checked 2026-10-13 for Ch8 | Recommendation "Use GAE with λ=0.9 but neither Huber loss nor PPO-style value loss clipping"; "PPO-style value loss clipping hurts the performance regardless of the clipping threshold" | HTML full text, first 100k characters; continuous control, not language models |
| [DeepSeekMath](https://arxiv.org/abs/2402.03300) (Shao et al.), re-checked 2026-10-13 for Ch9 | KL coefficient 0.04; 64 outputs per question; policy lr 1e-6; "a single update following each exploration stage"; k3-style KL estimator added to the loss, not the reward | HTML full text, first 100k characters (the page renders the group size as "6464") |
| [SWE-Bench+](https://arxiv.org/abs/2410.06992) (Aleithan et al., v1 2024-10-09), re-checked 2026-10-13 for Ch9 | 32.67% of successful patches had the solution "directly provided in the issue report"; 31.08% suspicious due to weak tests | Abstract page inspected |
| [SWE-Bench Illusion](https://arxiv.org/abs/2506.12286) (Liang et al., v1 2025-06-14), re-checked 2026-10-13 for Ch9 | Buggy file paths identified from issue text alone with "up to 76% accuracy", "up to 53%" on repositories outside SWE-Bench | Abstract page inspected |
| DAPO, Dr. GRPO, GSPO, MiniMax-M1/CISPO, RLOO (rows above), used for Ch9 | Variant definitions in Ch9's diff table are taken from the Ch20 summaries of these sources | Not re-read for Ch9; TinyAgent presets are not reproductions of the papers' settings |
| [verl](https://github.com/verl-project/verl/tree/00094bd9cd3fef9cf8903daf60ea4a2bcf832efc) at `00094bd` (project-pinned commit in `upstream.lock.json`), inspected 2026-10-13 for Ch10 | `core_algos.kl_penalty`: k1 (`kl`), `abs`, k2 (`mse`), k3 (`low_var_kl`, log-ratio clamped to ±20 and value to ±10); a `+` suffix applies a straight-through k2 gradient, with the comment that k1 and k3 have the right expected value but not the right expected gradient. `actor.yaml`: `use_kl_loss: false`, `kl_loss_coef: 0.001`, `kl_loss_type: low_var_kl`, `entropy_coeff: 0`. `ray_trainer.apply_kl_penalty`: in-reward KL `token_level_scores - beta * kld` with an adaptive or fixed controller (`algorithm.kl_penalty: kl`). `workers/utils/losses.py`: `policy_loss -= entropy_coeff * entropy_loss`; `policy_loss += kl_loss * kl_loss_coef` | Source read on the vp host's `.upstream/verl` checkout; not run |
| [DAPO](https://arxiv.org/abs/2503.14476) (v1 2025-03-18), re-checked 2026-10-13 for Ch10 | Sec. 2.3 removes the KL term: the long-CoT "model distribution can diverge significantly from the initial model, thus this restriction is not necessary"; Sec. 3.1 entropy "decreases quickly"; Clip-Higher because "the probability of an unlikely 'exploration' token is too tightly bounded to be uplifted", epsilon\_low = 0.2, epsilon\_high = 0.28 | HTML full text, the quoted sections |
| [ProRL](https://arxiv.org/abs/2505.24864) (v1 2025-05-30), re-checked 2026-10-13 for Ch10 | Sec. 2.3.1: a KL penalty "provides a stronger and more stable solution" and "helps maintain entropy"; "the KL term may increasingly dominate the loss"; periodic hard reset of the reference to a recent policy snapshot with optimizer reinitialization, triggered when validation stagnates or degrades; Sec. 2.2.1: with raised temperature "entropy continues to decline steadily" | HTML full text, the quoted sections; the KL coefficient value was not found in the text read |
| [The Entropy Mechanism of RL for Reasoning LMs](https://arxiv.org/abs/2505.22617) (Cui et al., v1 2025-05-28), used for Ch10 | Entropy change as a covariance between log-probability and the logit update (Ch10 re-derives the first-order identity itself) | See the Ch20 row above; not re-read for Ch10 |
| [Ng, Harada, Russell (1999)](https://dl.acm.org/doi/10.5555/645528.657613), "Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping", ICML, checked 2026-10-13 for Ch11 | Potential-based shaping $\gamma\Phi(s') - \Phi(s)$ preserves optimal policies, and is the only form that does without further assumptions | Bibliographic metadata via Semantic Scholar; the PDF text was not machine-readable. The sufficiency proof in Ch11 is re-derived and checked numerically, not quoted |
| [Christiano et al. (2017)](https://arxiv.org/abs/1706.03741), "Deep reinforcement learning from human preferences" (v1 2017-06-12), checked 2026-10-13 for Ch11 | Reward models fit to pairwise human comparisons as the RL reward | Abstract page inspected; the Bradley–Terry loss in Ch11 is the standard form, written out by the book |
| Bradley and Terry (1952), "Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons", *Biometrika* 39 | Origin of the paired-comparison model used for reward models (Ch11) | Bibliographic citation only, 2026-10-13 |
| [Gao, Schulman, Hilton (2022)](https://arxiv.org/abs/2210.10760), "Scaling Laws for Reward Model Overoptimization" (v1 2022-10-19), checked 2026-10-13 for Ch11 | A gold reward first improves and then degrades as a proxy reward model is optimized; functional forms differ for RL and best-of-n and scale with RM size | Abstract page inspected; no numbers used |
| [Baker et al. (2025)](https://arxiv.org/abs/2503.11926), "Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation" (v1 2025-03-14), checked 2026-10-13 for Ch11 | o3-mini monitored for reward hacking in agentic coding environments; a CoT monitor in the training reward helps "at low optimization levels"; with more optimization, "obfuscated reward hacking, hiding their intent within the CoT" | Abstract page inspected; the abstract does not name specific exploits such as test special-casing |
| [Let's Verify Step by Step](https://arxiv.org/abs/2305.20050), [Math-Shepherd](https://arxiv.org/abs/2312.08935), [One Token to Fool LLM-as-a-Judge](https://arxiv.org/abs/2507.08794), [Rubrics as Rewards](https://arxiv.org/abs/2507.17746) (rows above), used for Ch11 | Process reward models; judge "master key" false positives; rubric rewards | Taken from the Ch20 summaries of these sources; not re-read for Ch11 |
| [Pardo, Tavakoli, Levdik, Kormushev (2017)](https://arxiv.org/abs/1712.00378), "Time Limits in Reinforcement Learning" (v1 2017-12-01), checked 2026-10-13 for Ch12 | If the time limit is part of the task, include remaining time in the input; if it only facilitates learning, bootstrap from the value at the end of a partial episode | Abstract page inspected |
| [Lindenbauer et al. (2025)](https://arxiv.org/abs/2508.21433), "The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management" (v1 2025-08-29), checked 2026-10-13 for Ch12 | In SWE-agent on SWE-bench Verified, observation masking halves cost relative to the raw agent and matches or slightly exceeds LLM summarization's solve rate (five model configurations) | Abstract page (v3) inspected; solve-rate figures not extracted |
| MCP tools specification, DAPO overlong filtering, DeepSWE compact filtering (rows above), used for Ch12 | Tool errors returned as `isError` results; masking truncated episodes from the loss | Taken from the Ch4, Ch9, and Ch20 records of these sources |
| [Chen et al. (2021)](https://arxiv.org/abs/2107.03374), "Evaluating Large Language Models Trained on Code" (v1 2021-07-07), checked 2026-10-13 for Ch17 | Sec. 2.1: unbiased pass\@k estimator $1 - \binom{n-c}{k}/\binom{n}{k}$ with $n \ge k$ (they use $n = 200$, $k \le 100$); direct computation is numerically unstable, so they give a product-form numpy implementation; the plug-in $1-(1-\hat p)^k$ is biased (underestimate, Appendix A); abstract reports 28.8% pass\@1 on HumanEval and 70.2% with 100 samples per problem | Abstract page and ar5iv full text (first 100k characters) inspected; the unbiasedness argument and product form in Ch17 are re-derived |
| [Miller (2024)](https://arxiv.org/abs/2411.00640), "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations" (v1 2024-11-01), checked 2026-10-13 for Ch17 | Evals as experiments; questions treated as samples from a larger unseen population; formulas for analyzing evals, comparing two models, and planning sample sizes | Abstract page inspected; Ch17's clustered-variance, paired, and power formulas are derived in the chapter, not quoted |
| Wilson (1927), "Probable inference, the law of succession, and statistical inference", *JASA* 22; Efron (1979), "Bootstrap methods: another look at the jackknife", *Annals of Statistics* 7 | Origins of the score interval and the bootstrap (Ch17) | Bibliographic citations only, 2026-10-13; Ch17 derives and checks both numerically |
| Yue et al., ProRL, Invisible Leash (rows above), used for Ch17 | The pass\@k narrowing debate that Ch17 measures on TinyAgent | Taken from the Ch20 summaries; not re-read for Ch17 |
| [Ring-1T / IcePop](https://arxiv.org/abs/2510.18855) (v1 2025-10-21), re-checked 2026-10-13 for Ch13 | Masking function $M(k) = k$ for $k \in [\alpha, \beta]$ and 0 otherwise, multiplying the clipped surrogate; recipe uses $\alpha = 0.5$, $\beta = 5$; a narrow band \[0.5, 2.0] "immediately destabilizes training"; mismatch grows without mitigation | HTML full text, first 100k characters; `mis_weights` in Ch13 implements this weight |
| TIS blog, AReaL, PipelineRL, Kimi k1.5, When Speed Kills Stability, Defeating Nondeterminism, MiniMax-M1, verl docs, slime, OpenRLHF (rows above), used for Ch13 | Async designs (staleness bound, interruptible/partial rollouts), decoupled objective, TIS cap 2, low-probability concentration of mismatch, batch invariance, FP32 LM head | Taken from the Ch20 summaries; not re-read for Ch13; the simulator and TinyAgent/Qwen3 measurements are the book's own |
| [NVIDIA H200](https://www.nvidia.com/en-us/data-center/h200/) product page, checked 2026-10-13 for Ch14 | H200 SXM: 141 GB, 4.8 TB/s memory bandwidth, BF16 Tensor Core 1,979 TFLOPS "with sparsity" (989 dense used in Ch14) | Vendor specification; achieved bandwidth and FLOPs are lower |
| Qwen3-8B and Qwen3-0.6B `config.json` in the check host's Hugging Face cache, read 2026-10-13 for Ch13-14 | Qwen3-8B: 36 layers, 8 KV heads, head dim 128, hidden 4096, vocab 151,936 (KV bytes per token in Ch14); Qwen3-0.6B: tied embeddings, hidden 1024 | Local files; parameter counts 8.2B and 32.8B in Ch13 are rounded model-card figures, not re-checked |
| [verl](https://github.com/verl-project/verl/tree/00094bd9cd3fef9cf8903daf60ea4a2bcf832efc) at `00094bd`, inspected 2026-10-13 for Ch18 | `trainer/ppo/metric_utils.compute_data_metrics`: `critic/score`, `critic/rewards`, `critic/advantages`, `critic/returns` (mean/max/min), `critic/vf_explained_var` when a critic is used, `response_length/mean`, `max`, `min`, `clip_ratio`; `core_algos` policy losses return `pg_clipfrac`, `pg_clipfrac_lower`, and `ppo_kl` (masked mean of the negative log-ratio, k1); `ray_trainer` logs `actor/entropy` and, when `rollout_log_probs` is in the batch, `utils/debug/metrics.calculate_debug_metrics`: `training/rollout_probs_diff_{mean,max,std}` (absolute differences of token probabilities, despite the helper's name `calculate_log_prob_diff`) and a rollout–actor Pearson correlation on the response mask | Source read on the vp host's `.upstream/verl` checkout; not run; default enablement of each metric not audited |
| Defeating Nondeterminism, Ch13 Section 13.6, Ch11 hacking run (rows and chapters above), used for Ch18 | Batch invariance as the numerical source of irreproducibility; bf16 batch-composition log-prob differences; the public-test hacking setup | Taken from the earlier records; Ch18's ten failure runs and invariant matrix are the book's own TinyAgent measurements |
| [vLLM / PagedAttention](https://arxiv.org/abs/2309.06180) (Kwon et al., v1 2023-09-12), checked 2026-10-13 for Ch19 | KV-cache memory managed with paging, "near-zero waste in KV cache memory", reported 2-4x throughput over FasterTransformer and Orca | Abstract page only; not run |
| [SGLang](https://arxiv.org/abs/2312.07104) (Zheng et al., v1 2023-12-12), checked 2026-10-13 for Ch19 | Runtime with RadixAttention for KV-cache reuse; reported up to 6.4x throughput on agent, reasoning, JSON, and multi-turn workloads | Abstract page only; not run |
| [OpenRLHF paper](https://arxiv.org/abs/2405.11143) (v1 2024-05-20), re-checked 2026-10-13 for Ch19 | Abstract: "built upon Ray, vLLM, DeepSpeed, and HuggingFace Transformers" | Abstract only |
| [HybridFlow](https://arxiv.org/abs/2409.19256) (verl; v1 2024-09-28), re-checked 2026-10-13 for Ch19 | Hybrid single-/multi-controller programming model for RLHF dataflows; reported 1.53x-20.57x throughput over baselines | Abstract only |
| DeepSWE, SkyRL-Agent, R2E-Gym, SWE-Gym, SWE-smith, SWE-rebench, rLLM, verl (`00094bd`, `76e837e`), slime, OpenRLHF, AReaL, SWE-bench harness, Defeating Nondeterminism (rows above), used for Ch19 Section 19.8 | Production mapping of each book function: task sources, containerized environments, inference engines, trainers, async modes, metrics, evaluation harness | Taken from the Ch4, Ch13, Ch18, and Ch20 records; not re-read for Ch19 |
| Qwen3-0.6B real-model lab (`tutorial/labs/qwen_grpo_bugfix.py`), run 2026-10-14 for Ch19 | Measured: held-out pass\@1 0.200 -> 0.800 after 30 GRPO steps, entropy 0 by step 15; results in `tutorial/labs/results/` | Book's own measurement; one seed; local HF cache, HF\_HUB\_OFFLINE=1; not a benchmark |
| [Claude Code tools reference](https://code.claude.com/docs/en/tools-reference), checked 2026-10-14 for Ch19 Section 19.7 and `tutorial/labs/qwen_grpo_tools.py` | Tool names and parameters: Read(`file_path`, `offset`, `limit`), Edit(`file_path`, `old_string`, `new_string`, `replace_all`), Write(`file_path`, `content`), Bash(`command`, `timeout`); Edit is exact-match only and needs a unique `old_string` unless `replace_all`; read-before-edit applies to some models only; Bash runs each command in a separate process (default timeout 2 minutes, maximum 10); Grep and Glob absent by default on macOS, Linux, and WSL | Documentation page only, read by the coordinating editor and confirmed against the tool schemas of a live session; parameter names of Grep and Glob not documented there; the lab's tool descriptions, read-before-edit rule, and Bash allowlist are the book's own choices, not Claude Code's implementation |
| Qwen3-0.6B tools lab (`tutorial/labs/qwen_grpo_tools.py`), run 2026-10-14 for Ch19 | Measured: held-out pass\@1 0.100 -> 0.488 after 18 GRPO steps with Read/Edit/Write/Bash tools; tool failures 0.85 -> 0.21 per episode; Bash calls 0.07 -> 0.00; results in `tutorial/labs/results/` | Book's own measurement; one seed; process isolation with resource limits, not a security sandbox; an earlier attempt stopped by out-of-memory at step 2 is logged, not used as a result; not a benchmark |
| [Hairline](https://github.com/lucasmarkes/hairline) (Lucas Marques, MIT) and its [design essay](https://lucasmarkes.com/lab/hairline), checked 2026-10-14 | Visual language for mechanism diagrams: one isometric projection, one stroke weight, rounded solids with a bright outline and dim crease, filled plates, callouts outside the drawing, one accent stroke | README and essay read; no code copied; `scripts/hairline.py` is an independent static-SVG re-implementation without pointer interaction |

## What remains unknown

Private vendor loss details, training mixtures, and final-stage scheduling are not established by these sources. We describe public methods and our own experiments. A claim of adoption needs a specific public report with scope and date, rather than a general statement that “all labs do this.”


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.