Sources and implementation ledger - Agentic RL

Reviewed: 2026-10-13. This ledger identifies what was actually inspected. It is not a comprehensive bibliography.

Source	Evidence used	Boundary
D2L and softmax source	Teaching structure and indexing explanation	Reference, no copied prose/code
DeepSWE release report	Coding-agent system motivation	Public report; no benchmark reproduction
Pinned rLLM SWE example	Tool/environment integration reference; README inspected	SHA resolved 2026-10-13; no installation or training run
R2E-Gym	Public repository reference	HEAD resolved to `353348a0ff690f2592025eff41b3fef4201a4d8b`; code not audited
PyTorch gather	Tensor indexing semantics	Educational implementation uses installed PyTorch 2.9.1
PPO, GAE	Sampled-action surrogate and value-based estimator	Tiny implementations, no full critic trainer
DeepSeekMath	GRPO origin	Does not specify all present-day variants
DAPO, Dr. GRPO	Public design alternatives	Not reproduced at benchmark scale
DPO	Preference objective	Artificial sequence likelihood pairs
On-policy distillation	Student-prefix teacher supervision	Does not imply one mandatory divergence
Tail-corrected top-k	Fixed-prefix gradient proposal; full text inspected	September 2026 preprint, not reproduced
SparseOPD	Vocabulary selection research direction	Abstract inspected; no implementation audit
DIAL-OPD	Timestep supervision allocation; full text inspected	October 2026 preprint, author-reported results
Control-variate OPD	Variance-reduction proposal	Abstract inspected; no implementation audit
Entropy-Aware OPD	Divergence selection research direction	Abstract inspected; no implementation audit
verl torch kernels	Selected log-probability paths inspected locally	Pinned SHA; hardware branches differ
verl core algorithms	Group IDs, standard deviation, response-shaped advantages	Actual helper differs from our explicit population-std teaching option
Mintlify settings, CLI	docs.json and MDX reading interface	Locally installed CLI may predate newest commands
rLLM	Ch20: framework behind DeepSWE; README inspected 2026-10-13	HEAD `4e828c81680e5a85b638ceedeaf09d1e90109ca8` (old `agentica-project/rllm` archived at `7b47687`); code not audited
R2E-Gym paper	Ch20: environment curation, SFT training, hybrid verifiers; abstract and full text inspected 2026-10-13	Author-reported 34.4% pass@1 and 51.0% Best@26; repo HEAD `0d94c4eb9431cd195c55a7ea3abd54006c9a1735`
SWE-Gym	Ch20: 2,438 tasks, RFT and verifier results; abstract and full text inspected 2026-10-13	Not RL; author-reported numbers
SWE-RL	Ch20: single-turn GRPO with patch-similarity reward; abstract and full text inspected 2026-10-13	41.0% uses 500 samples plus reranking
SkyRL v0 README, SkyRL-Agent	Ch20: multi-turn RL results and recipe; README at v0 commit and paper full text inspected 2026-10-13	SkyRL-v0 Notion blog did not render; repo HEAD `a1e34dbf21003f66bd7921487fd47db6c154aba2`
Golubev et al. 2508.03501	Ch20: RFT then DAPO for SWE agents	Abstract only, 2026-10-13
Kimi-Dev	Ch20: workflow RL, test-writer reward, selection vs pass@1	Abstract and full text, 2026-10-13; 60.4% uses test-time selection
LEGO-RL	Ch20: harness-native GSPO for coding agents	Abstract only, 2026-10-13
SWE-smith, SWE-rebench, SWE-rebench V2, Skywork-SWE, SWE-Factory	Ch20: task mining and synthesis	Abstracts only, 2026-10-13
Absolute Zero, Self-play SWE-RL	Ch20: self-play rewards (learnability; injector/solver)	Abstract and full text, 2026-10-13; SSR absolute scores not extracted
GSPO, MiniMax-M1 / CISPO, RLOO	Ch20: objective definitions, clip ranges, MoE and FP32-head claims	Full text, 2026-10-13; not reproduced
VAPO, DeepSeek-R1	Ch20: value-based variant; R1 context	Abstracts only, 2026-10-13
Entropy Mechanism, 80/20 high-entropy tokens, Reasoning with Exploration, ARPO	Ch20: entropy dynamics and exploration	Entropy Mechanism full text; others abstract only, 2026-10-13
Yue et al., ProRL, Invisible Leash, CoT-Pass@K, Pass@k Training, RL Squeezes, Spurious Rewards, One-shot RLVR	Ch20: pass@k capability debate	Yue and ProRL full text; others abstract only, 2026-10-13; all math-domain
ArCHer, RAGEN, GiGPO, Wei et al. 2505.11821, SWEET-RL, TRACE, SWE-MILE	Ch20: multi-turn credit assignment	RAGEN and GiGPO full text; others abstract only, 2026-10-13
MEM1, SUPO, Context-Folding, ReSum, FoldAct, CAMG-RL	Ch20: context management in agent RL	Abstracts only, 2026-10-13; table numbers not used
Let’s Verify Step by Step, Math-Shepherd, PRIME, SWE-RM, Agent-as-a-Judge, Rubrics as Rewards, One Token to Fool	Ch20: process rewards, verifiers, judges	Abstracts only, 2026-10-13
AReaL, PipelineRL, Magistral, Kimi k1.5, Ring-1T / IcePop	Ch20: async designs, staleness, partial rollouts, mismatch masking	Full text, 2026-10-13; AReaL speedup stated as 2.77x (abstract) and 2.57x (intro)
LlamaRL, ROLL Flash, Trust Region Masking, HybridFlow, OpenRLHF paper	Ch20: async systems and sequence-level masking	Abstracts only, 2026-10-13
TIS blog, When Speed Kills Stability, Defeating Nondeterminism	Ch20: training-inference mismatch and corrections	TIS and nondeterminism posts full text; Speed Kills partially read, 2026-10-13
verl docs (one_step_off, fully_async, rollout_corr)	Ch20: async modes and rollout-correction presets	Docs at `76e837e` read 2026-10-13; code not audited
slime, OpenRLHF	Ch20: framework architecture and async options	READMEs at pinned SHAs, 2026-10-13
MiniLLM, Qwen3 report, OPD blog, TCOD, ReOPD	Ch20: on-policy distillation evidence	Qwen3 Table 21 and OPD blog full text; others abstract only, 2026-10-13
SWE-bench, SWE-bench Verified card, Multimodal, Multi-SWE-bench, SWE-bench-Live, Terminal-Bench, LiveCodeBench	Ch20: benchmark definitions	Abstracts and dataset card, 2026-10-13; OpenAI Verified announcement returned 403 and is not cited
SWE-Bench Pro, SWE-Bench Pro Verified	Ch20: contamination-resistant design; git-object leak	Full text (Pro), partial full text (Pro Verified), 2026-10-13
SWE-Bench+, SWE-Bench Illusion, UTBoost, SWE-bench issue #465, METR note	Ch20: benchmark validity problems	Abstracts; issue page and METR note read in full, 2026-10-13
DeepSWE release report (2025-07-02), re-checked 2026-10-13 for Chapter 1	Base model Qwen3-32B; rLLM; 4.5K R2E-Gym problems with SWE-bench Verified repositories filtered out; GRPO++ modifications (clip-higher, no KL loss, no reward std normalization, length normalization, leave-one-out, compact filtering, no entropy loss); reported 42.2% pass@1 (avg of 16), 59% hybrid test-time scaling, 64 H100 GPUs for six days	Blog report inspected, not reproduced; the report gives both 59% and 59.2% for the hybrid result
SWE-bench (Jimenez et al., v1 2023-10-10), checked 2026-10-13	2,294 task instances from real GitHub issues and pull requests in 12 Python repositories; best evaluated model in the paper resolved 1.96%	Abstract page inspected; harness and SWE-bench Verified construction not inspected
SWE-RL (Wei et al., v1 2025-02-25), checked 2026-10-13	RL on Llama 3 with software-evolution data; rule-based reward from similarity to the ground-truth solution; reported 41.0% on SWE-bench Verified (70B)	Abstract page inspected; exact similarity function not checked
R2E-Gym (Jain et al., v1 2025-04-09), checked 2026-10-13	Procedural environments from commits with test generation and back-translation; more than 8.7K tasks; reported 34.4% pass@1 (32B) and 51% with hybrid execution-based and execution-free verifiers on SWE-bench Verified	Abstract page inspected; code not audited
Approximating KL Divergence (Schulman, posted 2020-03-07), checked 2026-10-13	Definitions of the k1, k2, k3 estimators with r = p(x)/q(x), x ~ q; k1 and k3 unbiased, k2 biased with low variance	Blog post inspected; used in Appendix A and Chapter 10
Qwen3-0.6B model card	Recommended sampling (thinking: T=0.6, top-p 0.95, top-k 20; non-thinking: T=0.7, top-p 0.8, top-k 20; no greedy); history should keep only final output, not thinking (Ch. 3, 5)	Inspected 2026-10-13; tokenizer, template, and weights used from the check host’s local HF cache (no `generation_config.json` cached)
rLLM `BaseEnv` at pinned SHA	`reset() -> tuple[dict, dict]`, `step(action) -> tuple[Any, float, bool, dict]`, `from_dict` (Ch. 4)	Inspected 2026-10-13; signatures only, not run
R2E-Gym README at pinned SHA	8.1K+ problems across 13 repos, `RepoEnv`, `apply_patch`, Docker images of ~300-500 MB per instance (Ch. 4)	Inspected 2026-10-13; README only, code not audited
SWE-bench dataset card	Field definitions of `FAIL_TO_PASS`, `PASS_TO_PASS`, `test_patch`, `base_commit` (Ch. 4)	Inspected 2026-10-13
SWE-bench evaluation guide	Patches applied and tests run in Docker; image cache levels `base`/`env`/`instance` (Ch. 4)	Inspected 2026-10-13; harness not run
MCP tools specification 2025-06-18	Tool definition (`name`, `description`, `inputSchema`), `isError` results, server input-validation requirements (Ch. 4)	Inspected 2026-10-13
Williams (1992), “Simple statistical gradient-following algorithms for connectionist reinforcement learning”, Machine Learning 8	Origin of the REINFORCE name and estimator (Ch. 6)	Bibliographic citation only, 2026-10-13; derivation in Ch. 6 is self-contained
GAE (Schulman et al., v1 2015-06-08), checked 2026-10-13	Exponentially weighted advantage estimator analogous to TD(λ); value functions reduce variance “at the cost of some bias” (Ch. 7)	Abstract page inspected; Ch. 7 derivation and λ measurements are the book’s own
DeepSeekMath (Shao et al., v1 2024-02-05), checked 2026-10-13	GRPO introduced as a PPO variant (Ch. 7 attribution of the group-mean baseline)	Abstract page inspected; the abstract does not state the advantage formula, which Ch. 7 states as the common implementation
Back to Basics / RLOO (Ahmadian et al., v1 2024-02-22), checked 2026-10-13	REINFORCE-style optimization outperforming PPO in RLHF; leave-one-out baseline for LLMs (Ch. 7)	Abstract page inspected; the leave-one-out identity in Ch. 7 is derived, not quoted
Kool, van Hoof, Welling (2019), “Buy 4 REINFORCE Samples, Get a Baseline for Free!” (ICLR 2019 workshop)	Leave-one-out baseline over multiple samples (Ch. 7)	Bibliographic citation only, 2026-10-13
Dr. GRPO (Liu et al., v1 2025-03-26), checked 2026-10-13	Sec. 3.1: response-length bias from 1/\|o_i\| and question-level difficulty bias from dividing by std; Sec. 3.2: remove both, use a constant normalizer such as the generation budget (Ch. 6 aggregation, Ch. 7 std normalization)	HTML full text sections 3.1-3.2 read; experiments not reproduced; TinyAgent numbers in Ch. 6-7 are the book’s own
STaR (Zelikman et al., v1 2022-03-28), checked 2026-10-13	Ch15: generate rationales, keep those leading to correct answers, fine-tune, repeat; rationalization given the correct answer	Abstract page inspected; not reproduced
RFT for math reasoning (Yuan et al., v1 2023-08-03), checked 2026-10-13	Ch15: rejection-sampling fine-tuning; more distinct reasoning paths give larger gains; helps weaker models more	Abstract page inspected; numbers not used
DPO (Rafailov et al., v1 2023-05-29), checked 2026-10-13	Ch15: closed-form reparameterization of the KL-regularized objective into a classification loss without sampling	Abstract page inspected; derivation re-done in the chapter
Iterative RPO (Pang et al., v1 2024-04-30), checked 2026-10-13	Ch15: DPO loss with an added NLL term, which the abstract calls “crucial”	Abstract page inspected; not reproduced
DAgger / no-regret imitation (Ross, Gordon, Bagnell, v1 2010-11-02), checked 2026-10-13	Ch16: sequential prediction violates i.i.d. assumptions; iterative learner-state data collection	Abstract page inspected; the $\epsilon H^2$ vs $\epsilon H$ bound is cited from the paper’s known analysis, not re-derived from the full text
GKD (Agarwal et al., v1 2023-06-23), checked 2026-10-13 for Ch16	Train/inference mismatch of KD; student trained on self-generated outputs with teacher feedback; alternative divergences; combinable with RL fine-tuning	Abstract page inspected
MiniLLM (Gu et al., v1 2023-06-14), checked 2026-10-13 for Ch16	Reverse KL instead of forward KL, optimized on-policy; reported lower exposure bias, 120M-13B	Abstract page inspected
Qwen3 Technical Report (v1 2025-05-14), checked 2026-10-13 for Ch16	Sec. 4.5 strong-to-weak distillation: off-policy then on-policy, logits aligned by minimizing KL; Table 21 (Qwen3-8B, from the same off-policy-distilled checkpoint at AIME’24 55.0): RL 67.6 at 17,920 GPU hours, on-policy distillation 74.4 at 1,800 GPU hours	HTML full text, Section 4.5 and Table 21 only
On-Policy Distillation (Kevin Lu, Thinking Machines Lab, 2025-10-27), checked 2026-10-13 for Ch16	Per-token advantage = negative reverse KL on sampled tokens; discount factor zero; no top-k logit distillation; Qwen3-8B-Base after SFT (60% AIME’24) to 70% in about 150 steps; 9-30x compute reduction vs extrapolated SFT depending on accounting	Full post; a June 2026 update notes the cookbook models changed
PPO (Schulman et al., v1 2017-07-20), checked 2026-10-13 for Ch8	Clipped surrogate objective; multiple epochs of minibatch updates per batch	Abstract page inspected; Ch8 derivations and TinyAgent measurements are the book’s own
TRPO (Schulman et al., v1 2015-02-19), checked 2026-10-13 for Ch8	Surrogate lower bound with a KL penalty; KL-constrained natural-gradient step with line search	Bibliographic and method summary only; bound stated in Ch8 in qualitative form
Kakade and Langford (2002), “Approximately Optimal Approximate Reinforcement Learning”, ICML	Performance-difference identity used for the per-token surrogate (Ch8)	Bibliographic citation only, 2026-10-13; derivation in Ch8 is self-contained
Dual-clip PPO (Ye et al., v1 2019-12-20), checked 2026-10-13 for Ch8	Abstract lists “dual-clip PPO” among the method’s strategies	Abstract page inspected; the c = 3 value in Ch8 is a common choice, not checked against the paper
What Matters In On-Policy RL? (Andrychowicz et al., v1 2020-06-10), checked 2026-10-13 for Ch8	Recommendation “Use GAE with λ=0.9 but neither Huber loss nor PPO-style value loss clipping”; “PPO-style value loss clipping hurts the performance regardless of the clipping threshold”	HTML full text, first 100k characters; continuous control, not language models
DeepSeekMath (Shao et al.), re-checked 2026-10-13 for Ch9	KL coefficient 0.04; 64 outputs per question; policy lr 1e-6; “a single update following each exploration stage”; k3-style KL estimator added to the loss, not the reward	HTML full text, first 100k characters (the page renders the group size as “6464”)
SWE-Bench+ (Aleithan et al., v1 2024-10-09), re-checked 2026-10-13 for Ch9	32.67% of successful patches had the solution “directly provided in the issue report”; 31.08% suspicious due to weak tests	Abstract page inspected
SWE-Bench Illusion (Liang et al., v1 2025-06-14), re-checked 2026-10-13 for Ch9	Buggy file paths identified from issue text alone with “up to 76% accuracy”, “up to 53%” on repositories outside SWE-Bench	Abstract page inspected
DAPO, Dr. GRPO, GSPO, MiniMax-M1/CISPO, RLOO (rows above), used for Ch9	Variant definitions in Ch9’s diff table are taken from the Ch20 summaries of these sources	Not re-read for Ch9; TinyAgent presets are not reproductions of the papers’ settings
verl at `00094bd` (project-pinned commit in `upstream.lock.json`), inspected 2026-10-13 for Ch10	`core_algos.kl_penalty`: k1 (`kl`), `abs`, k2 (`mse`), k3 (`low_var_kl`, log-ratio clamped to ±20 and value to ±10); a `+` suffix applies a straight-through k2 gradient, with the comment that k1 and k3 have the right expected value but not the right expected gradient. `actor.yaml`: `use_kl_loss: false`, `kl_loss_coef: 0.001`, `kl_loss_type: low_var_kl`, `entropy_coeff: 0`. `ray_trainer.apply_kl_penalty`: in-reward KL `token_level_scores - beta * kld` with an adaptive or fixed controller (`algorithm.kl_penalty: kl`). `workers/utils/losses.py`: `policy_loss -= entropy_coeff * entropy_loss`; `policy_loss += kl_loss * kl_loss_coef`	Source read on the vp host’s `.upstream/verl` checkout; not run
DAPO (v1 2025-03-18), re-checked 2026-10-13 for Ch10	Sec. 2.3 removes the KL term: the long-CoT “model distribution can diverge significantly from the initial model, thus this restriction is not necessary”; Sec. 3.1 entropy “decreases quickly”; Clip-Higher because “the probability of an unlikely ‘exploration’ token is too tightly bounded to be uplifted”, epsilon_low = 0.2, epsilon_high = 0.28	HTML full text, the quoted sections
ProRL (v1 2025-05-30), re-checked 2026-10-13 for Ch10	Sec. 2.3.1: a KL penalty “provides a stronger and more stable solution” and “helps maintain entropy”; “the KL term may increasingly dominate the loss”; periodic hard reset of the reference to a recent policy snapshot with optimizer reinitialization, triggered when validation stagnates or degrades; Sec. 2.2.1: with raised temperature “entropy continues to decline steadily”	HTML full text, the quoted sections; the KL coefficient value was not found in the text read
The Entropy Mechanism of RL for Reasoning LMs (Cui et al., v1 2025-05-28), used for Ch10	Entropy change as a covariance between log-probability and the logit update (Ch10 re-derives the first-order identity itself)	See the Ch20 row above; not re-read for Ch10
Ng, Harada, Russell (1999), “Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping”, ICML, checked 2026-10-13 for Ch11	Potential-based shaping $\gamma\Phi(s') - \Phi(s)$ preserves optimal policies, and is the only form that does without further assumptions	Bibliographic metadata via Semantic Scholar; the PDF text was not machine-readable. The sufficiency proof in Ch11 is re-derived and checked numerically, not quoted
Christiano et al. (2017), “Deep reinforcement learning from human preferences” (v1 2017-06-12), checked 2026-10-13 for Ch11	Reward models fit to pairwise human comparisons as the RL reward	Abstract page inspected; the Bradley–Terry loss in Ch11 is the standard form, written out by the book
Bradley and Terry (1952), “Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons”, Biometrika 39	Origin of the paired-comparison model used for reward models (Ch11)	Bibliographic citation only, 2026-10-13
Gao, Schulman, Hilton (2022), “Scaling Laws for Reward Model Overoptimization” (v1 2022-10-19), checked 2026-10-13 for Ch11	A gold reward first improves and then degrades as a proxy reward model is optimized; functional forms differ for RL and best-of-n and scale with RM size	Abstract page inspected; no numbers used
Baker et al. (2025), “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation” (v1 2025-03-14), checked 2026-10-13 for Ch11	o3-mini monitored for reward hacking in agentic coding environments; a CoT monitor in the training reward helps “at low optimization levels”; with more optimization, “obfuscated reward hacking, hiding their intent within the CoT”	Abstract page inspected; the abstract does not name specific exploits such as test special-casing
Let’s Verify Step by Step, Math-Shepherd, One Token to Fool LLM-as-a-Judge, Rubrics as Rewards (rows above), used for Ch11	Process reward models; judge “master key” false positives; rubric rewards	Taken from the Ch20 summaries of these sources; not re-read for Ch11
Pardo, Tavakoli, Levdik, Kormushev (2017), “Time Limits in Reinforcement Learning” (v1 2017-12-01), checked 2026-10-13 for Ch12	If the time limit is part of the task, include remaining time in the input; if it only facilitates learning, bootstrap from the value at the end of a partial episode	Abstract page inspected
Lindenbauer et al. (2025), “The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management” (v1 2025-08-29), checked 2026-10-13 for Ch12	In SWE-agent on SWE-bench Verified, observation masking halves cost relative to the raw agent and matches or slightly exceeds LLM summarization’s solve rate (five model configurations)	Abstract page (v3) inspected; solve-rate figures not extracted
MCP tools specification, DAPO overlong filtering, DeepSWE compact filtering (rows above), used for Ch12	Tool errors returned as `isError` results; masking truncated episodes from the loss	Taken from the Ch4, Ch9, and Ch20 records of these sources
Chen et al. (2021), “Evaluating Large Language Models Trained on Code” (v1 2021-07-07), checked 2026-10-13 for Ch17	Sec. 2.1: unbiased pass@k estimator $1 - \binom{n-c}{k}/\binom{n}{k}$ with $n \ge k$ (they use $n = 200$ , $k \le 100$ ); direct computation is numerically unstable, so they give a product-form numpy implementation; the plug-in $1-(1-\hat p)^k$ is biased (underestimate, Appendix A); abstract reports 28.8% pass@1 on HumanEval and 70.2% with 100 samples per problem	Abstract page and ar5iv full text (first 100k characters) inspected; the unbiasedness argument and product form in Ch17 are re-derived
Miller (2024), “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations” (v1 2024-11-01), checked 2026-10-13 for Ch17	Evals as experiments; questions treated as samples from a larger unseen population; formulas for analyzing evals, comparing two models, and planning sample sizes	Abstract page inspected; Ch17’s clustered-variance, paired, and power formulas are derived in the chapter, not quoted
Wilson (1927), “Probable inference, the law of succession, and statistical inference”, JASA 22; Efron (1979), “Bootstrap methods: another look at the jackknife”, Annals of Statistics 7	Origins of the score interval and the bootstrap (Ch17)	Bibliographic citations only, 2026-10-13; Ch17 derives and checks both numerically
Yue et al., ProRL, Invisible Leash (rows above), used for Ch17	The pass@k narrowing debate that Ch17 measures on TinyAgent	Taken from the Ch20 summaries; not re-read for Ch17
Ring-1T / IcePop (v1 2025-10-21), re-checked 2026-10-13 for Ch13	Masking function $M(k) = k$ for $k \in [\alpha, \beta]$ and 0 otherwise, multiplying the clipped surrogate; recipe uses $\alpha = 0.5$ , $\beta = 5$ ; a narrow band [0.5, 2.0] “immediately destabilizes training”; mismatch grows without mitigation	HTML full text, first 100k characters; `mis_weights` in Ch13 implements this weight
TIS blog, AReaL, PipelineRL, Kimi k1.5, When Speed Kills Stability, Defeating Nondeterminism, MiniMax-M1, verl docs, slime, OpenRLHF (rows above), used for Ch13	Async designs (staleness bound, interruptible/partial rollouts), decoupled objective, TIS cap 2, low-probability concentration of mismatch, batch invariance, FP32 LM head	Taken from the Ch20 summaries; not re-read for Ch13; the simulator and TinyAgent/Qwen3 measurements are the book’s own
NVIDIA H200 product page, checked 2026-10-13 for Ch14	H200 SXM: 141 GB, 4.8 TB/s memory bandwidth, BF16 Tensor Core 1,979 TFLOPS “with sparsity” (989 dense used in Ch14)	Vendor specification; achieved bandwidth and FLOPs are lower
Qwen3-8B and Qwen3-0.6B `config.json` in the check host’s Hugging Face cache, read 2026-10-13 for Ch13-14	Qwen3-8B: 36 layers, 8 KV heads, head dim 128, hidden 4096, vocab 151,936 (KV bytes per token in Ch14); Qwen3-0.6B: tied embeddings, hidden 1024	Local files; parameter counts 8.2B and 32.8B in Ch13 are rounded model-card figures, not re-checked
verl at `00094bd`, inspected 2026-10-13 for Ch18	`trainer/ppo/metric_utils.compute_data_metrics`: `critic/score`, `critic/rewards`, `critic/advantages`, `critic/returns` (mean/max/min), `critic/vf_explained_var` when a critic is used, `response_length/mean`, `max`, `min`, `clip_ratio`; `core_algos` policy losses return `pg_clipfrac`, `pg_clipfrac_lower`, and `ppo_kl` (masked mean of the negative log-ratio, k1); `ray_trainer` logs `actor/entropy` and, when `rollout_log_probs` is in the batch, `utils/debug/metrics.calculate_debug_metrics`: `training/rollout_probs_diff_{mean,max,std}` (absolute differences of token probabilities, despite the helper’s name `calculate_log_prob_diff`) and a rollout–actor Pearson correlation on the response mask	Source read on the vp host’s `.upstream/verl` checkout; not run; default enablement of each metric not audited
Defeating Nondeterminism, Ch13 Section 13.6, Ch11 hacking run (rows and chapters above), used for Ch18	Batch invariance as the numerical source of irreproducibility; bf16 batch-composition log-prob differences; the public-test hacking setup	Taken from the earlier records; Ch18’s ten failure runs and invariant matrix are the book’s own TinyAgent measurements
vLLM / PagedAttention (Kwon et al., v1 2023-09-12), checked 2026-10-13 for Ch19	KV-cache memory managed with paging, “near-zero waste in KV cache memory”, reported 2-4x throughput over FasterTransformer and Orca	Abstract page only; not run
SGLang (Zheng et al., v1 2023-12-12), checked 2026-10-13 for Ch19	Runtime with RadixAttention for KV-cache reuse; reported up to 6.4x throughput on agent, reasoning, JSON, and multi-turn workloads	Abstract page only; not run
OpenRLHF paper (v1 2024-05-20), re-checked 2026-10-13 for Ch19	Abstract: “built upon Ray, vLLM, DeepSpeed, and HuggingFace Transformers”	Abstract only
HybridFlow (verl; v1 2024-09-28), re-checked 2026-10-13 for Ch19	Hybrid single-/multi-controller programming model for RLHF dataflows; reported 1.53x-20.57x throughput over baselines	Abstract only
DeepSWE, SkyRL-Agent, R2E-Gym, SWE-Gym, SWE-smith, SWE-rebench, rLLM, verl (`00094bd`, `76e837e`), slime, OpenRLHF, AReaL, SWE-bench harness, Defeating Nondeterminism (rows above), used for Ch19 Section 19.8	Production mapping of each book function: task sources, containerized environments, inference engines, trainers, async modes, metrics, evaluation harness	Taken from the Ch4, Ch13, Ch18, and Ch20 records; not re-read for Ch19
Qwen3-0.6B real-model lab (`tutorial/labs/qwen_grpo_bugfix.py`), run 2026-10-14 for Ch19	Measured: held-out pass@1 0.200 -> 0.800 after 30 GRPO steps, entropy 0 by step 15; results in `tutorial/labs/results/`	Book’s own measurement; one seed; local HF cache, HF_HUB_OFFLINE=1; not a benchmark
Claude Code tools reference, checked 2026-10-14 for Ch19 Section 19.7 and `tutorial/labs/qwen_grpo_tools.py`	Tool names and parameters: Read(`file_path`, `offset`, `limit`), Edit(`file_path`, `old_string`, `new_string`, `replace_all`), Write(`file_path`, `content`), Bash(`command`, `timeout`); Edit is exact-match only and needs a unique `old_string` unless `replace_all`; read-before-edit applies to some models only; Bash runs each command in a separate process (default timeout 2 minutes, maximum 10); Grep and Glob absent by default on macOS, Linux, and WSL	Documentation page only, read by the coordinating editor and confirmed against the tool schemas of a live session; parameter names of Grep and Glob not documented there; the lab’s tool descriptions, read-before-edit rule, and Bash allowlist are the book’s own choices, not Claude Code’s implementation
Qwen3-0.6B tools lab (`tutorial/labs/qwen_grpo_tools.py`), run 2026-10-14 for Ch19	Measured: held-out pass@1 0.100 -> 0.488 after 18 GRPO steps with Read/Edit/Write/Bash tools; tool failures 0.85 -> 0.21 per episode; Bash calls 0.07 -> 0.00; results in `tutorial/labs/results/`	Book’s own measurement; one seed; process isolation with resource limits, not a security sandbox; an earlier attempt stopped by out-of-memory at step 2 is logged, not used as a result; not a benchmark
Hairline (Lucas Marques, MIT) and its design essay, checked 2026-10-14	Visual language for mechanism diagrams: one isometric projection, one stroke weight, rounded solids with a bright outline and dim crease, filled plates, callouts outside the drawing, one accent stroke	README and essay read; no code copied; `scripts/hairline.py` is an independent static-SVG re-implementation without pointer interaction

What remains unknown

Private vendor loss details, training mixtures, and final-stage scheduling are not established by these sources. We describe public methods and our own experiments. A claim of adoption needs a specific public report with scope and date, rather than a general statement that “all labs do this.”