Editorial guide

​

Who the reader is

​

What we take from D2L

1. One build. A chapter derives a mechanism, implements it from scratch in a short block, and saves the result with #@save. Later chapters call arl.name(...) instead of repeating the code. By the capstone, the reader has built the whole training stack.
2. Problem first. Each section opens with the problem the previous section left open, then states the mechanism that resolves it.
3. From scratch, then concise. Build the transparent version first. Use the optimized or library version only after the reader can check it against the transparent one.
4. Run everything. Every printed number in the book comes from executing the block above it.
5. Close with summary and exercises. Every chapter ends with ## Summary (3–6 bullet points) and ## Exercises (4–8 items). The exercises include at least one change-an-assumption experiment and one design-review question.

​

Chapter anatomy

# N. Title                           <- H1 is the only H1; numbered
Opening paragraph(s): the question carried over from the previous chapter, why it matters
in practice, and what this chapter builds.

## N.1 Section title                 <- numbered sections, 4-8 per chapter
   motivation -> definition/derivation -> code -> output -> interpretation
   ...
## N.k Design problems in practice   <- REQUIRED. Real decisions, symptoms, fixes.
## Summary
## Exercises

​

The “Design problems in practice” section

Problem: <a short name>. Symptom: what an engineer sees (a metric, a log line, a behavior). Cause: the mechanism, linked to the chapter’s math. Options: two or more fixes, each with its cost. Default: what to try first and why, labeled as the book’s recommendation rather than a universal rule.

​

Callouts

• > **Pitfall.** for a mistake that compiles, runs, and gives the wrong answer.
• > **In production.** for how large systems implement or scale the idea. Cite a source, or state that the claim is a general engineering observation.
• > **Evidence.** for the status of a claim: measured here, reported by a paper, or a hypothesis.

​

Code

• Size. Aim for 5–20 lines per block and no more than 30. When a function is long, split it across blocks: define helpers first and explain each one. Saved blocks may reach 40 lines only when splitting would hide the mechanism.
• One idea per block. Define something, or run something, or show something. A block that defines a function and then demonstrates it is acceptable when both parts are short.
• Visible mechanism. Use explicit tensor operations for the concept being taught (gather, masks, exp of log-ratios). PyTorch built-ins are allowed for everything else (nn.Linear, F.cross_entropy, optimizers).
• Shapes. State tensor shapes in comments using the book’s axis names: B (batch of trajectories), G (group size), T (sequence length), V (vocabulary), D (model width). For example, # logp: [B, T].
• Small, interpretable output. Print a few clearly labeled numbers rather than whole tensors. Round to 3–4 significant figures. Print tensors only when they are small enough to read, at most about 4x8.
• Determinism. Seed everything with torch.manual_seed, or pass a torch.Generator. CPU float reductions depend on the intra-op thread count, so RL trajectories do too. A chapter that trains should call torch.set_num_threads(4) in its first unsaved block (the checker resets the count before each chapter), and its figure module should do the same. Pin dtype where precision matters. Everything runs on CPU. A chapter should finish in under 60 seconds on the check host. Chapters 9 and 19 may take up to 4 minutes.
• Naming. Use descriptive names (old_logp, advantages, action_mask), not single letters, except for standard math (r, eps, gamma, lam).
• No hidden helpers. A block may use only Python, PyTorch, the standard library, earlier blocks in the same chapter, and agentrl functions saved by earlier chapters. The checker enforces this.

​

The #@save mechanism

• A saved block contains only imports, def, class, and constant assignments. It does not print or call anything at top level.
• A saved block prints nothing, so it has no Output: block.
• A later chapter starts with import agentrl as arl (plus import torch and other imports) and calls arl.name.
• Within the chapter that defines a function, call it by its bare name.
• Save only functions the outline lists in that chapter’s API, or that a later chapter will plausibly reuse. A demo-only helper stays unsaved.
• Never change the signature of a function another chapter saved. If a chapter needs to extend one, define a new function and explain why.
• A saved function needs a one-line docstring that states shapes.

​

Output blocks

```python
print(1 + 1)
```

Output:

```text
2
```

​

Mathematics

$$
\nabla_\theta J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}\Big[\sum_t \nabla_\theta \log \pi_\theta(a_t\mid s_t)\, A_t\Big]
$$

​

Figures

1. Data figures are computed from book code. Write a module scripts/figures/chNN_<slug>.py that exposes FIGURES = {"name": fn}. Each function computes its data with torch and agentrl, then draws with scripts/svgplot.py. Render them on the check host with scripts/remote_figures.sh <name> chNN. Examples: the variance of an estimator versus the baseline, clipped-objective landscapes, the bias and variance of KL estimators, training curves, and pass@k curves.
2. Mechanism diagrams use the hairline style in scripts/hairline.py, adapted from Lucas Marques’ Hairline (MIT) and re-implemented independently. Systems, state machines, and loops become isometric scenes: rounded solids drawn as a bright outline with dim inner creases on a dotted floor, placed on a screen-aligned grid so flows read horizontally and vertically, with tracks routed through the visible aisle in front of the solids. Text-heavy diagrams (tables, triage maps, component maps, token strips) use the flat hairline primitives (card, farrow, wrapped_strip). Both use one thin stroke and ink tones. Exactly one stroke family carries the accent color, and it marks the thing the figure is about, such as policy tokens that carry gradient, rewarded transitions, or the path that computes reward. Text never sits on a solid. Numbered callouts with thin leaders sit outside the drawing, short readouts sit in a corner, and each figure has a FIG N.K caption. Ownership in token strips is shown by outline: accent for policy, solid for tool, dotted for prompt, and no box for template tokens. Keep every piece of information the figure needs; the restraint is visual, not editorial.
3. Data plots stay in svgplot (axes, heatmaps, Gantt timelines, value-encoded strips), because isometric projection distorts the quantities they show.
4. Mermaid is for simple flowcharts and sequence diagrams that the site renders natively.

​

Evidence and honesty

• Separate measured here (printed by a block or saved to tutorial/labs/results/), reported by a source (with a link and date), and hypothesis or recommendation.
• Never report an experiment that was not run. Tiny CPU experiments demonstrate mechanisms. They are not benchmark results, so say so where a reader could confuse them.
• Cite primary sources (papers, official repositories at pinned commits, official docs). Record every external source in sources.md with the date inspected and the boundary of what was checked. Vendor-private recipes are unknown unless they are publicly documented. Never write “all labs do X”.
• The running environment executes only trusted, authored code inside the process. State this whenever execution appears, and explain the isolation that real model-generated code requires.

​

Prose style

• Write in English with a direct, technical voice. Use paragraphs for explanation and bullets for enumerations, not as a substitute for explanation.
• Define every term before using it. When a term has competing meanings in the literature (for example, “GRPO”, “on-policy”, “KL penalty”), state which one the book uses.
• After each output, explain what the numbers show and why they had to come out that way.
• Avoid hype words such as “simply”, “powerful”, “magic”, “crucial”, and “note that”.
• Use one running example: the BugFix gym from Chapter 4 (a family of small Python bug-fix tasks with tools read, edit <op>, test, and submit), learned by TinyAgent, the small transformer policy from Chapter 3. The clamp bug from Chapter 1 is task 0 of that family.

​

Verification checklist (before you call a chapter done)

1. Run scripts/remote_check.sh <name> --refresh NN, then run it again without --refresh so the check passes cleanly.
2. Check that every block is at most 30 lines (40 for saved blocks). The checker reports longest_block_lines.
3. Check that every number quoted in prose matches the refreshed output.
4. Check that every figure is rendered, previewed, and referenced from the text.
5. Check that the chapter has Design problems in practice, Summary, and Exercises sections.
6. Check that new sources are added to sources.md.
7. Check that python3 scripts/build_library.py succeeds locally.

​

The Chinese edition