Agentic RL: From Policy Gradients to Coding Agents

中文版 is available: the prose is translated; code, outputs, and figures are identical. This book teaches reinforcement learning for language-model agents. That means models that call tools, read observations, act over many turns, and are trained against verifiers. It starts from the definition of a policy gradient and ends with a complete, tested training stack for a small coding agent. Along the way it covers the design decisions that post-training teams make every day: how to aggregate a loss over variable-length tool trajectories, when a critic is worth its cost, how to keep the inference engine and the trainer consistent, how to detect a reward hack before it reaches an evaluation, and how to read a training dashboard that is going wrong.

How the book works

The book builds one system, one chapter at a time:

The BugFix gym (Chapter 4) is a family of small Python repair tasks. The agent fixes them with the tools read, test, edit, and submit. Tests execute real code, a hidden test set guards against overfitting, and an exploit operation lets reward hacking happen for real.
TinyAgent (Chapter 3) is a small transformer policy that is trained with RL on a CPU in about a minute. It shows the phenomena that large runs show: format learning, entropy collapse, length effects, credit assignment across turns, and reward hacking.
The real-model thread uses the actual Qwen3-0.6B tokenizer and model wherever model-specific behavior matters, such as chat templates, tool-call tokens, retokenization drift, and the log-probability mismatch between the inference engine and the trainer.

Each mechanism is first derived, then implemented from scratch in a short code block, then run. Its printed output appears directly below the code. In the style of D2L, a function built in one chapter is marked #@save and collected into the agentrl package, so later chapters call arl.ppo_policy_loss(...) instead of repeating it. By the capstone, the reader has written every piece of the stack. Every chapter ends with Design problems in practice. This section takes problems engineers actually hit, and for each one it gives the symptom, the cause, the options, and a recommended default. Every chapter then gives a summary and exercises. Appendix B collects open-ended design-review problems, with model answers.

Part	Chapter	The question it answers
	0. Notation and how to use this book	How do I read and run this book?
I. Foundations	1. What is agentic RL?	What changes when a model acts in an environment?
	2. Reinforcement learning fundamentals	What are policies, returns, and values, and how is an LLM agent an MDP?
	3. Language models as policies	What exactly is the probability of an action?
	4. Environments and tools: the BugFix gym	What does the agent act on, and what counts as success?
	5. Trajectories, templates, and masks	How does a multi-turn episode become training tensors?
II. Policy optimization	6. Policy gradients	How does a reward become a gradient?
	7. Advantages and credit assignment	Which actions deserve the credit?
	8. PPO: reusing samples safely	How can expensive rollouts be reused for several updates?
	9. GRPO and the critic-free family	What do GRPO, Dr. GRPO, DAPO, RLOO, and GSPO each change?
	10. KL, entropy, and regularization	How do we keep the policy from drifting or collapsing?
III. Rewards and agents	11. Rewards, verifiers, and reward hacking	What should be rewarded, and how will the agent cheat?
	12. Multi-turn agents and long horizons	What breaks when episodes get long?
IV. Systems	13. Rollout systems and off-policy correction	How do inference engines, trainers, and environments fit together?
	14. Scaling and efficiency	Where do the tokens, memory, and GPU-hours go?
V. Beyond pure RL	15. SFT, rejection sampling, and preferences	When are demonstrations or comparisons the better signal?
	16. Distillation for agents	How can a teacher score the student’s own trajectories?
VI. Practice	17. Evaluation	Did the agent actually improve?
	18. Debugging RL runs	What do common failures look like on a dashboard?
	19. Capstone: train a coding agent end to end	Can we assemble and evaluate the whole stack?
	20. Research frontier and open problems	What is unsettled, and how would we test it?
Appendices	A. Mathematical tools	Which identities and estimators do the chapters rely on?
	B. Design review problems	Can I reason through a real post-training design question?

Running the code

Every Python block runs. The checker executes the chapters in order, rebuilds agentrl from the #@save blocks of earlier chapters, and compares each block’s stdout with the recorded output:

scripts/remote_check.sh <workdir> [chapter-prefix ...]   # runs on the GPU host; CPU is enough
python3 scripts/build_library.py                         # regenerate agentrl/ from the chapters
scripts/remote_figures.sh <workdir> [chNN ...]           # regenerate data figures
python3 scripts/build_tutorial_site.py                   # regenerate the Mintlify edition in website/

The real-model lab trains Qwen3-0.6B on the BugFix gym on one H200, and Chapter 19 reports its measured results. The editorial guide defines the book’s conventions, and the source ledger records every external source with the date it was inspected and what was checked.

What the experiments do and do not show

TinyAgent experiments are real and run on a CPU, but they are small. They demonstrate mechanisms and make failure modes reproducible. They are not benchmark results. Results reported by papers are labeled as reported, with dates. Recommendations are labeled as the book’s recommendations. The environment executes only trusted code written for the book; Chapter 4 explains the isolation that real model-generated code requires.

Start here

Front matter

Part I · Foundations

Part II · Policy optimization

Part III · Rewards and agents

Part IV · Systems

Part V · Beyond pure RL

Part VI · Practice

Appendices

Reference

Agentic RL: From Policy Gradients to Coding Agents

How the book works

Contents

Running the code

What the experiments do and do not show

​How the book works

​Contents

​Running the code

​What the experiments do and do not show

How the book works

Contents

Running the code

What the experiments do and do not show