Skip to main content
Every training loop so far ran in one process. train_rl sampled 64 episodes with the current weights, waited for the last one to finish, took its gradient steps, and sampled again with the new weights. Sampling and training never overlapped, the sampler and the trainer were the same Python object, and sample_logp agreed with the trainer’s recomputation to 2×10−62 \times 10^{-6} (Section 8.6). None of that survives contact with a real coding agent. An episode takes minutes of container time, episode durations have a long tail, generation runs on an inference engine with its own kernels, and the trainer runs on a different set of accelerators with a different parallelism layout. The system that connects them decides how much of the hardware does useful work, and it also decides which policy actually generated the data. The second consequence is the subject of this chapter as much as the first. We build four things. A discrete-event simulator of a rollout pipeline measures what synchronous, one-step-ahead, and fully asynchronous designs buy in throughput and what they cost in staleness. A training loop that samples with a policy kk updates old measures what staleness does to TinyAgent, with and without importance correction. A measurement on the real Qwen3-0.6B shows that an inference-style forward pass and a trainer-style forward pass disagree about log-probabilities even with identical weights. And two corrections for that disagreement, truncated and masked importance sampling, are derived, simulated, and tested on TinyAgent with a deliberately mismatched sampler.

​
13.1 The components of an RL system

A production RL system for agents has four kinds of component. Their names vary between frameworks; their jobs do not.
  • The inference engine (vLLM, SGLang) generates tokens. It owns the KV cache, batches thousands of concurrent sequences with continuous batching, and returns token ids together with the log-probability of each sampled token under the distribution it sampled from. Its kernels are tuned for decode throughput: paged attention, fused sampling, often bf16 or FP8 weights. Failure modes: out-of-memory from KV-cache growth on long episodes, preemption and recomputation of sequences under memory pressure, and log-probabilities that differ from the trainer’s (Section 13.6).
  • The trainer (FSDP or Megatron, inside frameworks such as verl, slime, OpenRLHF, or AReaL) recomputes log-probabilities, computes advantages, and takes gradient steps. It shards parameters, gradients, and optimizer states across accelerators. Failure modes: memory blow-up on long sequences (Chapter 14), gradient-normalization bugs across microbatches, and silent divergence when its data came from a policy other than the one it assumes.
  • Environment workers run the tools: a container per episode with the repository checked out, a shell, a test runner. They are CPU-bound, slow, and unreliable. Failure modes: container start-up latency, hung test processes, flaky tests, worker crashes in the middle of an episode, and resource exhaustion when thousands of episodes run at once.
  • Weight synchronization moves new parameters from the trainer to the engine after each update, by an NCCL broadcast between the two process groups, by resharding in place when they share accelerators (“colocated”), or by writing a checkpoint that the engine reloads. Failure modes: the sync takes seconds to minutes and stalls generation; a partial sync leaves the engine with a mixture of old and new layers; a version counter that is not attached to every sample makes staleness unmeasurable.
A controller ties them together: it tracks the policy version, decides when to launch episodes and when to train, and holds finished episodes in a buffer. The framework documentation cited in Chapter 20 describes these pieces for verl, slime, and OpenRLHF; we read their documentation and READMEs, not their code. In this book’s code, the four components are already present in miniature: rollout_batch is the engine, BugFixEnv is an environment worker, rl_update is the trainer, and weight synchronization is free because sampler and trainer are the same nn.Module. What a real sync moves is the parameters. The block counts them for TinyAgent and does the arithmetic for real model sizes, under labeled assumptions about bandwidth.
Output:
The weights of a 32B model are 66 GB in bf16. Within a node that is under a second; across nodes it is several seconds; through a checkpoint reload it is about a minute, every step. Those numbers, not the model’s FLOPs, are why colocated designs reshard in place and why disaggregated designs broadcast over NCCL. They are also inputs to the simulator: the sync cost is part of every step’s critical path.

​
13.2 Where the time goes: long-tailed episodes

The second input is the duration of an episode. An agent episode alternates generation and tool calls. Each turn generates a few hundred tokens at a per-sequence decode rate of tens of tokens per second (the engine’s aggregate throughput is high, but each sequence advances one token per decode step), then waits for a tool: a file read returns in milliseconds, a test suite in seconds to minutes. The number of turns varies by task and by attempt. agent_episode_durations samples durations from that structure: a geometric number of turns, a uniform number of tokens per turn, and a log-normal tool latency. The defaults are assumptions chosen to look like a coding agent, not measurements of one.
What matters for a synchronous step is not the mean duration but the duration of the slowest of the BB episodes in the batch, because the trainer cannot start until it has all of them. We draw a large pool and compare the mean with the expected maximum of a batch, estimated by splitting the pool into batches.
Output:
The median episode takes about a minute and the mean about a minute and a half, but one episode in a hundred takes several times that. The slowest of a batch grows with the batch size like a high quantile of the distribution: the expected straggler is 2.6 times the mean at B=8B = 8, 4.3 times at B=64B = 64, and 5.4 times at B=512B = 512. A synchronous system spends each step waiting for that straggler, and while it waits, the slots whose episodes have already finished do nothing. This is the straggler problem, and it gets worse as the batch grows, which is the opposite of what scaling the batch is supposed to do.

​
13.3 Synchronous, one-step-ahead, and asynchronous pipelines

Three designs answer the straggler problem differently.
  • Synchronous (on-policy). Sample a batch with version vv, train on it, sync to v+1v+1, repeat. Every episode is trained by the version that generated it, so the gradient is exactly the on-policy estimator of Chapter 6. The step time is E[max⁡i≤Bdi]+ttrain+tsync\mathbb{E}[\max_{i \le B} d_i] + t_{\text{train}} + t_{\text{sync}}, and generation and training never overlap.
  • One step ahead (one-step off-policy). While the trainer trains on batch nn, the engine already samples batch n+1n+1 with version vnv_n. Training overlaps generation, and every episode is exactly one version stale. verl documents this mode (Chapter 20).
  • Fully asynchronous. Generation slots never wait: whenever an episode finishes, the slot starts another with the newest weights. The trainer takes the oldest BB finished episodes whenever it is free. A staleness bound η\eta throttles generation so it cannot run more than η\eta batches ahead of training; AReaL’s condition, quoted in Chapter 20, allows a new episode only while the number started is below (v+1+η)B(v + 1 + \eta) B. With η=0\eta = 0 the design reduces to synchronous.
A weight sync in an asynchronous system meets episodes that are still running. Either the engine drains (stops starting episodes, waits for the in-flight ones to finish under the old weights, then syncs), or it interrupts them: in-flight episodes pause during the sync and continue under the new weights. The second option is called partial rollout (AReaL calls these interruptible rollouts; Kimi k1.5’s partial rollouts cap the token budget per iteration and resume later). An interrupted episode’s early tokens come from version vv and its later tokens from v+1v+1 or later, so no single policy generated it. The simulator below models exactly these choices. It has n_slots generation slots (concurrent episodes in the engine), one trainer that consumes batch_size episodes per step and takes train_time seconds, and a sync that takes sync_time seconds and pauses all generation. It is a discrete-event simulation: time jumps from one event (an episode finishing, a training step finishing, a sync finishing) to the next. Episode durations are consumed in order from one tensor, so every design sees the same episodes. It records every interval it simulates, so we can draw the timeline, and it reports throughput, slot utilization, and the staleness of every trained episode: the trainer’s version at the start of its step minus the version that started the episode.
Each pass of the outer loop does four things. It fills free slots while the throttle allows, unless a sync is running or, in drain mode, waiting. It starts a training step if the trainer is idle and at least batch_size episodes are finished, recording each one’s staleness. It advances time to the earliest pending event. And it processes that event: a finished sync publishes the next version, a finished training step requests a sync, a finished episode joins the buffer. A requested sync starts at once with partial rollouts, pushing the end time of every in-flight episode back by sync_time, or, in drain mode, only when no episode is in flight. Slot utilization counts an interrupted episode’s pause as busy time, a small overstatement. We run five designs on the same 64 slots, 64 episodes per step, 60 seconds of training, and 10 seconds of sync, for 20 steps.
Output:
The synchronous pipeline trains 451 episodes per hour, and its slots are busy 19% of the time. Both numbers follow from Section 13.2: a step lasts about as long as the slowest of 64 episodes (415 s expected) plus 70 s of training and sync, and 64/485 s≈47564 / 485\ \text{s} \approx 475 episodes per hour; the simulation is slightly lower because the slowest episode varies from batch to batch. Overlapping training with generation but still draining before every sync (“one step ahead, drain”) gains only 12%, since training is 60 s of a step that is mostly straggler wait. Without draining, the picture changes completely. With partial rollouts and η=1\eta = 1, slots are busy 81% of the time and throughput is 4.1 times the synchronous one; at η=2\eta = 2, 95% and 4.6 times. Here the bound is no longer the straggler but generation capacity: 64 slots finishing a 96-second episode each give at most 64/96×3600≈2,40064 / 96 \times 3600 \approx 2{,}400 episodes per hour, less the time the syncs pause them. A loose bound that drains (η=4\eta = 4, drain) is as slow as the synchronous design, because every sync still waits for the longest episode in flight. Two columns show the cost. The mean staleness of trained episodes is about 1 at η=1\eta = 1 and 1.5 at η=2\eta = 2, but the maximum is 5 and 7: the throttle counts episodes started, so a straggler that started at version 0 keeps running while newer episodes pass it, and is trained four or five versions later. And the trained episodes are slightly shorter on average than the synchronous ones (93 to 94 s against 98 s): taking the oldest finished episodes first means a batch is filled with fast ones, and the slow ones are trained later or, at the end of a run, never. Both effects are invisible in a throughput dashboard. Figure 13.1: Simulated timelines of the first steps of a synchronous pipeline (top) and an asynchronous pipeline with staleness bound 1 and partial rollouts (bottom), 8 generation slots and 8 episodes per step. Each bar is one episode on one slot, shaded by the policy version that started it; aqua bars are training steps and yellow bands are weight syncs. Figure 13.1 draws the simulator’s events for a small pipeline with 8 slots. In the synchronous half, every step has the same shape: most slots finish early and wait, idle, until the longest episode of the batch completes; then the trainer runs while all slots wait again. The four steps take 1,130 simulated seconds. In the asynchronous half, a slot starts a new episode as soon as its previous one finishes, the trainer starts as soon as 8 episodes are done, and the four steps take 493 s. Look at the long bars that cross the yellow sync bands: those are partial rollouts, episodes started under one version and finished under the next, and the label on each bar is only the version that started it. Figure 13.2: Throughput against staleness for the simulated pipeline (64 slots, 64 episodes per step). Left: episodes trained per hour as the staleness bound grows, with partial rollouts and with draining. Right: mean and maximum staleness of trained episodes with partial rollouts. Figure 13.2 sweeps the bound. Throughput with partial rollouts rises steeply from η=0\eta = 0 to η=2\eta = 2 and is flat after that, because generation capacity becomes the limit; draining keeps throughput near the synchronous level for every bound. On the right, the mean staleness of trained episodes stays below the bound and stops growing at 1.5, but the maximum exceeds the bound for η≤6\eta \le 6. The design lesson is that most of the throughput of asynchrony is available at a staleness bound of 1 or 2, and that a bound on generation does not bound the age of the oldest data unless it is also enforced when the trainer consumes episodes.
Evidence. Everything in this section is a simulation under stated assumptions: the duration model of Section 13.2, a fixed training time, and a sync that pauses all generation. It shows mechanisms (straggler waits, overlap, the drain penalty, staleness that a count-based throttle does not bound). Speedups reported by real systems (2.77x for AReaL, 2.72x for ROLL Flash on agentic tasks; Chapter 20) come from different workloads and are not reproduced here.

​
13.4 Importance correction for a behavior policy

Asynchrony changes the estimator. Let μ\mu be the behavior policy, whatever actually produced the tokens: a version of the weights kk updates old, an inference engine with different numerics, or a mixture of versions within one interrupted episode. The trainer holds πθ\pi_\theta. The policy gradient of Chapter 6 is an expectation under πθ\pi_\theta, and the data come from μ\mu. Per token, with the per-token approximation of Section 8.1, ∇θJ≈Ey∼μ[πθ(y∣c)μ(y∣c) A^ ∇θlog⁡πθ(y∣c)].\nabla_\theta J \approx \mathbb{E}_{y \sim \mu}\Big[\frac{\pi_\theta(y \mid c)}{\mu(y \mid c)}\, \hat A\, \nabla_\theta \log \pi_\theta(y \mid c)\Big]. Ignoring the ratio, which is what rl_update does when it recomputes old_logp with the trainer, estimates Eμ[A^∇log⁡πθ]\mathbb{E}_\mu[\hat A \nabla \log \pi_\theta] instead. That is a biased estimator of the gradient we want, and the bias grows with the distance between μ\mu and πθ\pi_\theta. Including the full ratio removes the bias and brings back Chapter 8’s problem: the ratio is unbounded, and its variance explodes where μ\mu puts little mass on tokens that πθ\pi_\theta likes. PPO’s clip does not solve this. The clip bounds πθ/πold\pi_\theta / \pi_{\text{old}}, where πold\pi_{\text{old}} is the trainer’s copy of the policy at the start of the update. When μ≠πold\mu \ne \pi_{\text{old}}, there are two ratios, and AReaL’s decoupled objective (Chapter 20) keeps them apart: a behavior weight πold/μ\pi_{\text{old}}/\mu that corrects which distribution sampled the data, and PPO’s clipped ratio πθ/πold\pi_\theta/\pi_{\text{old}} that limits how far this update moves. With w=πold/μw = \pi_{\text{old}}/\mu treated as a constant, L(θ)=− Ey∼μ[ w⋅min⁡(ρ A^, clip(ρ,1−ϵlow,1+ϵhigh) A^)],ρ=πθπold.L(\theta) = -\,\mathbb{E}_{y\sim\mu}\Big[\, w \cdot \min\big(\rho\, \hat A,\ \mathrm{clip}(\rho, 1-\epsilon_{\text{low}}, 1+\epsilon_{\text{high}})\, \hat A\big)\Big], \qquad \rho = \frac{\pi_\theta}{\pi_{\text{old}}}. Since w≥0w \ge 0, wmin⁡(ρA^,clip(ρ)A^)=min⁡(ρ wA^,clip(ρ) wA^)w \min(\rho \hat A, \mathrm{clip}(\rho)\hat A) = \min(\rho\, w\hat A, \mathrm{clip}(\rho)\, w \hat A), so the decoupled loss is Chapter 8’s ppo_policy_loss with per-token advantages wA^w \hat A. That is how the code below implements it. Two ways to tame ww are in use:
  • Truncated importance sampling (TIS): w=min⁡(πold/μ, C)w = \min(\pi_{\text{old}}/\mu,\ C). The TIS blog post (Yao et al., 2025; Chapter 20) proposes it for the sampler–trainer mismatch, and verl’s rollout-correction presets use C=2C = 2. Truncation bounds the second moment of each weighted term by C2C^2 times the unweighted one, and it leaves a bias: tokens with πold/μ>C\pi_{\text{old}}/\mu > C are under-weighted, which means tokens that the sampler under-sampled relative to the trainer stay under-represented.
  • Masked importance sampling (MIS), IcePop in the Ring-1T report: w=πold/μw = \pi_{\text{old}}/\mu inside a band [α,β][\alpha, \beta] and 00 outside, with α=0.5\alpha = 0.5 and β=5\beta = 5 in the report. A token whose two probabilities disagree by more than the band is treated as noise and removed from the gradient, rather than kept with a capped weight.
Both are saved as functions of the two log-probability tensors and the action mask. They return detached weights, zero off the mask.
How much bias and variance does each weight leave? A one-step problem answers that exactly, because there the true gradient is known. Take a softmax policy over V=100V = 100 tokens with logits zz and a fixed reward per token. The gradient of Eπ[r]\mathbb{E}_{\pi}[r] with respect to zz is π⊙(r−Eπr)\pi \odot (r - \mathbb{E}_\pi r). The sampler’s logits are zz plus Gaussian noise of scale σ\sigma, a stand-in for a mismatch of growing size. Each estimate uses 64 sampled tokens with a batch-mean baseline, and we repeat it 4,000 times to measure its squared bias and its variance, both relative to the squared norm of the true gradient.
Output:
Read each cell as (squared bias, variance), both relative to ∥∇J∥2\lVert\nabla J\rVert^2; the mean squared error is their sum. For a small mismatch (σ≤0.5\sigma \le 0.5, a KL of about 0.03 nats or less), ignoring the ratio already leaves a bias of 1% to 5% of the gradient’s squared norm, and all three corrections remove it at no cost in variance, because almost no ratio comes near the cap or the band edges. For a large mismatch, the three corrections separate. Full IS stays unbiased and its variance grows to 1.3 and then 10.2 times the squared gradient: a single token that μ\mu rarely samples but π\pi likes carries an enormous weight. TIS keeps the variance near the uncorrected one and gives up most of the bias correction, since the tokens that need weights above 2 get 2. MIS is worse than no correction at σ≥1\sigma \ge 1: it removes exactly the tokens that carry the largest correction. Truncated and masked IS are tools for a small mismatch with occasional outliers, which Section 13.6 will show is the regime of numerical mismatch; they are not a cure for a policy that has moved far from the data.

​
13.5 Stale policies on TinyAgent

The simulator says how stale the data get; it cannot say what staleness does to learning. For that we train TinyAgent with a behavior policy that lags the trainer by a fixed number of updates. offpolicy_update is rl_update of Chapter 9 with one change: the batch’s sample_logp, the behavior policy’s log-probabilities recorded at sampling time, enters through the decoupled weights. With correction='none' the weights are the action mask, and the update is exactly rl_update: the trainer recomputes old_logp and never looks at μ\mu. It also reports the mean absolute log-ratio between πold\pi_{\text{old}} and μ\mu (mismatch) and the share of policy tokens whose ratio exceeds the cap (over_cap).
train_offpolicy keeps the last lag + 1 versions of the weights and samples every step with the oldest, so after the first lag steps every batch is exactly lag updates stale, the situation of a one-step-ahead pipeline at lag=1 and of an asynchronous one with a fixed staleness. An optional sampler function transforms the behavior model before sampling; Section 13.6 uses it to make a numerically different sampler.
With lag=0 and no correction, this must be train_rl step for step, since both consume the same random streams in the same order. Checking that before trusting any comparison costs five steps.
Output:
The rewards agree to the last digit at every step, so any difference in the experiment below comes from staleness and correction, not from the loop. Now the experiment: 40 steps of the default configuration from the RL base agent, with the behavior policy 0, 2, 4, or 8 updates stale, without correction, and at 4 and 8 with TIS (C=2C = 2) and at 8 with MIS ([0.5,5][0.5, 5]).
Output:
Without correction, staleness 2 did no harm in this run: the held-out pass rate is 0.76 against 0.70 for the on-policy baseline, a difference inside single-seed noise (about 0.05 standard error on 80 episodes). Staleness 4 slowed learning, with training reward 0.43 against 0.66 in steps 20 to 29, and ended at 0.44 held-out. Staleness 8 destroyed the skill: reads per episode fell to 0.14 and the held-out pass rate to 0.00, below the 0.20 of a blind guess. The mismatch column explains the dose response. The mean ∣log⁡πold−log⁡μ∣|\log \pi_{\text{old}} - \log \mu| per policy token is 0.07 nats at lag 2, 0.20 at lag 4, and 0.70 at lag 8: TinyAgent’s policy moves fast (its entropy falls from 0.57 to about 0.1 within 40 steps of Chapter 9’s run), so eight updates are a large distance, and an uncorrected gradient computed from samples of a policy that read less pushes the current one in the wrong direction. Correction recovers most of the loss. With TIS (C=2C = 2), lag 4 reaches 0.70 held-out, the on-policy number, and lag 8 reaches 0.55; with MIS at lag 8, 0.64. Only 1.6% to 3% of policy tokens had a ratio above 2, so the cap is not what did the work. The weights below 1 did. This is a different regime from the one-step simulation of Section 13.4, where the mismatch was symmetric noise on the logits and large ratios above the cap were what carried the bias. Staleness is a systematic shift: the current policy has sharpened, so with a mean absolute log-ratio of 0.14 to 0.70 nats in these runs but few ratios above 2, most of the disagreement is ratios below 1, which TIS passes through unchanged: tokens that the stale policy liked and the current one does not are down-weighted, which is the bias that the uncorrected update cannot see. The MIS run also shows a side effect: 1.5 reads per episode, more than any other run. A plausible cause, which we did not test, is that masking drops the gradient of exactly the tokens whose probability changed most, including the ones that would have suppressed a redundant read.
Evidence. Measured here, one seed per configuration, 40 steps on a 157k-parameter policy. The ordering (no correction degrades with lag; TIS and MIS recover much of it) is the mechanism this section is about. The specific lags at which TinyAgent breaks reflect its learning rate and how fast its policy changes; a large model trained with a learning rate of 10−610^{-6} moves far less per update, which is why public asynchronous systems tolerate staleness bounds of 4 to 8 (AReaL’s reported defaults, Chapter 20).

​
13.6 The training–inference mismatch

Staleness is a mismatch we choose. The second kind arrives without being chosen. In a production system the behavior policy is the inference engine, and even at the same weights its probabilities are not the trainer’s. The engine computes each sampled token’s distribution in a decode step: one new token per sequence, attention over a KV cache, a batch whose composition changes from step to step as sequences start and finish, kernels chosen for that shape. The trainer computes the same distributions in one forward pass over the whole sequence, with a different batch, different padding, often different parallelism and kernels. Floating-point addition is not associative, so different reduction orders give different results, and bf16’s 8-bit mantissa makes the differences visible. Section 3.6 measured the precision part on one sequence; here we measure the whole path on the real Qwen3-0.6B, loaded from the check host’s cache, on its GPU when one is available.
engine_sample imitates an inference engine without being one: a left-padded batch, a KV cache, one decode step per token, sampling at temperature 1, and the sampled token’s log-probability taken from that step’s logits (normalized in fp32, as selected_log_probs does). vLLM and SGLang add paged attention, continuous batching, and fused sampling kernels, which change the numbers further; this is the minimal version of the same computation path.
The trainer side recomputes the sampled tokens’ log-probabilities with one full forward pass per batch of complete sequences, right-padded, as a trainer does. batch_size sets how many sequences share a forward pass.
We compare the engine with a bf16 trainer that scores each sequence alone and one that scores all 30 in one padded batch, and repeat with fp32 weights. The last two rows compare trainers with each other: the same weights and precision, only the batch composition differs.
Output:
The engine-style and trainer-style passes disagree by 0.03 nats per token on average in bf16, with a median of 0.019 and a 99th percentile of 0.13; the largest single-token gap is 0.23 nats, a probability ratio of 1.26. Summed over a 32-token continuation, the gap reaches 0.68 nats, a sequence-level ratio of about 2. Three comparisons locate the source. Recomputing in fp32 instead of bf16 reduces the mismatch with the bf16 engine only a little (mean 0.025), because the engine’s own bf16 numbers are as far from fp32 as the bf16 trainer’s are. The fourth row is the most important one for practice: the same bf16 trainer gives log-probabilities that differ by 0.03 nats on average depending only on whether each sequence is scored alone or in a padded batch of 30, which is as large as the whole engine–trainer mismatch. In fp32, the same comparison differs by 6×10−66 \times 10^{-6}. bf16 kernels are not batch-invariant: the batch shape changes the matmul tiling and reduction order, and the 8-bit mantissa makes the difference visible. An inference engine changes its batch composition at every decode step, so its numbers cannot match any one trainer pass. The measurement was deterministic: rerunning it in a fresh process on the same GPU reproduced every number. Nondeterminism here means dependence on the batch, not randomness between runs. Where does the mismatch sit? We bin the 960 tokens by their engine probability.
Output:
The mismatch is concentrated on uncertain tokens: tokens the engine sampled with probability below 0.1 differ by 0.045 nats on average, those above 0.9 by 0.002. This is consistent with the report in When Speed Kills Stability that the mismatch concentrates on low-probability tokens (Chapter 20). In ratio terms, every token-level ratio π/μ\pi/\mu lies in [0.79,1.22][0.79, 1.22], far inside a TIS cap of 2 and IcePop’s band [0.5,5][0.5, 5], so at this model size and step, token-level TIS and MIS change nothing beyond reweighting by ratios near 1. Sequence-level ratios range from 0.51 to 1.64 over only 32 tokens; a sequence-level correction for a 30,000-token agent trajectory would see ratios spread over orders of magnitude, which is why public proposals for long sequences mask or truncate at the sequence level rather than weight by the raw product. Figure 13.3: The training–inference mismatch on Qwen3-0.6B with identical weights. Left: distribution of |log pi_trainer - log mu_engine| over 960 sampled tokens for a bf16 trainer, an fp32 trainer, and two bf16 trainers that differ only in batch composition (log scale; fp32 trainer pairs agree exactly and are not shown). Right: mean absolute mismatch by engine probability of the sampled token. Figure 13.3 shows both facts as distributions. The cumulative distributions of the bf16 engine-versus-trainer gap and of the gap between two bf16 trainers lie on top of each other, with the fp32 trainer only slightly to the left: the mismatch is a property of bf16 kernels under changing batch shapes, and higher precision in only one of the two engines does not remove it. The right panel shows its dependence on the token’s probability. At the left end, about 3% of tokens differ by less than 10−410^{-4} nats in every comparison.

​
A mismatched sampler on TinyAgent

Does a mismatch of this size hurt training, and do the corrections help? TinyAgent runs sampler and trainer in one process, so we make the sampler different on purpose. FP8 rollouts are one motivation in the TIS post: the engine runs with low-precision weights for speed while the trainer keeps full precision. fp8_copy rounds every weight of a copy to float8_e4m3fn and back, a 3-bit mantissa.
Output:
The FP8-rounded sampler differs from the trainer by 0.009 nats per policy token, less than a third of Qwen3-0.6B’s bf16 mismatch, and no token’s ratio exceeded the cap. Both runs reach 0.75 held-out, against 0.70 for the exact sampler in Section 13.5; the reward curves differ by sampling noise. TIS had nothing to correct. This agrees with the TIS post’s own report that where the probability difference is small, TIS brings no gain (Chapter 20). It is also a useful negative result for a design review: before adding a correction, measure the mismatch distribution and the share of tokens whose ratio leaves the band. A correction that never activates costs nothing but proves nothing either, and the mismatch that matters in practice is the one that grows, through staleness, quantization of a larger model, or a mixture-of-experts router that flips (Appendix B, Problem 20).

​
13.7 Design problems in practice

Problem: straggler episodes. Symptom: accelerator utilization of the inference fleet falls as the batch grows; step time tracks the slowest episode; the tail of the episode-duration histogram moves with every change of task mix. Cause: a synchronous step waits for max⁡idi\max_i d_i over the batch, which grows like a high quantile of the duration distribution (Section 13.2). Options: (a) asynchronous generation with partial rollouts, which removes the wait and adds staleness. (b) A wall-clock or turn timeout that truncates the tail, which changes the task: truncated episodes need a policy (Chapter 12), and masking them out of the loss (DeepSWE’s compact filtering) biases training toward tasks that finish quickly. (c) Over-sampling: launch more than BB episodes and train on the first BB to finish, which is the same length bias in a different form. Default: the book recommends (a) with the duration of trained versus launched episodes logged, plus a generous timeout as a safety net. The block below measures what a timeout at the 95th percentile buys a synchronous pipeline.
Output:
Truncating the 5% longest episodes at 279 s raises synchronous throughput from 451 to 661 episodes per hour, a 47% gain that asynchrony beats by a factor of three, and it changes the training data: every truncated episode needs a reward decision, and the truncated ones are not a random 5%.
Problem: choosing a staleness limit. Symptom: an asynchronous run is faster per hour and worse per sample than its synchronous baseline; mismatch between the trainer’s old_logp and the sampler’s log-probs rises with throughput; in the worst case the policy collapses. Cause: the gradient is estimated from μ≠πold\mu \ne \pi_{\text{old}}. Section 13.5 measured the dose response on TinyAgent: staleness 2 was harmless, 4 cost about a third of the held-out pass rate without correction, and 8 destroyed the skill. Options: (a) a tight bound (1 or 2), which caps throughput (Figure 13.2 shows most of the gain arrives by then). (b) A looser bound with decoupled TIS or MIS. (c) Enforce the bound at consumption too, dropping episodes older than η\eta, since a count-based throttle bounds the mean staleness but not the maximum. Default: the book recommends η∈{1,2}\eta \in \{1, 2\} with TIS from the start, and logging the staleness histogram of every trained batch, not just its mean. Policy speed matters as much as the bound: staleness is measured in updates, and a larger learning rate makes each update a bigger step.
Problem: the cost of weight synchronization. Symptom: step time has a fixed floor that does not shrink with more inference capacity; the engine’s throughput graph shows periodic gaps. Cause: every update must reach every engine replica, and generation pauses while it does (Section 13.1 estimated seconds to a minute per sync for a 32B model). Options: (a) colocate engine and trainer and reshard in place, which avoids the network but serializes generation and training. (b) Broadcast over NCCL from the trainer to a disaggregated engine. (c) Sync less often (several updates per sync), which raises staleness. (d) Overlap the broadcast with generation and swap weights atomically, which costs memory for two copies. Default: (b) with the sync time logged as its own phase. The block shows how the synchronous and asynchronous designs respond to the sync cost.
Output:
Raising the sync cost from 1 s to 60 s costs the synchronous design 11% of its throughput, because its step is dominated by the straggler. It costs the asynchronous one 37%, because in a design that has removed the straggler wait, the sync pause is a large part of what remains. The faster the pipeline, the more the sync matters.
Problem: environment worker failures and retries. Symptom: a few percent of episodes end with a worker error (container crash, hung test process, out-of-memory); the reward of the surviving episodes looks better than it should, or the trained batch drifts toward short tasks. Cause: failures are not independent of the episode. If a worker fails with a constant hazard per second, a long episode is more likely to fail, so dropping failed episodes removes long ones preferentially, and retrying them silently doubles their cost. Options: (a) drop failed episodes and accept the bias. (b) Retry on a fresh worker, from the start or from a checkpointed state, and count the retry’s cost. (c) Mark the episode as an environment failure, give it no gradient, and keep its task in the next batch so the task mix is preserved. Default: (c), with infrastructure failures logged separately from agent failures, never scored as reward 0: a reward of 0 for a crashed container teaches the policy to avoid whatever made containers crash. The block measures the length bias of dropping failures under a hazard of one failure per 2,000 seconds.
Output:
With a constant hazard, 4.5% of all episodes fail, but 13.6% of the longest tenth do, three times the average. Dropping failures shortens the mean surviving episode from 96 to 92 s. If length correlates with difficulty, as it does for coding tasks, dropping failures also drops hard tasks.
Problem: nondeterminism. Symptom: the same seed gives different rollouts on a rerun; the ratio πold/μ\pi_{\text{old}}/\mu is not exactly 1 at step 0; a bug cannot be reproduced. Cause: Section 13.6 measured it: in bf16, the same trainer with the same weights gives log-probabilities that differ by about 0.03 nats per token depending only on which other sequences share the batch, while in fp32 the two agree exactly on this GPU. Inference engines change batch composition at every decode step. Options: (a) accept it, recompute old_logp in the trainer, and correct the remaining sampler mismatch with TIS or MIS. (b) Batch-invariant kernels, which make results independent of batch composition at a throughput cost; the Thinking Machines post reports a bitwise-identical sampler and trainer and a KL of exactly 0 (Chapter 20). (c) Higher precision for the parts that matter most (the LM head and log-softmax in fp32, as MiniMax-M1 reports). Default: (a) plus (c), and monitor the step-0 mismatch distribution (mean, p99, max of ∣log⁡πold−log⁡μ∣|\log \pi_{\text{old}} - \log \mu|) as a health metric: a sudden rise means a kernel, precision, or version bug, not learning.

​
Summary

  • A production RL system has an inference engine, a trainer, environment workers, and weight synchronization. Each has its own failure modes, and the version of the policy that generated each sample must be recorded, or staleness cannot be measured.
  • Agent episode durations are long-tailed, so a synchronous step waits for its slowest episode: 4.3 times the mean duration at 64 episodes per step in the simulator, with slots busy 19% of the time. Asynchronous generation with partial rollouts raised simulated throughput 4.1 to 4.6 times; draining before each sync removed almost all of the gain. A count-based staleness bound limited the mean staleness but not the maximum.
  • Off-policy data need the decoupled view: a behavior weight πold/μ\pi_{\text{old}}/\mu for the sampling distribution, PPO’s clip for the update. In a one-step simulation, TIS and MIS removed the bias of a small mismatch at no variance cost and did not fix a large one; full IS fixed the bias at exploding variance.
  • On TinyAgent, sampling with a policy 8 updates old collapsed the held-out pass rate to 0.00; TIS recovered 0.55 and MIS 0.64 (one seed each). Staleness 2 did no harm.
  • On Qwen3-0.6B with identical weights, engine-style and trainer-style bf16 log-probabilities differed by 0.03 nats per token on average (99th percentile 0.13), concentrated on low-probability tokens, and two bf16 trainer passes differed by as much from batch composition alone; fp32 passes agreed to 10−510^{-5}. Token-level ratios stayed within [0.79,1.22][0.79, 1.22].

​
Exercises

  1. Derive the expected synchronous step time E[max⁡i≤Bdi]+ttrain+tsync\mathbb{E}[\max_{i \le B} d_i] + t_{\text{train}} + t_{\text{sync}} for exponentially distributed durations with mean mm, and show that it grows like mln⁡Bm \ln B. Check the prediction against simulate_pipeline with exponential durations.
  2. Extend simulate_pipeline so the trainer drops episodes whose staleness exceeds max_staleness at consumption. Plot throughput and the fraction of episodes discarded against η\eta. At what bound does discarding stop costing throughput?
  3. Change an assumption: rerun the staleness experiment of Section 13.5 with RLConfig(lr=3e-4). Does lag 8 still collapse? Explain the result in terms of the mismatch column.
  4. Implement sequence-level TIS: one weight per episode, min⁡(exp⁡(∑tlog⁡πold−log⁡μ),C)\min(\exp(\sum_t \log \pi_{\text{old}} - \log \mu), C), broadcast to its tokens. Compare it with token-level TIS at lag 8. Which one has more weights at the cap, and why?
  5. Measure the Qwen3-0.6B mismatch with the LM head and log-softmax in fp32 but the rest of the model in bf16 (on both sides). How much of the mismatch does it remove? Relate your result to the MiniMax-M1 report cited in Chapter 20.
  6. In engine_sample, decode the same 30 prompts in batches of 1, 10, and 30. How large is the engine’s own batch-composition mismatch? What does this imply for comparing log-probs between two runs of the same engine?
  7. Design review: your asynchronous agent run with η=4\eta = 4 trains 3 times faster per hour than the synchronous baseline but reaches a lower held-out pass rate at the same number of samples. List the measurements you would ask for (staleness histogram, mismatch distribution, length of trained versus launched episodes, capped-ratio fraction) and the decision each one informs.