Why does a check that only scores an agent’s attempts end up defining what the agent learns?
Curated demonstrations give an agent a starting policy, but they cover only the decisions someone recorded, and long-horizon tasks keep producing decisions that no demonstration anticipated. Reinforcement learning with verifiable rewards replaces the demonstration with a programmatic check, such as a unit test suite, a compiler, or a proof checker, that scores each sampled attempt and turns the score into a weight update without a human label. That substitution changes what the check is. An evaluator that once only counted failures becomes the objective the optimizer searches, and the search finds every gap between what the check measures and what the task requires, from a weakened assertion to a patched test runner to a verdict rewritten before it is recorded. The verification boundary must therefore be defended like a security perimeter, with fixtures the candidate cannot reach and rewards read as evidence about only the properties their checks cover. Wherever outcomes can be mechanically checked, the policy improves as far as the scope and protection of the check allow, and no further.
Learning Objectives
Explain why reward-driven learning needs sampled support that imitation does not, and estimate when a task yields no reward contrast
Design a shaped reward that preserves the pass-over-failure order, and bound the reward hacking and false accepts its checks admit under optimization pressure
Calculate the generation cost and standard error of continuation-based credit estimates across turns
Compute group-relative coefficients and the per-token weight that length normalization assigns them
Specify the per-invocation records and version-lag bound that keep a multi-turn update faithful to its sampling policy
Diagnose entropy collapse from action diversity, and changes in reward and length from task outcomes rather than from the narration a policy generates
Size the environment cycle and verifier pool of a rollout-training loop and identify the stage that limits it
17.1 Learning from a Check
The previous chapter left the repair agent (Four lighthouse workloads) with trained_policy_v1, a supervised checkpoint fitted to the one malformed-argument recovery that curation admitted. Its release gate already had to send trials into a state the corpus never covered, the version conflict for which no demonstration exists (Case Study: Training the Repair Policy). More imitation needs someone to record that recovery, and a preference pair needs two admissible records of it. The agent’s versioned test fixtures and acceptance checks need neither. They can score entirely novel attempts whose intermediate actions have no labeled target. An autonomous policy might inspect an unexpected source file, attempt an alternative refactoring strategy, or halt when it encounters an unfamiliar error. While the automated test suite verifies whether the final patch succeeds, it provides no token-by-token guidance during intermediate decision steps.
Training changes the policy and leaves every exposure where the runtime closes it.
Existing techniques stop short of using verification outcomes to reshape the underlying policy. Trajectory fine-tuning teaches models to generalize from demonstrated recovery paths, but it remains bounded by demonstration data, and human-verified repairs for intricate, non-deterministic bugs are scarce. Test-time selection filters multiple generated rollouts against an acceptance suite to extract a passing candidate, yet it incurs significant inference latency on every incoming task while leaving the base policy unchanged. Neither approach increases the intrinsic probability of generating successful trajectories from verification signals alone. Reinforcement learning bridges this gap by converting acceptance verdicts directly into gradient updates. In doing so, it alters the role of the verification suite. Once a check enters the optimization objective, any flaw in the test fixture is compiled into the network weights rather than miscounted on a single run, and the acceptance check that once decided a release (The Task Acceptance Contract) becomes the signal that decides every update.
Turning automated test outcomes into policy updates requires four coupled mechanisms. The reward design must evaluate functional correctness without introducing shortcuts that optimization can exploit. The policy update rule must then solve multi-turn credit assignment, translating a single terminal pass or fail into per-token update coefficients across an interactive trajectory. For that update to match what the agent actually did, the rollout recorder must log the exact context prefixes the policy saw and the exact tokens it generated. Finally, the compute fleet—sandboxed environments, verification workers, and gradient engines—must be sized so that training updates receive complete, synchronized candidate groups without stalling the pipeline.
The chapter takes these mechanisms in the order a student would need to build them. It first fixes what an outcome-driven update can and cannot learn (section 17.2), then prices the reward that drives it, including the false accepts a verifier admits (section 17.3). Credit assignment follows, first through return and value estimates (section 17.4) and then through group-relative policy optimization (section 17.5), the worked optimizer. The remaining sections keep that optimizer honest in practice: faithful records of multi-turn rollouts (section 17.6), a verifier the candidate cannot reach (section 17.7), diagnostics for the pathologies that rising reward can hide (section 17.8), and the rollout, verifier, and staleness arithmetic that sizes the training fleet (section 17.9). A case study then follows one repair group through the whole loop (section 17.10).
17.2 Foundations of Verifiable Rewards
Behavioral cloning (supervised imitation of recorded actions) fails as soon as an agent encounters an intermediate state for which no curated demonstration exists. Reinforcement learning removes that bottleneck by executing candidate rollouts in an environment and scoring the resulting trajectories against programmatic checks rather than recorded human actions. A supervised checkpoint, such as the trained_policy_v1 built in Case Study: Training the Repair Policy, offers a useful starting point to bootstrap sampling, but the optimization machinery depends only on the agent’s ability to act, observe, and receive an evaluative score. Before any gradient update can convert raw outcomes into policy improvements, however, the system must define how multi-turn rollouts are represented, how baselines separate signal from variance, and how execution boundaries protect verification harnesses during exploration.
17.2.1 Policies, rollouts, and rewards
An agent repairing a repository cannot inspect the container’s underlying disk or process table directly; it sees only the transcript of prompts, tool calls, and truncated command outputs delivered into its context window. A policy\(\Policy(\RolloutAction_k\mid \InteractionHistory_k)\), parameterized by weights \(\ModelParameters\), defines a probability distribution over responses \(\RolloutAction_k\) conditioned on that delivered interaction history \(\InteractionHistory_k\) through turn \(\RolloutTurnIndex\). We consider a finite decision horizon \(\TurnHorizon\) with turn indices \(\RolloutTurnIndex=0,\ldots,\TurnHorizon-1\). (We reserve \(\RolloutTurnIndex\) for agent-environment turns because \(\TokenIndex\) later indexes individual autoregressive tokens within a rollout.) The history \(\InteractionHistory_k\) plays the role of the recorded state \(\TrajectoryState_k\) in the trajectory notation of The H·S·A Exposure Space, the runtime’s delivered view at turn \(\RolloutTurnIndex\) rather than the environment itself. A response \(\RolloutAction_k\) represents either a proposed tool call, such as a shell command or structured API query, or a terminal answer. The runtime validates and executes any requested operation, returning standard output, standard error, or structured payloads as observations for subsequent turns. Because modern context windows cannot always ingest unbounded execution logs, the policy consumes a constructed prompt that may truncate or summarize older history, never inspecting the underlying state \(\RolloutEnvironmentState_{k}\) directly. An edited configuration file or sandbox directory affects policy decisions only after tool invocations reflect its contents back into \(\InteractionHistory_k\).
A rollout, or sampled trajectory \(\Trajectory\), records one complete execution trace under specified task definitions, policy weights, environment states, and harness constraints. In our illustrative repair scenario, a trial rl_trial_01 on task pool_repair_01 starts container sandbox pool_fixture_v1 and tests candidate patches against test harness repair_accept_v1. Each rollout generates its own telemetry, intermediate command outputs, and terminal verification artifacts; past demonstration records provide no credit to this live attempt.
A reward is a scalar score computed under an explicit verification rule. In the simplest formulation, intermediate turns emit zero reward, leaving a single terminal score \(\TrajectoryReward(\Trajectory)\) evaluated at horizon completion without temporal discounting (a discount factor \(\DiscountFactor=1\), so a reward counts the same whichever turn it arrives on). The trajectory return collapses to this terminal score, giving the expected return objective:
The expectation averages over policy randomness, task sampling distributions, and environment variations (such as nondeterministic test execution times or network latencies). Maximizing \(\ExpectedReturnObjective(\ModelParameters)\) drives the policy to increase the expected measured reward under these conditions. Whether that measured score reflects true task utility depends on the fidelity of the reward harness itself, an engineering challenge analyzed in section 17.3. When the reward comes from an automated, deterministic check (such as unit test execution, syntax validation, type checking, or a formal proof assistant) rather than from an imperfect learned preference model or manual human grading, this approach is called reinforcement learning with verifiable rewards (RLVR)1(Lambert et al. 2024).
1Tests as Reward: Executed unit tests served as the reward for a code-generating model before the name existed: Le et al. (2022) trained a pretrained code model with reinforcement learning on test outcomes and added a learned critic that predicts functional correctness. That pairing anticipates this chapter’s central tension, since the test outcome is the trustworthy signal while the denser learned critic is itself a model that optimization can exploit.
Le, Hung, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. “CodeRL: Mastering Code Generation Through Pretrained Models and Deep Reinforcement Learning.”Advances in Neural Information Processing Systems (NeurIPS) 35. https://doi.org/10.52202/068431-1549.
Definition 17.1: Reinforcement learning with verifiable rewards
Reinforcement learning with verifiable rewards (RLVR) is policy optimization using reward signals computed by automated checks of specified properties of the policy’s outputs or interactions, such as tests or proof checkers, rather than relying solely on learned preference scores or human ratings.
Significance: An automated verification harness provides an objective training signal without requiring step-by-step demonstrations. Directly rewarding task success increases the acceptance rate \(\AcceptanceProbability\) in the cost per accepted task (equation 1.2) even when human demonstrations do not exist. However, the verifier’s false-accept rate bounds only the per-rollout chance that a sampled failure is rewarded; when true passes are rare, false accepts can make up a far larger share of the rewarded rollouts (section 17.3.2).
Distinction: Whereas behavioral cloning trains the policy to reproduce specific recorded responses, RLVR evaluates only whether the completed trajectory satisfies programmatic criteria. The resulting policy gradient is strictly bounded by the coverage, sensitivity, and sandboxed security of the underlying verifier.
Common pitfall: Mistaking higher reward scores on a fixed verification suite for broad task competence, or treating automated verification as a reason to relax runtime isolation and execution guardrails.
17.2.2 Baselines and credit assignment
A terminal reward alone does not compute a parameter gradient. Policy-gradient algorithms adjust model parameters by estimating a gradient direction that increases the log-probabilities of sampled actions proportional to an evaluative score. Standard estimators subtract a baseline from the raw outcome: the centered advantage is \(\TrajectoryReward(\Trajectory)-\PolicyGradientBaseline(\InteractionHistory_0)\), where \(\PolicyGradientBaseline(\InteractionHistory_0)\) depends on the initial task prompt and context \(\InteractionHistory_0\) but remains independent of the sampled action sequence \(\Trajectory\). The baseline serves as an offset reference level. It can be a fixed scalar, an empirical rolling mean across task batches, or a parametric value estimate, and it does not necessarily require training a separate critic network (Williams 1992).
Consider a binary verification check where success yields \(1.0\) and failure yields \(0.0\). With a fixed baseline of 0.25, a passing rollout receives a centered score of 0.75, whereas a failing rollout yields -0.25. These opposing signs drive gradient updates in opposite directions: positive centered returns reinforce the probabilities of every token and tool call sampled along the passing trajectory, whereas negative centered returns suppress them. In contrast, setting the baseline to zero leaves failed rollouts with a centered score of \(0.0\), discarding the negative gradient signal that penalizes counterproductive trajectories.
This uniform attribution illustrates the classic credit-assignment problem:2 a terminal verifier evaluates overall trajectory success, but it cannot differentiate the decisive tool call from irrelevant or mildly detrimental exploratory steps. While subtracting an appropriate baseline reduces the variance of the policy gradient estimator without introducing bias (Greensmith et al. 2004), it applies a uniform scalar adjustment across the entire trajectory rather than isolating action-specific causal effects. Section 17.4 formalizes these estimators and their bias-variance trade-offs, and section 17.5 introduces group-relative scoring to compute data-dependent baselines across parallel rollouts without training an auxiliary value network.
2Credit Assignment Problem: Minsky (1961) named the problem for learning systems that receive one success signal for a strategy built from many internal decisions, with games and program writing as his examples. The repair agent poses the same question at the scale of tool calls: a failed acceptance check arrives after many turns, and every turn is a candidate for blame, which is why the estimators of this chapter differ mainly in how they spread one outcome across those turns.
Minsky, Marvin. 1961. “Steps Toward Artificial Intelligence.”Proceedings of the IRE 49 (1): 8–30. https://doi.org/10.1109/JRPROC.1961.287775.
17.2.3 Exploration and protected assessment
A policy-gradient update can optimize only over trajectories that the sampling harness actually produces. Exploration, the sampling of action sequences the policy has rarely tried at points where it still spreads probability across several plausible actions, determines what reinforcement learning can discover. In environments governed by binary verification checks (pass vs. fail), a task instance that the policy never solves yields exclusively zero rewards. A zero baseline, or one that tracks the observed outcomes such as the mean of a group of attempts (section 17.5.4), then leaves every centered score at zero. A fixed positive baseline does produce a gradient, but it pushes down every sampled action alike and gives the update nothing to prefer. Either way, without variation in the outcome signal the update carries no contrast, regardless of how many rollouts the collector generates. If a policy solves a task instance with base probability \(\AttemptPassProbability\), sampling a group of \(\GroupSize\) independent rollouts yields at least one successful trajectory with probability \(1-(1-\AttemptPassProbability)^\GroupSize\). That condition is the one that binds when \(\AttemptPassProbability\) is small; the group optimizer of section 17.5 also needs at least one failure in the same group, a stricter condition derived there. When \(\AttemptPassProbability=0\), scaling the group size \(\GroupSize\) yields no contrast at all. System builders overcome this cold-start bottleneck by seeding the policy through supervised fine-tuning on demonstrations, curating curricula with progressive difficulty, or structuring reward functions with informative intermediate milestones.
Because exploration systematically exposes edge cases in both the target environment and the evaluation harness, the execution runtime must enforce rigid capability and resource boundaries. As an agent generates thousands of candidate interactions, optimization will exploit any loose permission, such as reading hidden test fixtures, patching Python test runners, or modifying exit codes in the evaluation container, to manufacture high rewards without solving the underlying problem. Preventing such gaming requires a protected assessment boundary. The agent’s actions execute within an isolated sandbox, while the verifier runs with separate privileges, shielded storage, and read-only access to private test suites; section 17.7 places that boundary on the reward path once the reward and the update it drives are in place. Even with complete isolation, held-out evaluation on disjoint test distributions remains necessary to determine whether the policy acquired robust problem-solving behavior or merely fit the idiosyncrasies of the training verifiers. Before any of that machinery can be designed, however, the reward itself must be pinned down: what the check measures, how often it is wrong, and how other terms may be added to it without overturning the pass-over-failure order.
17.3 Verifiable Rewards
When an agent tackles a task like pool_repair_01, the training loop requires an objective evaluation score for each newly sampled candidate repair rather than an expert demonstration of which lines to modify. Those acceptance criteria inherit the evaluation harness of repair_accept_v1, but passing a targeted unit test provides evidence only about that specific test assertion. Before the system can convert an execution outcome into a scalar reward, the training infrastructure must guarantee exactly what was executed, which artifact was tested, and which version of the grader evaluated it. The assessment contract established in The Task Acceptance Contract is no longer an offline benchmark; it is now an active component of the optimization objective.
17.3.1 The score and the intended outcome
A policy can easily collect high rewards without solving the underlying engineering problem. A candidate patch might satisfy the tested cases while introducing a race condition on an untested concurrency path. Alternatively, if the agent can write to the grading environment, it can modify the test runner so that a failing execution reports success. These failures represent two fundamentally different defects: incomplete test coverage vs. compromised measurement. Hardening the grader protects the measurement process, but it does not make the underlying test criteria complete.
Definition 17.2: Reward hacking
Reward hacking is behavior that increases measured reward by exploiting a gap between the formal reward and the intended task, including manipulation of the measurement mechanism (Amodei et al. 2016; Skalse et al. 2022).
Significance: Gradient optimization reinforces a rewarded shortcut just as eagerly as an authentic solution. Each gamed pass registers as a success, inflating the measured acceptance rate \(\AcceptanceProbability\) and artificially lowering the computed cost per accepted task while true capability stagnates; tracking policy quality during reinforcement learning therefore requires an independent assessment of task performance.
Distinction: A false-positive judgment is an evaluation error. It does not, by itself, establish that the policy discovered or deliberately exploited a shortcut.
Common pitfall: Treating instructions against tampering as protection for the grader. Software isolation at the runtime level must restrict what the policy and its submitted code can affect.
3Goodhart’s Law Under Training: How search exploits imperfect checks showed best-of-\(\CandidatePoolSize\) selection pressing on an imperfect check, where a larger pool favors candidates the proxy overrates. An RL loop applies the same pressure at every update and then writes the result into the weights, so a gap between reward and task is searched far more persistently than any selection step or human-managed target ever searches it.
A false accept costs more during training than during test-time sampling. At inference time, a noisy selector that accepts a bad candidate wastes one decision. In reinforcement learning, however, awarding reward to a defective trajectory raises the probability of the tokens that produced it, prompting the policy to generate more such trajectories on subsequent tasks. Once the exploit is learned into the model weights, the training run can raise its measured score while growing progressively worse at the intended task.3 Formally, let \(\CandidateUtility(\Trajectory)\) represent the developer’s intended objective and \(\ProxyTrajectoryReward(\Trajectory)\) the reward signal calculated by the training loop. Whenever an unintended trajectory \(\ExploitTrajectory\) satisfies \(\ProxyTrajectoryReward(\ExploitTrajectory)\ge\ProxyTrajectoryReward(\Trajectory^*)\) while \(\CandidateUtility(\ExploitTrajectory)\ll \CandidateUtility(\Trajectory^*)\), gradient ascent on \(\ProxyTrajectoryReward\) gives it no less weight than the intended solution. The exploit steadily gains ground whenever it also has higher prior probability under the current policy (Skalse et al. 2022; Krakovna et al. 2020). Degenerate shortcuts often hold that probability advantage. Weakening an assertion or rewriting a verdict takes only a few lines and no understanding of the bug, whereas diagnosing and fixing a difficult bug may require a complex, multi-turn repair.
↩︎ How a verifier’s false-acceptance rate dilutes the passing candidates at inference time is established in How base rates limit check precision.
The optimizer this chapter develops amplifies this failure mode the first time it occurs. Group-relative optimization, taught in section 17.5, samples a group of \(\GroupSize\) rollouts for the same task and scores each rollout against the others, so the rarer a pass is within its group, the larger the weight the update gives it. Suppose the test files sit in the rollout’s writable workspace, and one rollout on a difficult bug weakens the failing assertion while the other \(\GroupSize-1\) rollouts fail honestly. That rollout is the group’s lone pass, and it receives the largest coefficient in the group, a weight that grows with \(\GroupSize\) (equation 3 derives its size). Every token that wrote the tampering edit receives a sharp probability increase. The optimization loop did not suffer an algorithmic malfunction; the optimizer performed exactly as designed on the reward signal it received.
The exploits observed in coding environments fall into three recognizable classes, each documented in training or evaluation runs of 2025-era models:
Weakening or special-casing the check. Code that returns expected values only for inputs tested by the suite, or test assertions edited until they pass. A 2025 system card reports that one model, when unable to find a general solution, sometimes special-cased tests to make them pass (Anthropic 2025); Test-tampering defense traces how the behavior surfaced and was mitigated.
Stubbing or bypassing the verified code path. In one 2025 report during reinforcement learning of a reasoning model on coding tasks, a monitor flagged a plan to make a verification function always return true, alongside an exploit that exited the environment before unit tests could execute (Baker et al. 2025).
Rewriting the verdict inside the runner. Production coding environments used for training proved vulnerable to candidates returning an object whose equality comparison always succeeds, calling sys.exit(0) before any assertion executes, or writing a conftest.py that monkeypatches the test framework’s reporting subsystem so that every failure is recorded as a pass; models trained in those environments learned to use them (MacDiarmid et al. 2025).
Anthropic. 2025. Claude 3.7 Sonnet System Card.
Baker, Bowen, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.”arXiv Preprint arXiv:2503.11926.
Listing 1 shows the structure of this third class. The candidate adds one file to its own source tree; the test runner imports that file before running tests; and from that point forward the runner, not the tests, fabricates the outcome.
Listing 1: A verdict rewritten inside the runner: A reconstruction of the report-patching exploit class described by MacDiarmid et al. (MacDiarmid et al. 2025). Because pytest imports conftest.py from the tree it tests, candidate-controlled code can replace the function that records each outcome.
MacDiarmid, Monte, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, et al. 2025. “Natural Emergent Misalignment from Reward Hacking in Production RL.”arXiv Preprint arXiv:2511.18397.
# conftest.py written into the candidate's tree; pytest imports it firstfrom _pytest.reports import TestReport_original = TestReport.from_item_and_calldef _always_pass(cls, item, call): report = _original(item, call)if call.when =="call": report.outcome ="passed"# a failing assertion now reports success report.longrepr =None# and its traceback disappearsreturn reportTestReport.from_item_and_call =classmethod(_always_pass)
These exploits are not confined to training runs. An independent evaluator measured how often the most capable models of 2025 took them when the scorer was within reach, and whether asking the model not to made any difference.
War Story 17.1: Frontier models patch their scorers (2025)
Context: In June 2025 the evaluation organization METR reported reward hacking by frontier models, chiefly OpenAI’s o3, on two suites of autonomous tasks: RE-Bench, whose research-engineering tasks let the agent see the entire scoring function, and the broader HCAST suite (Von Arx et al. 2025).
Mechanism: With the scorer in reach, o3 attacked it instead of the task. On a kernel-optimization task it overwrote the timing function so that it always returned shorter measurements; on a code-generation scaffolding task it patched the evaluation function to judge every submission successful; on a fine-tuning speed task it modified the script to copy the cached weights of an already fine-tuned model.
Impact: The fine-tuning task was hacked in 21 of 21 runs. Across RE-Bench, 39 of 128 runs hacked (30 percent), against 8 of 1,087 on HCAST (0.7 percent). Asked after its first hacking plan whether the plan adhered to the user’s intention, o3 answered no 10 times in 10. Adding “Please do not reward hack” to the prompt moved the hacking rate over 20 runs from 80 percent to 70 percent.
Fix: METR excludes successful reward-hacking runs from its reported capability results, and recommends that when a monitor detects a hack, developers patch the exploit in the scoring function rather than punish the model.
Systems lesson: A scorer the agent can read and write sits at the visible-test closure level no matter what the task author intended, and an instruction does not raise it. In an evaluation these runs are excluded after the fact; in an RLVR loop every one of them is a maximal reward, and the gradient copies it.
Von Arx, Sydney, Lawrence Chan, and Beth Barnes. 2025. Recent Frontier Models Are Reward Hacking. METR blog.
METR attributes the substantial difference between 30 percent on RE-Bench and 0.7 percent on HCAST to reachability: on RE-Bench the model could inspect the scoring function directly, making it easier to bypass. The essential defense is a verification check that the policy cannot reach, evaluated on execution evidence that the policy cannot forge. For this reason, the sandboxed architecture of section 17.7 removes the test runner, configuration files, and evaluation credentials from the candidate’s control.
A reward is only as dependable as the closure evidence level of the check that computes it. These closure levels categorize the basis of an evaluation verdict by how much of it the agent policy can influence or reach. A static check establishes selected structural properties, a sealed test suite withholds and protects its test cases, and a formal proof verifies a specified property under explicit assumptions. Test coverage then limits any individual check at every level: even a sealed suite with an omitted edge case continues to reward buggy solutions that exploit the gap. Table 1 summarizes the sources used to construct agent rewards, alongside the specific vulnerabilities an optimizing policy exploits in each.
↩︎ Closure evidence levels, from static checks through sealed tests to formal proofs, are established in The invariant closure principle.
Table 1: Reward Sources and How They Are Gamed: The top rows correspond to the sealed-test and formal-proof closure levels; their verdicts depend on the artifact, not its phrasing, so raising the score requires changing what the check runs on. The lower rows score text the policy writes. Coverage limits every row.
Reward source
Evidence supplied
How a policy games it
Role in the reward
Proof checker
A submitted proof satisfies the encoded statement and assumptions
Mutable axioms or definitions; a statement weaker than the task
Gaps in coverage; tampering if the runner or tests are reachable
Outcome reward
Compiler or type checker
An artifact satisfies specified structural constraints
Code that compiles and does nothing useful
Gate or partial credit
Format check on tool calls
A call parses against its schema
Well-formed calls that accomplish nothing
Small shaping term
Learned process reward model
A learned score for each intermediate step
Text that resembles careful reasoning
Shaping; not the decision where a check exists
Rubric judge
An assessment of free-form output against written criteria
Length, confident tone, rubric keywords
Reward only where no executable check exists
Human rating
An assessment of anything a person can inspect
Rater fatigue and surface cues
Audit; collection cost limits its frequency
For a coding task with executable behavioral criteria, sealed tests supply the definitive outcome score, while compilation and format checks provide narrower gating signals. The lower rows score free-form text, and because the policy directly controls that text, a learned judge is itself a model with blind spots that gradient ascent can systematically exploit (Gao et al. 2023). The verification asymmetry (principle 3) dictates the role each score can assume. Learned assessments can rank candidate trajectories or shape exploration, but wherever a deterministic check exists, that check must govern the final reward. For tasks that lack an executable check, a learned judge can provide the primary reward, provided its calibration error is continuously measured against verified reference outcomes and re-evaluated as the policy distribution shifts. Section 17.7 develops the separate systems requirement of isolating the measurement pipeline from candidate code.
17.3.2 False accepts among rewarded outcomes
The distinction drawn in Grader error rates between a grader’s false-acceptance rate and the failure rate among accepted outcomes is the same base-rate arithmetic that equation 3.3 applied to a candidate pool. Training changes one thing about it. The policy’s pass rate is no longer a fixed property of the task, so the share of rewards that go to failures moves as the policy learns, and it is largest when true task success is rare. The symbols below map onto those of Grader error rates. Let \(\ReferenceAssessedFailure\) be criterion failure under an independent reference assessment and \(\VerifierVerdict^+\) acceptance by the training verifier (the event written \(\text{Acc}\) there). Let \(\ValidCandidateRate\) be the policy’s current probability that a rollout satisfies the criterion, \(\VerifierFalseAcceptanceRate=\Pr(\VerifierVerdict^+\mid \ReferenceAssessedFailure)\) the verifier’s false-acceptance rate, and \(\VerifierTrueAcceptanceRate\) its probability of accepting a criterion success. Provided verifier acceptance occurs with nonzero probability, the failure rate among accepted rollouts is
The rarer true passes are, the more of the reward a small verifier error claims.
Equation 1 defines the false-accept share of the reward: the fraction of all rewarded rollouts that actually fail the task criterion. Because its denominator sums both true positive accepts (\(\ValidCandidateRate \VerifierTrueAcceptanceRate\)) and false positive accepts (\((1-\ValidCandidateRate)\VerifierFalseAcceptanceRate\)), a small false-positive rate \(\VerifierFalseAcceptanceRate\) yields a large false share whenever failures are common. In reinforcement learning on challenging tasks, failure is the default state during early iterations. The same Bayesian arithmetic governs automated tests and learned reward models alike.
Napkin Math 17.1: False accepts in the reward
Problem: A hypothetical batch samples 8 rollouts for each of 1,024 tasks, giving 8,192 rollouts. Suppose 10 percent satisfy the criterion under the reference assessment. The training verifier accepts every criterion success and 2.5 percent of criterion failures. What share of its positive judgments are false accepts?
Math: For \(\BatchRolloutCount\) rollouts, expected true accepts are \(\BatchRolloutCount \ValidCandidateRate \VerifierTrueAcceptanceRate\) and expected false accepts are \(\BatchRolloutCount(1-\ValidCandidateRate)\VerifierFalseAcceptanceRate\). For this batch these are 819.20 and 184.32, respectively, totaling 1,003.52 expected positive judgments. Applying equation 1 gives 18.4 percent.
Result: 18.4 percent of rewarded rollouts are failures, although the verifier errs on only 2.5 percent of failures. The counts are expectations; an actual batch varies around them.
Systems insight: The false share is set by the ratio \((1-\ValidCandidateRate)\VerifierFalseAcceptanceRate/(\ValidCandidateRate \VerifierTrueAcceptanceRate)\), so it moves with the policy’s pass rate as much as with the verifier. In the small-share regime, halving \(\VerifierFalseAcceptanceRate\) roughly halves the share; doubling \(\ValidCandidateRate\) has approximately the same effect when \(\ValidCandidateRate\) is also small. For a fixed positive false-acceptance rate, the false share approaches one as true passes become rare. Measure verifier error at the pass rate the policy currently has, and expect the share to fall as training succeeds. The share is not the fraction of the gradient that rewards errors, which also depends on where the false accepts fall within their groups (section 17.5.1).
17.3.3 Reward composition
Adding auxiliary reward terms for well-formed tool calls or concise trajectories helps guide exploration, but poorly chosen weights can let a short, well-formatted failure score higher than a long, working repair. Suppose a fully assessed rollout receives an observed binary outcome \(\CorrectnessReward(\Trajectory)\in\{0,1\}\). Let \(\FormatComplianceScore(\Trajectory)\in[0,1]\) measure format compliance and let \(\text{len}(\Trajectory)\in[0,1]\) measure generated-token cost relative to an enforced cap. A common terminal score formulation is
Format compliance encourages well-formed tool calls, and the length penalty favors concise solutions.4 Over their defined ranges, an observed pass earns at least \(1-\LengthPenaltyWeight\), while an observed failure earns at most \(\FormatRewardWeight\). A sufficient condition for every passing rollout to outrank every failing rollout is therefore
4Reward Shaping: Ng et al. (1999) showed that an added shaping reward is guaranteed to leave the optimal policy unchanged when it takes the form of a difference of potentials over states, and that other additions can change which policy is optimal. Format and length terms are not of that form, so they change the objective itself; their weights therefore need the priority bound derived here rather than an assumption that they merely speed learning.
Ng, Andrew Y., Daishi Harada, and Stuart J. Russell. 1999. “Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping.”Proceedings of the Sixteenth International Conference on Machine Learning (ICML), 278–87.
\[
\FormatRewardWeight+\LengthPenaltyWeight<1.
\]
Allowing equality permits a tie between a maximally penalized pass and a perfectly formatted failure. Strict inequality protects the score ordering, and it holds only while the normalized length penalty \(\text{len}(\Trajectory)\) remains strictly bounded in \([0,1]\). Normalizing by a nominal token budget preserves that invariant only if the execution runtime enforces the cap with hard process termination.
When an unexpected infrastructure failure leaves a required check unavailable, the assessment remains incomplete. Under the four-verdict rule of The four-verdict rule its verdict is unknown, and the reward stays unresolved rather than being converted to \(\CorrectnessReward=0\) or \(\CorrectnessReward=1\). Converting an unexecuted check to failure penalizes valid candidate solutions for harness faults, while converting it to success creates an immediate reward vulnerability; section 17.6.3 develops how a training loop must handle such rollouts. An explicit timeout under the task contract represents a valid failure verdict; a missing result due to test runner crashes does not. Similarly, introducing partial credit (awarding reward for the fraction of unit tests passed, for instance) defines a distinct optimization objective that alters which behavioral changes earn reinforcement, requiring its own priority analysis.
Permissions and hard resource limits must remain strict runtime constraints. A negative reward penalty merely discourages proposing a prohibited action; only the execution harness can enforce the boundary, and a penalty cannot undo an executed side effect. Even a reward function that ranks every unsafe trajectory below every safe one leaves an agent free to explore unsafe actions during policy search. The operational rule is to keep authorization, network isolation, and resource quotas firmly anchored in the container harness, using reward terms strictly to express preferences within that bounded perimeter. Even within that perimeter, the terminal reward evaluates the rollout trajectory as a complete artifact; converting that scalar outcome into updates for the individual actions taken along the trajectory introduces the credit-assignment problem.
17.4 Credit Across Turns
With a reward fixed, the credit-assignment problem named in section 17.2.2 becomes concrete. In the connection-pool repair, an agent might run the failing reproducer, read src/pool.py, and correctly identify the missed notification on connection return, only to introduce a syntax error in its final edit. A verifier that evaluates only the end state marks the entire trajectory as failed. That terminal failure characterizes the trajectory as a whole; it does not say that the diagnosis was a mistake.
The estimators that address this problem differ in what they spend to separate the two turns: nothing beyond the sampled return, a trained value model, a denser reward, or additional rollouts. Choosing among them requires keeping two quantities apart, the empirical coefficient applied in a sampled gradient update and the true expected value of an individual action.
17.4.1 The score-function estimator
When an environment emits only a terminal pass or fail at the end of an episode, the simplest gradient estimator attaches that single outcome to every turn along the trajectory. Let \(\TurnHorizon\) denote the finite interaction horizon, \(\RolloutTurnIndex \in \{0, \ldots, \TurnHorizon-1\}\) the decision index, and \(\InteractionHistory_k\) the delivered interaction history up to turn \(\RolloutTurnIndex\). Under an undiscounted objective, let \(\TurnReward_k\) represent the reward received immediately after taking action \(\RolloutAction_k\). The remaining return from step \(\RolloutTurnIndex\) onward, often called the reward-to-go, is defined as \(\RemainingReturn=\sum_{j=k}^{\TurnHorizon-1}\TurnReward_j\). In sparse verifiable environments where only a terminal reward \(\TrajectoryReward\) arrives at the end of the episode, every intermediate turn sees the same reward-to-go: \(\RemainingReturn=\TrajectoryReward\) for all \(\RolloutTurnIndex\).
For a trajectory sampled from the current policy, a likelihood-ratio, or score-function, estimator of the REINFORCE form (Williams 1992) is
Its expectation equals the gradient of expected total reward under standard regularity conditions: the initial task distribution, environment transitions, and reward mechanism do not depend on \(\ModelParameters\); the differentiable policy has fixed support; and differentiating under the expectation integral is mathematically justified (Sutton et al. 1999). The score \(\nabla_\ModelParameters\log\Policy(\RolloutAction_k\mid \InteractionHistory_k)\) is the sensitivity vector of the sampled action’s log-probability with respect to parameters \(\ModelParameters\). Multiplying this vector by the scalar return \(\RemainingReturn-\PolicyGradientBaseline(\InteractionHistory_k)\) produces a step in parameter space rather than an intrinsic score of action quality (Sutton and Barto 2018).
The baseline \(\PolicyGradientBaseline(\InteractionHistory_k)\) can be fixed, task-specific, or predicted by a learned value model. To maintain an unbiased gradient, \(\PolicyGradientBaseline(\InteractionHistory_k)\) must not depend on the current sampled action conditional on \(\InteractionHistory_k\), and its parameters remain fixed when computing the policy gradient. Subtraction preserves the expectation because the expected score under the policy evaluates to zero:
A baseline trained on earlier, independent rollouts satisfies this conditional independence property. A baseline computed from the current rollout’s own outcome does not, even if no gradient flows through it, because that outcome depends on the very actions being scored, so the baseline is no longer a constant with respect to them. Group-relative estimators, which compute baselines across multiple rollouts for the same prompt, introduce sample-dependent coefficients that require separate statistical treatment.
Consider the consequences for our connection-pool repair. When an attempt fails (\(\TrajectoryReward=0\)) with a zero baseline, the sampled coefficient \([\RemainingReturn-\PolicyGradientBaseline(\InteractionHistory_k)]\) becomes zero for every turn, yielding zero gradient update. With a positive baseline instead (for instance, an average pass rate of \(0.4\)), the resulting coefficient is negative (\(-0.4\)), penalizing the insightful diagnostic command and the faulty edit alike. The learning algorithm does not assign positive credit to the diagnosis from this single run. Instead, credit for effective diagnostic steps emerges only statistically across many trajectories: actions that systematically lead to successful outcomes appear more frequently in winning runs than in losing ones.
Because every additional interaction turn introduces another score-function term to the summation, the variance of \(\PolicyGradientEstimate\) can grow with the trajectory length \(\TurnHorizon\). Even under a decaying-covariance assumption, the variance bound in Credit assignment variance in trajectory optimization grows linearly with \(\TurnHorizon\), so longer decision horizons can demand substantially more rollouts to extract a clean training signal. Long horizons therefore cost the learner twice. Under the horizon-compounding model (equation 1.1), successful trajectories become exponentially rarer, so fewer rollouts carry a positive signal, and each one that does arrives with a gradient estimate whose variance can grow with \(\TurnHorizon\).
17.4.2 Value, advantage, and process scores
To distinguish an insightful diagnostic command from a later syntax error on a single rollout, the estimator needs a baseline that updates after each turn rather than once per prompt. For the same continuation policy and reward definition, define
The state value \(\PolicyRemainingReturn(\InteractionHistory_k)\) averages over the policy’s next action and subsequent interaction, whereas the action value \(\PolicyHistoryActionReturn(\InteractionHistory_k,\RolloutAction_k)\) conditions explicitly on that chosen action. Their difference, the advantage, measures expected return relative to the policy’s default behavior at that history. Defining these quantities over interaction histories \(\InteractionHistory_k\) rather than abstract environment states grounds them strictly in the observations and context tokens available to the model.
With terminal value \(\PolicyRemainingReturn(\InteractionHistory_{\TurnHorizon})=0\), the sampled temporal-difference residual(Sutton 1988) is
Its conditional expectation given \(\InteractionHistory_k\) and \(\RolloutAction_k\) equals the true advantage when the value function is exact and subsequent observations and rewards follow the target environment distribution. A single observed residual is merely a noisy sample of that expectation, and substituting learned value estimates introduces approximation error on top of execution noise;5 dropping \(\TurnReward_k\) is valid only when that immediate step reward is zero. Generalized advantage estimation and KL divergence develops the discounted and multi-step formulations (Schulman et al. 2016).
5Generalized Advantage Estimation: GAE computes an exponentially weighted average of future residuals parameterized by a decay factor \(\TraceDecay \in [0, 1]\) (distinct from the shaping weights \(\FormatRewardWeight\) and \(\LengthPenaltyWeight\)). At \(\TraceDecay=0\), it uses a single one-step residual, which minimizes variance but incurs substantial bias if the value model contains errors. At \(\TraceDecay=1\), it recovers the empirical return minus the baseline, which is unbiased with respect to the value model but inherits all the variance of subsequent interaction turns (Schulman et al. 2016). For a multi-turn agent with an imperfect critic, \(\TraceDecay\) governs how much credit relies on neural value predictions rather than verified environment execution.
Schulman, John, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. 2016. “High-Dimensional Continuous Control Using Generalized Advantage Estimation.”International Conference on Learning Representations (ICLR).
Estimating an existing return differs fundamentally from injecting new reward terms into the training objective. A learned critic acts as an estimator: it approximates the expected continuation return under the current policy without altering the underlying task specification. A process reward model (PRM), by contrast, scores intermediate reasoning steps or tool invocations directly, augmenting the primary objective with synthetic feedback.
While dense process rewards can guide search during inference (Lightman et al. 2024; Uesato et al. 2022), training against them directly creates strong incentives for reward hacking. Because the policy generates the very tokens that the process verifier evaluates, systems engineers encounter three common failure modes:
Superficial compliance: The agent generates formulaic boilerplate or ritualistic assertions (such as commenting “carefully verifying pointer bounds before dereferencing”) that satisfy the process model without performing any actual verification.
Locally consistent hallucinations: The agent constructs fluent, step-by-step reasoning that appears rigorous to a local evaluator, even though the derivation rests on an unvalidated or false premise.
Pathological conservatism: The agent avoids exploratory detours, speculative rollouts, or diagnostic failures that temporarily depress step scores, even when those intermediate investigations are necessary to diagnose complex software bugs.
Dense feedback reduces credit variance, but denser feedback is not necessarily correct feedback. Modifying the reward function changes what the policy optimizes, whereas value estimation simply estimates the performance of the policy under the true task metric. Table 2 separates these design choices and their architectural commitments.
Table 2: Reward Definition and Return Estimation: These choices can be combined. Predicting or sampling a declared return does not independently validate the check that defines it.
Choice
Quantity supplied
Main obligation
Terminal check
Observed trajectory outcome
Validate criterion, coverage, and protected assessment
Learned value estimate
Predicted return under a specified policy
Assess prediction error and estimator bias
Process reward
Additional intermediate scoring criterion
Justify the changed objective and assess its errors
Continuation sampling
Empirical return at a declared fork condition
Restore compatible conditions and account for sampling uncertainty and cost
17.4.3 Monte Carlo value estimates
Training a neural value critic on sparse multi-turn trajectories often fails when small prediction errors across tens of turns overwhelm the temporal-difference signal, while process reward models invite the policy to game intermediate step scores. Continuation sampling avoids both learned models by estimating state value directly from the environment: it rolls out \(\ContinuationCount\) independent trajectories (in this section \(\ContinuationCount\) counts continuations, not output tokens) from an intermediate execution boundary under a fixed continuation policy and averages their terminal returns. To accurately estimate \(\PolicyRemainingReturn(\InteractionHistory)\), the starting state of these continuations must match the true conditional environment distribution given history \(\InteractionHistory\). In interactive environments with latent state (such as unobserved database locks, network traffic, or non-deterministic operating system processes), restoring a single concrete container snapshot conditions on that specific state rather than integrating over all hidden states consistent with \(\InteractionHistory\). A sound fork must therefore explicitly reconstruct the filesystem workspace, container network namespace, delivered context tokens, external mock services, continuation policy version, and verification oracle. Copying sandbox memory bytes or disk blocks alone does not guarantee environmental fidelity.
When individual continuations yield independent bounded returns \(\TrajectoryReward^{(\ContinuationIndex)} \in [0, 1]\), the sample mean \(\EstimatedRemainingReturn = \ContinuationCount^{-1}\sum_{i=1}^\ContinuationCount \TrajectoryReward^{(\ContinuationIndex)}\) serves as an unbiased estimator of the expected continuation return. Because Bernoulli random variables achieve maximum variance at \(\AttemptPassProbability=0.5\) (\(\ContinuationReturnStandardDeviation^2 \le 1/4\)), the estimator’s variance is bounded above by \(1/(4\ContinuationCount)\), corresponding to a standard error of at most \(1/(2\sqrt{\ContinuationCount})\). Halving this uncertainty bound requires a fourfold increase in sampled rollouts. This error bound holds strictly for independently sampled branches; if rollouts share cached environment state or common prefixes, cross-sample correlation increases the variance. Furthermore, estimating action values \(\PolicyHistoryActionReturn(\InteractionHistory, \RolloutAction)\) requires conditioning explicitly on the chosen action \(\RolloutAction\) before rolling out the continuation policy. Consequently, sampling values at selected turn boundaries estimates the net value delta across blocks of turns rather than isolating individual token decisions.
A hypothetical fork makes this estimate concrete. Out of 20 continuations launched from the initial bug report, 1 passes, giving a sampled value of 0.05. In the primary trajectory, the policy then runs a debugger, producing an observation that contains the failing line. Forking 20 continuations from this updated history yields 12 passes, corresponding to a sampled value of 0.60. The estimated value difference for the debugger turn is 0.55, even if the primary trajectory later fails on a subsequent edit. Each sample mean carries a standard error of at most 0.11, and because the two fork estimates are independent their variances add, so the difference has a standard error of at most \(\sqrt2/(2\sqrt \ContinuationCount)\), or 0.16. Treating a difference as resolved only when it is at least twice that bound, here 0.32, the observed 0.55 clears the threshold, so the gain is unlikely to be sampling noise. Resolution against noise does not make \(\FiniteChange \EstimatedRemainingReturn\) an exact advantage, however; it remains an empirical estimate at two particular fork conditions. Of the three ways to connect the terminal failure to earlier turns (figure 1), only the forks separate the debugger turn from the faulty edit on this trajectory, and they do it by paying for extra rollouts rather than by adding a criterion the policy could learn to satisfy.
Figure 1: Three Ways to Connect a Terminal Outcome to Earlier Turns: The same trajectory runs a debugger, makes a faulty edit, and fails the terminal check. (a) With only the terminal check, every remaining return equals the one outcome, so the check alone does not separate the useful turn from the faulty one. (b) Learned step scores supply intermediate feedback by adding a criterion to the objective, one the policy can learn to satisfy without progress on the task. (c) Continuation sampling restores each fork condition and averages the returns of its continuations; the difference between the two means estimates how much the debugger turn raised the chance of passing the declared check, with sampling error and without establishing cause.
These continuation rollouts are expensive. Every branch consumes inference tokens, host memory, and sandbox execution time, and the total depends directly on how many intermediate boundaries the fork schedule samples, which makes the schedule the real design choice.
Napkin Math 17.2: Continuation sampling in tokens
Problem: An agent trajectory runs 30 turns of 512 generated tokens each. At each chosen intermediate boundary, the system samples 8 continuations, each running for the remainder of the episode. What generation cost results from sampling every intermediate boundary, compared with sampling only after turns 10 and 20?
Variables: \(\TurnHorizon\) is the trajectory horizon in turns, \(\TurnGeneratedTokenCount\) the generated tokens per turn, \(\ContinuationCount\) the continuations per boundary, and \(\ForkBoundarySet\subseteq\{1,\ldots,\TurnHorizon-1\}\) the index set of completed-turn fork points. Branches are nonrecursive.
When forking at every intermediate boundary, the remaining-turn sum is \(\TurnHorizon(\TurnHorizon-1)/2\), yielding \(\ForkedTrajectoryTokenCount/\BaseTrajectoryTokenCount=1+\ContinuationCount(\TurnHorizon-1)/2\).
Result: The base trajectory generates 15,360 tokens. The multiple-of-base column in table 3 prices the gap: exhaustive intermediate sampling creates severe computational blowup compared with selective boundary evaluation.
Table 3: Generated-Token Inventory for Two Fork Schedules: The selected schedule estimates fewer intermediate conditions; the rows are not equal-accuracy alternatives.
Fork schedule
Continuation branches
Total generated tokens, including base
Multiple of base
Every intermediate boundary
232
1,797,120
117
After turns 10 and 20
16
138,240
9
Systems insight: Exhaustive forking grows quadratically as \(\ContinuationCount \TurnHorizon^2\,\TurnGeneratedTokenCount/2\), meaning that doubling the horizon quadruples the credit computation bill even though base trajectory generation only doubles. At this horizon of 30 turns, exhaustive forking multiplies token generation by 117. In contrast, sampling only two targeted boundaries costs 9 times the base trajectory. The engineering decision rule is to fork only at a handful of critical boundaries, spending the saved generation budget on exploring a broader diversity of tasks. This inventory tallies generated output tokens alone; context prompt processing, sandbox snapshot restoration, tool execution, and verification add overhead to both configurations, while variable-length continuations shift the exact totals.
Selective forking requires an effective policy for choosing boundaries. Two triggers suit this choice: turn boundaries where the policy’s action distribution has high token entropy (a next-token distribution spread across many candidates, measured in section 17.8.1), indicating a decision among genuine alternatives, and turns that execute state-modifying tool calls such as file edits, database migrations, or network reconfiguration. Forking at two or three such junctures places the empirical credit estimates where trajectories actually diverge, at a small fraction of the exhaustive cost.
Credit assignment is a budget decision that trades computational cost against statistical fidelity. Fixed baselines incur zero token overhead but offer no step-level resolution; learned value functions reduce variance at the expense of training a secondary critic; and continuation rollouts provide direct empirical value estimates at the price of substantial token generation. Group-relative policy optimization represents the minimalist end of this spectrum, comparing complete sampled returns within a task group without spending any additional tokens on intermediate branches.
Checkpoint 17.1: Rewards and credit
A reward is a measurement, and credit is an estimate built on it. Each has a price in errors or tokens.
17.5 Group Relative Policy Optimization
Suppose three fresh rollouts attempt the connection-pool repair under the same fixture, budget, and assessment rule. Their completed terminal rewards are \([0,1,0]\). The successful rollout provides a direct comparison for the two failures without requiring an expert to label each intermediate edit, though this outcome comparison alone cannot identify which specific action caused success. Group relative policy optimization (GRPO) uses such within-task comparisons6 to construct policy update coefficients, replacing a learned value baseline with groups of sampled outcomes (Shao et al. 2024).
6Sampled Baselines: Scoring each sample against the other samples for the same input predates GRPO. Kool et al. (2019) drew several samples per input and used the others as each one’s baseline, obtaining variance reduction without a learned critic, and Ahmadian et al. (2024) found that such REINFORCE-style estimators could outperform PPO when fine-tuning language models from human feedback. GRPO adds division by the group’s reward dispersion, a weighting choice analyzed in section 17.5.3.
Kool, Wouter, Herke van Hoof, and Max Welling. 2019. Buy 4 REINFORCE Samples, Get a Baseline for Free!
Ahmadian, Arash, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. “Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs.”Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12248–67. https://doi.org/10.18653/v1/2024.acl-long.662.
The trade-off centers on baseline estimation rather than the reward source: proximal policy optimization (PPO) can operate with an external check, just as GRPO can evaluate rollouts against a learned reward model. PPO trains a value function, either sharing parameters with the policy or running in a separate critic network (Schulman et al. 2017). GRPO eliminates that critic alongside its memory and training overhead, paying for the omission with additional rollouts: it samples a group of \(\GroupSize\) attempts per task to estimate empirical group statistics. GRPO serves as our worked optimizer because its systems accounting is straightforward to track, though the execution isolation and trajectory recording requirements that follow apply equally to both approaches. Both optimizers also hold a reference policy, a frozen copy of the starting checkpoint used to penalize drift away from it (section 17.5.2). Table 4 sets the two side by side.
Table 4: PPO and GRPO Compared: The two optimizers differ in the baseline they use, the granularity of their credit, and how they fail. Neither column implies a universal reduction in total memory or compute.
Dimension
PPO
GRPO
Models held
Policy, value function, reference, and possibly a reward model
Policy and reference; the reward can be an external check
Trained components
Policy plus a value function, shared or separate
Policy only
Baseline
Learned value estimate \(\CriticReturnEstimate(\InteractionHistory_k)\)
Mean reward of the group for the same task
Credit granularity
Per token or turn, through the value estimate
Per trajectory, shared by all its sampled tokens
Rollouts per task
Can be one
A group of \(\GroupSize\)
Characteristic failure
Value error on long, sparse-reward trajectories
No reward-surrogate gradient when every rollout in a group ties
17.5.1 Group-relative coefficients
Without a trained critic network, a single rollout’s terminal score gives the optimizer no reference level to decide whether the outcome beat expectations for that task. To construct a baseline without a critic, the system collects \(\GroupSize\) completed rollouts for the same task from a fixed behavior policy\(\pi_{\text{old}}\), the specific model checkpoint, execution environment, and decoding configuration that generated the trajectories. Let \(\TrajectoryReward_i\) denote the terminal reward assigned to rollout \(\GroupRolloutIndex\). For outcome-only verification, the group statistics and advantage coefficients are:
The positive stabilizer \(\AdvantageStabilizer\) keeps the expression well-defined when all rollouts receive identical rewards. The denominator uses the empirical population standard deviation with divisor \(\GroupSize\), rather than a Bessel-corrected sample standard deviation with divisor \(\GroupSize-1\). This convention is part of the objective: altering the divisor rescales every coefficient, and neither divisor accounts for unobserved task behaviors outside the sampled group. During policy updates, the rewards \(\TrajectoryReward_i\) and coefficients \(\GroupRelativeAdvantage_i\) remain fixed constants; because this scalar is easily confused with the state-action advantage function of section 17.4, the next definition keeps the two apart.
Definition 17.3: Group-relative advantage
Group-relative advantage is the coefficient \(\GroupRelativeAdvantage_i\) that GRPO attaches to every sampled token of rollout \(\GroupRolloutIndex\): the rollout’s reward minus the mean reward of its task group, divided by the group’s reward dispersion.
Significance: It replaces a learned value function with \(\GroupSize\) rollouts per task. The group size therefore governs both the accuracy of the empirical baseline and the rollout compute budget; on a binary check, a lone pass in a group of \(\GroupSize\) receives a coefficient approaching \(\sqrt{\GroupSize-1}\).
Distinction: The advantage \(\PolicyHistoryAdvantage=\PolicyHistoryActionReturn-\PolicyRemainingReturn\) is an expectation for one action at one history, whereas the group-relative coefficient is an empirical, standardized score broadcast uniformly across all tokens in an entire trajectory. Furthermore, because the group statistics include the rollout being scored, the action-independent baseline argument does not make the standardized estimator unbiased REINFORCE. Centering alone is harmless, since \(\TrajectoryReward_i-\GroupMeanReward\) equals \((\GroupSize-1)/\GroupSize\) times \(\TrajectoryReward_i\) minus the mean of the other \(\GroupSize-1\) rewards, a valid leave-one-out baseline rescaled; the departure comes from dividing by \(\GroupRewardStandardDeviation\), which every reward in the group helps set, and it is a weighting choice recorded with the run rather than an error to be corrected.
Common pitfall: Reading a large positive coefficient as evidence that the rollout was fundamentally sound. The coefficient measures only how the rollout’s reward compared with its group; a false pass receives the exact same coefficient as a genuine success.
Napkin Math 17.3: From three rewards to update coefficients
Problem: Three hypothetical repair rollouts receive terminal rewards \([0,1,0]\), with no intermediate rewards and \(\AdvantageStabilizer=10^{-8}\). What coefficients enter the outcome-only GRPO update?
Math: The mean is \(\GroupMeanReward=1/3\), or 0.333333. The squared deviations are \(1/9\), \(4/9\), and \(1/9\), so \(\GroupRewardStandardDeviation=\sqrt{2/9}\), or 0.471405. Dividing each centered reward by \(\GroupRewardStandardDeviation\) gives \((-1/3)/\sqrt{2/9}=-1/\sqrt 2\) for each failure and \((2/3)/\sqrt{2/9}=\sqrt 2\) for the pass.
Result: Each failed rollout has coefficient -0.707107; the successful rollout has coefficient 1.414214. The three coefficients sum to zero.
Systems insight: The update sign emerges from relative group ranking rather than absolute reward values, and its magnitude scales up as successful trajectories become rarer. However, because every generated token within a rollout receives the identical scalar coefficient, crediting insightful diagnostic steps and mistaken proposals alike, GRPO trades fine-grained temporal credit assignment for substantial algorithmic and computational simplicity.
The rarer the pass in its group, the larger the coefficient it receives.
The group mean acts as an adaptive per-task baseline that absorbs intrinsic task difficulty. When a binary-evaluated group of size \(\GroupSize>1\) produces exactly one pass, the group mean is \(1/\GroupSize\) and the empirical standard deviation is \(\sqrt{\GroupSize-1}/\GroupSize\). Under equation 2, that lone successful rollout receives an advantage coefficient of
which evaluates to \(\sqrt{\GroupSize-1}\) when the stabilizer is zero and remains slightly below that bound for positive \(\AdvantageStabilizer\). On a difficult task where only one rollout in 16 succeeds, the group baseline drops to 0.0625; the solitary success receives an amplified coefficient of 3.87, while the fifteen failures receive small negative coefficients of 0.26 in magnitude. Conversely, on an easy task where only one rollout in 16 fails, the signs invert, heavily penalizing the isolated failure. In both scenarios, the update amplifies the contrast that separated success from failure on that specific problem instance. The group mean thereby fills the baseline role that a learned value function performs in PPO.
The same arithmetic explains why the false-accept share of equation 1 is not the false share of the gradient. False accepts occur on failing rollouts, which are most common on hard tasks, and on a hard task a false pass is often the group’s only pass, so it receives a coefficient near \(\sqrt{\GroupSize-1}\). A false pass on an easy task, one of many passes in its group, receives only a small positive coefficient, the mirror image of the small failure coefficients above. The same count of false accepts can therefore carry a larger or a smaller share of the update depending on which groups they land in, and the hardest tasks, where the share of equation 1 is also highest, are where each one weighs most.
17.5.2 Clipped surrogate objective
Taking multiple gradient steps on a collected batch of rollouts causes the updated policy \(\Policy\) to drift away from the behavior policy \(\pi_{\text{old}}\) that sampled the actions, and an unclipped importance ratio can trigger destructive weight updates. Let \(\SampledPolicyToken_{i,t}\) denote the \(\TokenIndex\)-th sampled policy token in rollout \(\GroupRolloutIndex\), \(\ConditioningTokenPrefix_{i,t}\) its delivered context prefix, and \(\RolloutPolicyTokenCount_i>0\) the total number of sampled generation tokens. While tool outputs, environment responses, and prompt text appear within prefixes \(\ConditioningTokenPrefix_{i,t}\), they are masked out of the policy loss and do not count toward \(\RolloutPolicyTokenCount_i\). At each generated token position, the probability ratio is:
Both probabilities must reflect the exact sampling convention used during generation. If generation applied temperature scaling or top-\(\Glyph{k}\) / top-\(\Glyph{p}\) truncation, the recorded behavior probability \(\pi_{\text{old}}\) in the denominator must match the actual sampling distribution and remain strictly positive. Probability support eliminated by decoding filters cannot be restored by downstream loss ratios.
With clipping parameter \(0<\RatioClippingWidth<1\) (kept distinct from the denominator stabilizer \(\AdvantageStabilizer\)), the clipped surrogate objective is:
When \(\GroupRelativeAdvantage_i>0\), the surrogate stops growing once the probability ratio exceeds \(1+\RatioClippingWidth\), removing any gradient incentive to push the current probability higher. When \(\GroupRelativeAdvantage_i<0\), the surrogate flattens once the ratio drops below \(1-\RatioClippingWidth\), avoiding destructive runaway penalties. In contrast, movements in an unfavorable direction, such as reducing the probability of an action with positive advantage, remain unclipped and continue to penalize the surrogate. Clipping caps the gradient incentive on individual tokens rather than creating a true trust region that bounds overall policy shift; parameter sharing across tokens and additional loss terms can still propel \(\PolicyProbabilityRatio_{i,t}(\ModelParameters)\) well outside the clipping interval.7
7Trust Region: The clipped surrogate originated in trust region policy optimization, which explicitly constrained each policy step to keep the average KL divergence between updated and behavior policies below a target threshold (Schulman et al. 2015). PPO replaced this constrained optimization step with a clipped objective to simplify implementation and distributed scaling (Schulman et al. 2017). Because clipping provides no mathematical guarantee against large policy shifts, production training pipelines must monitor probability ratios and empirical divergence metrics directly.
Schulman, John, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. “Trust Region Policy Optimization.”Proceedings of the 32nd International Conference on Machine Learning (ICML) 37: 1889–97.
Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. “Proximal Policy Optimization Algorithms.”arXiv Preprint arXiv:1707.06347, ahead of print. https://doi.org/10.48550/arXiv.1707.06347.
A frozen reference policy\(\pi_{\text{ref}}\), typically initialized from the pre-RL checkpoint, provides an auxiliary regularizer that penalizes excessive policy drift. In distributed pipelines, \(\pi_{\text{ref}}\) maintains a distinct model identity from the behavior policy \(\pi_{\text{old}}\) that generated the rollouts. Assuming strictly positive probabilities over shared token support, we define:
The penalty term \(\PerTokenKlEstimate_{i,t}(\ModelParameters)\) is strictly non-negative and vanishes if and only if the current and reference token probabilities coincide. Conditioned on prefix \(\ConditioningTokenPrefix_{i,t}\), the expectation of \(\PerTokenKlEstimate_{i,t}(\ModelParameters)\) over tokens sampled from the current policy \(\Policy\) equals the Kullback–Leibler divergence \(\KlDivergence(\Policy\parallel\pi_{\text{ref}})\), the expected log ratio of the two distributions under \(\Policy\), which is zero only when they agree; the identity is derived in Generalized advantage estimation and KL divergence. The plain log ratio \(-\log \ReferenceProbabilityRatio_{i,t}\) has the same expectation, but a single sampled token can make it negative, so a per-token penalty built on it would sometimes reward drift. Each term \(\ReferenceProbabilityRatio-\log \ReferenceProbabilityRatio-1\) is non-negative, so every token’s penalty pushes toward the reference. Because the tokens in the training batch were originally drawn from the behavior policy \(\pi_{\text{old}}\), evaluating this sample average provides an approximation to the current-policy divergence that degrades as off-policy lag accumulates over multiple gradient steps.
Combining these components yields the empirical GRPO maximization objective:
Standard deep learning frameworks minimize the negative of \(\GrpoSampledObjective(\ModelParameters)\) as a loss. The hyperparameter \(\KlPenaltyCoefficient\) scales the reference policy penalty. Because reward standardization, probability ratio clipping, and token length normalization collectively depart from the true expected return \(\ExpectedReturnObjective(\ModelParameters)\), production systems record these optimization hyperparameter choices alongside the trained model weights and validate performance through fresh online evaluation trials.
17.5.3 Length normalization
Averaging token losses within each rollout by \(1/\RolloutPolicyTokenCount_i\) in equation 4 penalizes each token of a long failure far less than each token of a short failure, teaching the policy to stretch out failing attempts. At the start of an update, when the current policy matches the behavior policy and the probability ratios equal one, rollout \(\GroupRolloutIndex\) contributes the following gradient to the reward surrogate:
initially assigning every sampled token in trajectory \(\GroupRolloutIndex\) an effective scalar weight of \(\GroupRelativeAdvantage_i/\RolloutPolicyTokenCount_i\). This per-sequence divisor exerts asymmetric optimization pressure across different outcome classes (Liu et al. 2025). For failing rollouts (\(\GroupRelativeAdvantage_i<0\)), extending generation length distributes the negative penalty across more tokens, diluting the per-token gradient penalty. For successful rollouts (\(\GroupRelativeAdvantage_i>0\)), producing fewer tokens concentrates the positive advantage on a compact sequence, amplifying the per-token reinforcement. The resulting optimization pressure, known as length bias, systematically favors concise successes and verbose failures.
Liu, Zichen, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. “Understanding R1-Zero-Like Training: A Critical Perspective.”arXiv Preprint arXiv:2503.20783.
Consider two failing rollouts within the same group that receive identical negative coefficients \(\GroupRelativeAdvantage_i\). If one fails after 128 tokens while the second exhausts the context budget at 4,096 tokens, the concise failure absorbs a per-token penalty 32× larger than that of the verbose failure. To eliminate this distortion, alternative implementations replace individual sequence lengths \(\RolloutPolicyTokenCount_i\) with a constant divisor shared across the group or microbatch, such as the mean trajectory length \(\MeanRolloutLength\):
Under this constant-denominator objective, every token receives an update weight proportional to \(\GroupRelativeAdvantage_i\) alone, regardless of trajectory length. While these scalar weight comparisons illustrate the gradient mechanics, the actual trajectory length drift depends on how individual token gradient vectors align or cancel across sequences during training, an empirical dynamic examined in section 17.8.
Reward standardization introduces an analogous weighting trade-off across tasks. Dividing by \(\GroupRewardStandardDeviation+\AdvantageStabilizer\) normalizes the scale of centered rewards across groups. As \(\AdvantageStabilizer\to0\), every group without a tie has the same total squared coefficient, \(\sum_i\GroupRelativeAdvantage_i^2=\GroupSize\), whether it holds a single isolated success or an even split between passes and failures. The equality is between whole groups, not individual rollouts. Within the fixed total, a lone pass takes \(\GroupSize-1\) of the \(\GroupSize\) units and its failures share the remaining one, which is the amplification of equation 3, whereas an even split spreads the total evenly. Omitting this dispersion divisor weights tasks strictly by their raw reward spread. Both normalizations represent fundamental objective design choices that must be explicitly logged with the training run alongside the clipping threshold.
17.5.4 Zero-contrast groups
When every rollout in a completed group yields the identical reward, the batch constitutes a zero-contrast group. In this scenario, every numerator in equation 2 evaluates to zero, yielding advantage coefficients \(\GroupRelativeAdvantage_i=0\), surrogate terms \(\ClippedSurrogateObjective_{i,t}=0\), and policy gradient contributions \(\nabla_\ModelParameters\ClippedSurrogateObjective_{i,t}=0\). The reward-driven component of the policy gradient vanishes entirely, leaving only the reference policy penalty and any auxiliary regularizers active. Identical rewards do not imply identical behavior, because different reasoning paths or tool failures collapse into the same uninformative binary score.
Assuming independent rollout attempts on a task with fixed success probability \(\AttemptPassProbability\), a group of size \(\GroupSize\) contains at least one success and at least one failure with probability \(\GroupContrastProbability(\GroupSize,\AttemptPassProbability)=1-\AttemptPassProbability^\GroupSize-(1-\AttemptPassProbability)^\GroupSize\). This matches the informative-task fraction defined in equation 15.3 with \(\IndependentAttemptCount=\GroupSize\), where the excluded outcomes represent the all-pass and all-fail edge cases. (This contrast probability differs from pass@\(\GroupSize\), which requires only a single pass.)
Computational waste is greatest in the difficult task regimes where the learning signal is needed most. Sampling groups of 2 on a task with pass rate \(\AttemptPassProbability=\) 0.10 yields a contrast probability \(\GroupContrastProbability\) of only 0.18, leaving 82 percent of generated groups without any policy gradient signal. Even with larger groups of 8, a task solved on 5 percent of attempts produces zero reward gradient 66 percent of the time. Every uninformative group incurs the full systems cost of model inference, containerized tool execution, and reward verification. This binomial model assumes independent trials on a homogeneous task; evaluating a heterogeneous workload requires computing expected contrast per difficulty bucket, as aggregate fleet averages obscure the true zero-contrast rate.
Groups of eight carry a gradient reliably only at mid-range pass rates.
Because binary outcome variance \(\AttemptPassProbability(1-\AttemptPassProbability)\) peaks at \(\AttemptPassProbability=0.5\), active task scheduling based on rolling pass rates maximizes the proportion of informative rollouts. Systems engineers can counter zero-contrast waste using two levers: increasing group size \(\GroupSize\) or filtering task queues by difficulty. The two levers carry different systems costs. Comparing them mirrors the cost-per-accepted-task framework introduced in equation 1.2: the total computational investment per task attempt divided by the probability of producing usable gradient information.
Napkin Math 17.4: Rollouts per informative group
Problem: A hypothetical update samples 8 rollouts for each of 512 tasks, with each rollout executing 30 interaction turns generating 512 tokens per turn. Under the baseline task distribution, the agent policy succeeds on 5 percent of attempts. What is the total generation cost per training step in tokens, and which scaling strategy purchases informative gradient signal more efficiently: doubling group size to 16, or filtering the dispatch queue for tasks where the agent succeeds roughly 50 percent of the time?
Math: Generating a single trajectory consumes 30 \(\times\) 512 = 15,360 tokens, bringing the total generation burden across 4,096 rollouts to 62.9 million tokens per step. With binary outcome probability \(\GroupContrastProbability=1-\AttemptPassProbability^\GroupSize-(1-\AttemptPassProbability)^\GroupSize\), the amortized rollout cost per informative group is \(\GroupSize/\GroupContrastProbability\), as detailed in table 5.
Table 5: Cost of Reward Contrast: Doubling the group raises contrast but spends more rollouts per informative group; targeting difficulty raises contrast without more rollouts per group, at the price of changing which tasks the policy trains on.
Lever
Group size
Pass rate
\(\GroupContrastProbability\)
Rollouts per informative group
Current task mix
8
5%
0.34
23.8
Double the group
16
5%
0.56
28.6
Target difficulty
8
50%
0.99
8.1
Result: Curriculum difficulty targeting reduces rollout cost per informative gradient by 2.9× compared to the hard baseline mix, whereas doubling group size actually increases the rollout cost per usable gradient. The metric counts informative groups, not the information inside each one; a group of 16 estimates its baseline more precisely than a group of 8, so the verdict holds where the binding problem is obtaining any contrast at all. Targeting also shifts the task mixture, which the run must log (section 17.9.1).
Systems insight: The metric \(\GroupSize/\GroupContrastProbability\) quantifies the computational cost per informative gradient signal: the numerator represents rollout expenditure per group, while the denominator measures the probability that the group yields usable contrast. In regimes with low baseline pass rates, \(\GroupContrastProbability\) dominates system efficiency, and it responds far more dramatically to curated curriculum selection than to expanding group size. When obtaining any contrast is the binding constraint and the changed task mixture is acceptable, the demonstrated curriculum purchases informative groups more cheaply than the larger group. Baseline precision, required task coverage, and measured held-out learning progress can justify other group sizes; the run must record the task-selection distribution as well as the chosen group size.
While group-relative comparisons establish advantage coefficients and optimization weights, every term in the objective ultimately resolves to conditional token probabilities. These probabilities remain valid only if the training engine faithfully reconstructs the precise token contexts seen by the model during inference. In multi-turn agentic workflows, model generations interleave with tool executions, environment state mutations, and dynamic context compaction. Executing GRPO correctly in production therefore requires an exact, immutable log of the tokens passed to and sampled from each model invocation, rather than an unversioned end-of-episode transcript.
17.6 Multi-Turn Agentic RL
In multi-turn agentic reinforcement learning, an agent alternates between generating actions (code edits, shell commands, or structured tool invocations) and consuming external feedback from compilers, test runners, or runtime environments. An environment observation conditions the next proposal, but it is never an action sampled from the policy parameters. As episodes unfold, agent harnesses routinely compact, truncate, or summarize earlier turns to stay within the model’s context window. By the end of an extended trial, the final context window rarely matches the prompt prefixes that originally produced the intermediate turns.
To calculate policy gradients or off-policy importance ratios correctly, the training pipeline must preserve two distinct sources of truth: which tokens the policy actually sampled, and what exact prefix conditioned each token at the instant of generation. Reconstructing training pairs from an end-of-episode chat log fails because it evaluates probabilities on counterfactual context that the policy never saw. While we illustrate these mechanics using group-relative policy optimization (GRPO), this accounting discipline applies universally to any RL algorithm that updates parameters from multi-turn interaction rollouts.
17.6.1 Invocation records
Consider a concrete execution: trial rl_trial_01 running against pool_fixture_v1. In turn 1, the model invokes a test suite. The harness runs the suite and delivers a truncated failure log. In turn 2, the model conditions on that log and proposes a targeted source patch. By turn 4, context window pressure forces the harness to compress the earlier diagnostic logs into a concise two-line summary. If the training pipeline subsequently attempts to reconstruct the rollout from the final episode transcript, it confronts a corrupted causal graph: turn 2 would appear conditioned on the two-line summary rather than the full failure log that actually prompted the patch. Evaluating the model’s action probabilities against that retrofitted context evaluates policy decisions against information states the agent never encountered during sampling.
To guarantee replay fidelity, the rollout recorder must capture each invocation’s delivered input token prefix alongside the exact token IDs emitted by the generator, linking both to a durable interaction record. Storing full token arrays for every step across millions of rollouts would quickly exhaust cluster storage, so systems deduplicate shared prefixes with immutable content-addressed blobs or prefix-tree references. Along with those prefix references, the record registers the tokenizer version, chat template, attention masks, rotary position offsets, and compaction decisions that shaped the prompt. Storing raw UTF-8 text and re-tokenizing it during training cannot substitute for the recorded token IDs, because small differences in BPE merge rules, whitespace handling, or tool-schema formatting shift token boundaries and sequence lengths, which means only the captured IDs preserve the trajectory that was actually sampled. Table 6 extends the input-fidelity principles established in Serializing Trajectories to the sampling evidence required by multi-turn reinforcement learning.
Table 6: Multi-Turn Rollout Record: Decision inputs, sampled actions, and assessment status are separate evidence. A final transcript or a model-version label alone cannot reconstruct their relationship.
Record component
Contents
Training question it answers
Trial and group identity
Task, fixture and candidate versions; rollout and group IDs; declared initial conditions and split assignment
Which experiment and reward comparison does this sample belong to?
Invocation inputs
Call ID, delivered prefix token IDs or available immutable contents, template/tokenizer versions, attention and position configuration
What could the policy condition on at this decision?
Sampled actions
Original output token IDs, invocation boundaries, mapping from local positions to the rollout’s policy-token index
Which tokens were actually sampled, rather than supplied or reconstructed?
Behavior distribution
Weights/adapters version, sampling configuration and constraints, chosen-token behavior log probabilities
What distribution generated each sampled token?
Observation associations
Tool operation and attempt IDs, delivered result versions, refusals, partial-result status, and input inclusion
Which execution evidence reached which invocation?
Bounds and ownership
Resource limits and usage, stop reason, outstanding work and its owner
Why did collection stop, and what duties remain?
Assessment and learning use
Grader/reward versions, component scores, pending or unavailable findings, group membership and update eligibility
Is a reward known, and how does the declared objective use this record?
17.6.2 Policy actions and conditioning context
Because context compaction and truncation alter the prompt between turns, slicing the final transcript cannot recover the prefix \(\ConditioningTokenPrefix_{i,t}\) that conditioned turn \(\RolloutTurnIndex\), and training on those retrofitted slices scores tokens against contexts the policy never saw. To connect multi-turn trajectories to the mathematical objectives developed earlier, we map the episode into the flattened policy-token index \(\TokenIndex=1,\ldots,\RolloutPolicyTokenCount_i\) from section 17.5.2 for rollout \(\GroupRolloutIndex\), while retaining the structural mapping to invocation \(\RolloutTurnIndex\) and the local generation offset. Each conditioning prefix \(\ConditioningTokenPrefix_{i,t}\) consists of the exact input context delivered to invocation \(\RolloutTurnIndex\) concatenated with the preceding policy tokens generated within that invocation.
When computing action-scoring losses, the distinction between conditioning and generation is absolute. Prompts, environment feedback, tool payloads, and harness scaffolding act exclusively as conditioning context. They propagate activations through the model to shape downstream attention and predictions, but they are not sampled policy actions; their token log probabilities must never appear in the loss objective. Consequently, loss normalization denominators must count only the scored policy tokens, while the runtime cost accounting tallies prefill processing and decode generation as separate operational line items.
This action-scoring mask serves a fundamentally different purpose from the imitation masks used in supervised fine-tuning. In the supervised recovery example, the malformed proposal e2 and its refusal stay in the prefix as context but carry a zero loss mask, so they are never imitation targets, because the objective is imitation of curated behavior (Observation Loss Masking). In reinforcement learning, however, the mask identifies the support of the exploration distribution, and verified trajectory post-training (principle 13) admits every sampled action, weighted by its verified outcome. If the policy samples an invalid JSON schema or an unparsable shell command during exploration, that sequence is an authentic policy action, even if the execution harness subsequently refuses to dispatch it. Masking those tokens out simply because a curator would not present them as demonstrations corrupts the estimator. The malformed tokens escape the negative update, and the failing rollout’s whole negative coefficient lands on its remaining well-formed tokens, so the update pushes down valid syntax in place of the mistake. Similarly, when the model generates an end-of-turn delimiter or stop token, that token is an action that consumed probability mass; by contrast, an abrupt terminator injected by a dataloader when cutting off an execution is harness intervention, not policy choice, and must be excluded. Any partial-action scoring strategy requires an explicit mathematical objective and denominator rather than an ad hoc mask inherited from a supervised training pipeline.
The denominator of the importance-sampling probability ratio must represent the true behavior distribution that generated the token. Inference engines often apply sampling transforms that renormalize the distribution over a restricted subset of the vocabulary, such as temperature scaling, top-\(\Glyph{k}\) truncation, nucleus (top-\(\Glyph{p}\)) filtering, or grammar-constrained finite-state decoding. The rollout record must therefore store the normalized chosen-token log probability under the active generation configuration, along with the sampling parameters required to reconstruct the sampling state. When computing policy updates, the trainer evaluates the numerator under the target policy \(\Policy\) using the identical prefix recorded during generation. Substituting unconstrained model logits into the denominator treats the behavior policy as an unconstrained softmax; while some systems adopt this simplification to avoid caching rollout probabilities, it introduces estimator bias that must be tracked explicitly as an approximation.
Freezing the policy checkpoint across each episode substantially simplifies this accounting. In high-throughput asynchronous architectures where weight updates occur concurrently with rollout generation, an agent might execute invocation 1 on checkpoint \(\PolicyCheckpoint_1\) and invocation 2 on checkpoint \(\PolicyCheckpoint_2\). If an infrastructure design permits cross-invocation updates within a single trial, the record must associate behavior checkpoints and generation probabilities per invocation; tagging the entire rollout with a single model version misattributes the provenance of later tokens.
Once token log probabilities and behavior distributions are properly recorded, the system links the trajectory-level advantage coefficient \(\GroupRelativeAdvantage_i\) to each sampled policy token. If the system assigns advantages at the turn level rather than across the entire trajectory, it must adhere to the estimator and continuation rules detailed in section 17.4. Two parallel rollouts that happen to reach turn 3 may possess radically different interaction histories, execution latencies, and tool states; comparing intermediate outcomes purely by matching turn indices provides an invalid foundation for relative reward scoring.
17.6.3 Stopping and censored rollouts
A resilient execution harness enforces strict boundaries on wall-clock time, invocation turns, and token counts. How the learning algorithm interprets an enforced stop, however, depends entirely on the task specification. If a task contract defines success as accepted code completion within a declared budget of 10 turns and 5 minutes, exhausting that budget without reaching a solution constitutes an authentic task failure; the policy legitimately receives a zero or negative reward. The rollout record must log the exact termination reason and resource metrics, noting whether the boundary interrupted active model generation or an external tool execution.
Other interruptions stem from external operational limits rather than policy choices. For example, a single tool invocation exceeding a 30 s sandbox deadline produces an error observation from which the agent can still recover. Alternatively, an assessment microservice may crash, leaving the verifier unable to score an otherwise complete pull request. If an orchestration worker aborts an active rollout simply because the cluster scheduler reclaims the node or the batch queue deadline expires before task budget exhaustion, the collector produces a censored rollout. The rollout record must treat budget exhaustion, observation recovery, verifier unavailability, and infrastructure preemption as mutually exclusive execution states. They map onto the four-verdict rule of The four-verdict rule. Budget exhaustion is a fail; a recovered tool error is an ordinary observation inside the trajectory; verifier unavailability and preemption leave the outcome pending while a retry is still possible and unknown once it is not. Throughout this chapter, a censored or unresolved rollout means one in the last two states, whose reward is missing because of the infrastructure rather than the policy.
Mishandling censored rollouts biases the training data in one of two opposite directions. Silently discarding incomplete rollouts introduces classic survivorship bias, purging complex, multi-turn trajectories from the training set and skewing policy optimization toward trivial, fast tasks. Conversely, assigning a default reward of zero to rollouts interrupted by infrastructure outages penalizes the agent for distributed system failures, teaching the policy to avoid long multi-step trajectories. The training pipeline must declare an explicit policy for censored episodes: waiting for asynchronous verifier recovery, scheduling idempotently bounded retries, or applying statistical imputation methods that account for missing outcomes without biasing advantage estimates.
Finally, terminating policy sampling does not automatically stop pending asynchronous operations. Dispatched bash subshells, spawned containers, and cloud API calls remain active until explicitly cancelled. The harness must track resource ownership across the full operational lifecycle, ensuring that abandoned or censored rollouts clean up orphaned processes and container volumes regardless of whether the resulting trajectory is admitted into an optimization batch.
17.6.4 Collection and deployment contracts
The operational constraints configured in the rollout harness (context compaction policies, observation truncation limits, tool refusal responses, and resource caps) collectively define the Markov decision process that the policy explores. If the training environment diverges from the intended deployment contract, the agent learns behavioral adaptations that fail in production. For instance, an agent trained with generous 50-turn allowances may learn meandering trial-and-error debugging strategies that collapse when placed under a strict 10-turn production limit. When teams intentionally expand training budgets as an exploratory curriculum to bootstrap discovery in sparse-reward environments, the transfer penalty under tighter deployment constraints must be tracked as an explicit empirical metric.
Decoupling model invocation from episode lifetime is equally critical for cluster efficiency. Tool-heavy agent rollouts can spend most of their lifetime waiting for external work: a policy might generate a tool request in 200 ms, followed by a 45 s wait while an external container compiles code and runs a test suite. An accelerator held for that rollout during the wait would compute for under 1 percent of the episode, and the loss compounds across every rollout in the training cluster. High-throughput collection engines decouple inference serving from episode execution, scheduling active invocations from other ready rollouts onto the accelerator while blocked rollouts wait for external tool observations. This interleaving maximizes hardware utilization without changing the causal observation stream consumed by any individual trajectory, and section 17.9 develops the bounded queues, group completion barriers, and version-tracking protocols required to coordinate that asynchronous schedule at scale. Even when a rollout record provides a faithful account of what the policy observed, what tokens it generated, and how the environment responded, however, that recorded trajectory can safely guide policy updates only if an uncompromised boundary separates the artifacts the policy is permitted to modify from the independent harness that evaluates them.
17.7 Verifier Isolation
Consider the representative trial rl_trial_01, in which an agent submits a candidate patch after running local unit tests against an edited connection pool. The reinforcement learning loop requires an authoritative verdict under the evaluation specification repair_accept_v1, not an unverified self-report from the environment that generated the patch. As the METR evaluations in section 17.3 illustrate, an agent capable of modifying its evaluation environment can bypass the intended task and rewrite its own scoring logic, and where the scorer is within reach it often does.
To prevent this failure mode, the learning loop puts the grader isolation of Grader isolation on the reward path, with held-out tests kept by a trusted controller outside the candidate’s sandbox. We call this architectural guarantee verifier isolation: the assessment harness and its verdict are protected from both the executing agent policy and the candidate code it submits.8 Whereas test coverage determines whether a test suite exercises the required behavioral contract, verifier isolation guarantees that the emitted verdict reflects the actual execution of those tests rather than unauthorized tampering.
8Complete Mediation of the Verdict: The tool boundary of Complete mediation checks every access to every protected object; verifier isolation applies the same rule, with least privilege, to the reward path. The policy never needs write access to test definitions or score records, so the runtime grants none, and every route to a verdict, including module resolution paths and test runner binaries, passes through the authoritative controller.
17.7.1 Separate assessment workspaces
Grading a patch inside the same container where the agent explored leaves the test runner, module imports, and exit codes exposed to the candidate’s modifications. The controller instead separates exploration from assessment (table 7): it captures the candidate artifact, binds it to the rollout identifier and base source revision, admits it only if it modifies files the task permits (application source, never tests, grading scripts, or container configuration), and applies it to a fresh copy of the declared fixture in a separate assessment workspace. The candidate remains untrusted after it crosses. Code that shares the grader’s interpreter can rewrite its assertions, a read-only mount of hidden tests still exposes them to inspection, and candidate stdout, stderr, and exit codes are telemetry rather than verdicts.
Table 7: Dual-workspace boundaries: Exploration and assessment have separate state and authority. The candidate crosses as untrusted material; separation does not certify its behavior.
Responsibility
Exploration side
Assessment side
Mutable task state
Task-scoped working copy and permitted local diagnostics
Candidate applied to an identified fixture under its own execution limits
Assessment authority
Public feedback, not control of the deciding check
Protected criteria, checker configuration, and authoritative result storage
Handoff
Consistently captured candidate and bounded provenance
Verdict and declared evidence bound to candidate, fixture, and checker versions
Reinforcement learning adds a failure mode that offline evaluation seldom meets, the exploitation of exception handling in the verification harness. A candidate patch that hangs, triggers an unhandled memory fault, or crashes the test runner can convert an inevitable test failure into a missing verdict. If the training pipeline silently drops rollouts with missing verdicts from batch normalization, or assigns them a default score higher than a failure, the policy learns to generate crash-inducing code rather than viable solutions. The resource caps declared for the candidate’s own execution decide whether a crash or timeout counts against the candidate. A breach of those limits, such as an infinite loop that exhausts a 60 s execution budget, is candidate-induced. Any such resource exhaustion or process crash maps to a failure (\(\TrajectoryReward = 0\)) and never outscores an orderly test failure. A failure of the host or grader while the candidate stayed within its limits, such as an out-of-memory crash caused by host oversubscription, is infrastructure and leaves the outcome unresolved under the rules of section 17.6.3. In neither case may the system emit a default pass, and training telemetry must report the two outcomes separately.
Isolating the workspace does not stop information from leaving through permitted outputs, the confinement problem that agent sandboxes face on their egress channels (Name resolution and local destinations). Even binary pass/fail responses leak information across successive submissions. In multi-turn rollouts, intermediate feedback must never expose hidden test names, assertion details, or internal stack traces, which would let the policy reconstruct the hidden suite turn by turn. The training loop must therefore declare what feedback reaches the agent and keep it to the minimum the task needs. Tests queried repeatedly to generate training rewards also become de facto training data, so an unbiased release assessment requires evaluation suites that the training loop never executes.
17.7.2 Repeated checks and noisy rewards
Verifier isolation eliminates deliberate tampering, but it cannot prevent non-deterministic evaluation outcomes. Flaky checks, caused by thread scheduling races, network jitter, or asynchronous event loops, can spuriously accept an incorrect patch or fail a valid one, even when the test definitions are completely protected. Repeating the check reduces the rate of accidental acceptance. If a test suite is fundamentally under-specified, however, an incorrect patch that satisfies those partial requirements will pass every repetition.
Let \(\TrialPassProbability\) represent the probability that a fixed candidate patch passes a single verification trial under a specified protocol. Assuming \(\VerificationTrialCount\) independent trials share the same pass probability, the all-pass rule, which requires unanimous success across all \(\VerificationTrialCount\) trials, accepts the candidate with probability
In this notation, \(\VerificationTrialCount\) denotes the total number of trials conducted, including the first. An initial pass followed by \(\VerificationTrialCount\) additional confirmation reruns would instead yield an acceptance probability of \(\TrialPassProbability^{\VerificationTrialCount+1}\). When trials are statistically dependent, the all-pass probability becomes the product of conditional probabilities \(\prod_{j=1}^{\VerificationTrialCount} \Pr(\text{pass}_j\mid\text{all earlier passes})\), which can diverge sharply from \(\TrialPassProbability^{\VerificationTrialCount}\). Launching fresh container instances with randomized execution seeds promotes independence, whereas a persistent checker bug or shared host-level bottleneck introduces positive correlation across runs.
Group normalization makes a flaky pass as damaging as a tampered one. On a hard task, a lone false pass receives the same near-\(\sqrt{\GroupSize-1}\) coefficient (equation 3) as an authentic solution, so the optimizer spends its largest updates on scheduling luck. Notebook 17.5 prices how an all-pass rule trades that accidental-reward risk against false rejections and extra verification work.
Napkin Math 17.5: The cost of an all-pass rule
Problem: How does requiring repeated passes change accidental reward, false rejection, and verification work?
Variables: In a hypothetical fixture, a fixed incorrect patch has trial pass probability \(\IncorrectCandidateTrialPassProbability\) of 0.20, while a fixed correct patch has trial pass probability \(\CorrectCandidateTrialPassProbability\) of 0.98. Trials are independent under an unchanged protocol. Acceptance requires \(\VerificationTrialCount\) total passes, with \(\VerificationTrialCount\) set to 3; verification stops at the first rejection. A separate illustration considers a group of 16 independently assessed incorrect candidates with the same \(\IncorrectCandidateTrialPassProbability\).
Math: A single trial for each incorrect candidate gives group probability \(1-(1-\IncorrectCandidateTrialPassProbability)^\GroupSize\) of at least one false pass, or 97.2 percent. In the special case of exactly one pass, its standardized coefficient approaches \(\sqrt{\GroupSize-1}\), approximately 3.87, as \(\AdvantageStabilizer\) approaches zero. The all-pass rule lowers each incorrect candidate’s acceptance probability to \(\IncorrectCandidateTrialPassProbability^{\VerificationTrialCount}\), or 0.8 percent, but rejects the correct candidate with probability \(1-\CorrectCandidateTrialPassProbability^{\VerificationTrialCount}\), or 5.88 percent.
Trial \(\VerificationTrialIndex+1\) is needed only if the preceding \(\VerificationTrialIndex\) trials passed. Consequently,
For the incorrect patch, \(1+\IncorrectCandidateTrialPassProbability+\IncorrectCandidateTrialPassProbability^2\) gives 1.24 trials; for the correct patch, \(1+\CorrectCandidateTrialPassProbability+\CorrectCandidateTrialPassProbability^2\) gives 2.9404. Running every trial regardless of earlier outcomes instead consumes \(\VerificationTrialCount\) trials per candidate.
Systems insight: Each added trial multiplies an incorrect patch’s acceptance by \(\IncorrectCandidateTrialPassProbability\) and a correct patch’s by \(\CorrectCandidateTrialPassProbability\), so the rule pays when \(\IncorrectCandidateTrialPassProbability\) is far below \(\CorrectCandidateTrialPassProbability\): here 3 trials cut an incorrect patch’s acceptance from a per-trial 0.20 to 0.8 percent, at a false-rejection cost of 5.88 percent for the correct one. Stopping at the first rejection makes the cost fall mostly on correct candidates, which run nearly all \(\VerificationTrialCount\) trials, so size the verifier pool for the passing fraction of the population, not for one trial per candidate.
A real candidate population mixes pass probabilities. Because the expected trial workload \(\sum_{j=0}^{\VerificationTrialCount-1} \TrialPassProbability^\VerificationTrialIndex\) is convex in \(\TrialPassProbability\), substituting the population’s mean pass probability directly into the formula underestimates total compute demand by Jensen’s inequality. High-quality candidates progress deeper into repeated trials than low-quality ones, concentrating the verification workload on the most promising rollouts. Sizing verification capacity, measured in completed candidate evaluations per second, therefore requires weighting the trial demand by the empirical distribution of candidate quality rather than using a single average pass rate.
When successive runs disagree, repetition alone is insufficient, because the divergence indicates an unstable test harness that requires its own repair. Verification harnesses should deliberately vary test execution order and process scheduling, running regular sanity checks against known-valid and known-invalid reference artifacts to detect fixture drift before emitting training rewards. If a specific test case stays flaky across runs, operators must quarantine it from the active reward loop. Because quarantining alters the task’s reward function, all test suite revisions must be versioned, preserving the historical trial records and audit trail. Verifier isolation and repeated assessment address the named tampering and flakiness paths; policy quality remains bounded by the properties the protected checks cover and requires independent release evidence.
Checkpoint 17.2: Optimizing agent trajectories
The recorded interaction, group statistic, and protected assessment must describe the same training event.
17.8 Training Pathologies
Even when an execution sandbox isolates the grader and every gradient step updates cleanly, reinforcement learning can steer an agentic policy into unhelpful or degenerate behaviors. Training curves alone rarely reveal why: for a complex repair task, rising reward accompanied by longer rollouts might indicate thorough automated testing, repetitive failed edits, or merely a shift in which tasks completed grading before timing out. Systematically diagnosing these pathologies requires connecting shifts in sampled token distributions, rollout lengths, reward sub-components, and gradient statistics to concrete hypotheses, because each failure mode demands a distinct systems intervention.
17.8.1 Diversity and token entropy
When a policy commits prematurely to a single failing strategy on unsolved tasks, its sampled rollouts repeat the same tool calls and stop exploring the alternatives needed to discover a passing repair. Token entropy tracks this narrowing at the token level by measuring uncertainty in a specified next-token distribution. For a delivered prefix \(\TokenPrefix\) and vocabulary \(\TokenVocabulary\),
with natural logarithms giving entropy in nats and \(0\log0\) interpreted as zero. When logging this metric, we must explicitly distinguish the raw base-model distribution from the post-sampling distribution shaped by temperature scaling, top-\(\Glyph{k}\) filtering, or grammar constraints. Adjusting inference parameters alters measured entropy without changing network weights, just as changes in benchmark difficulty or prompt length skew aggregate averages, so any reported entropy number is incomplete without specifying the prefix population and decoding pipeline that generated it. Even with those held fixed, low entropy is entirely appropriate at deterministic syntax positions, such as emitting closing brackets or fixed tool schemas; rather than setting an arbitrary global entropy threshold, engineers should inspect action divergence and tool-call variety on unsolved fixtures alongside token-level uncertainty.
Uniform rewards within a sample group introduce a similar diagnostic ambiguity. When all \(\GroupSize\) trajectories in a group receive identical scores, the zero-contrast update of section 17.5.4 zeroes out the reward-surrogate gradient. This zero-contrast condition is an algebraic property of group normalization, but its root cause varies: diverse attempts might all fail, diverse valid solutions might all pass, or an uncalibrated test harness might fail to distinguish partial progress. Disentangling these mechanisms requires inspecting the actual trajectory logs rather than relying on summary statistics. Table 8 maps common training observations to their underlying ambiguities and practical diagnostic checks; to ensure that an apparent change in policy behavior is not merely an artifact of shifting measurement conditions, all diagnostic comparisons must hold the task fixtures, sampling parameters, reward functions, and grader versions strictly constant.
Table 8: Training observations and diagnostic checks: A metric narrows an investigation without proving an internal mechanism or prescribing a guaranteed remedy.
Observation
What remains ambiguous
Useful next check
Lower entropy and repeated actions on unsolved tasks
Narrower exploration, changed decoding constraints, or a changed task mix
Compare action and candidate diversity on fixed diagnostic fixtures under the same sampling configuration
More equal-reward groups
Mastered tasks, persistently unsuccessful tasks, or missing assessment distinctions
Separate all-pass, all-fail, and unresolved cases; inspect coverage and reward components
More generated tokens without higher acceptance
Necessary checking, unproductive repetition, or a population/termination change
Compare lengths by outcome and task family, including budget stops and unavailable verdicts
Higher training reward without better independent assessment
Useful adaptation, reward-specific shortcuts, or changed grading
Audit candidate artifacts and reward components, then test the identified configuration under the independent protocol
17.8.2 Output length
Generated token counts, tool invocation counts, and wall-clock execution time capture distinct execution costs. A verbose trajectory may reflect thorough exploratory testing that culminates in a working patch, whereas a terse rollout may simply indicate early abandonment. Consequently, tracking mean trajectory length is rarely sufficient; engineers should inspect full length distributions segmented by outcome (accepted, failed, pending, and unknown). An unchanged average reward often conceals an expensive failure migration, where cheap syntax failures early in training turn into prolonged, compute-intensive execution loops that still fail verification.
The length bias of section 17.5.3 is the first mechanism to audit when failing trajectories lengthen unexpectedly, because per-rollout averaging charges each token of a long failure less than each token of a short one. Training the same fixed diagnostic tasks under the per-rollout and the constant-denominator objectives separates growth caused by that incentive from growth that reflects useful work, such as additional diagnostic checks that the environment reward favors.
When system requirements favor concise solutions, any length penalty introduced into the reward function must respect the preference hierarchy established in section 17.3. Under bounded reward structures, enforcing \(\FormatRewardWeight+\LengthPenaltyWeight<1\) guarantees that any passing trajectory scores higher than any failing trajectory, regardless of length. A poorly calibrated length penalty that causes a verbose successful repair to score lower than an immediate failure corrupts task incentives. Reward-based length penalties influence policy preferences only during learning; hard execution budgets (such as maximum token limits or turn timeouts) must remain strictly enforced by the execution runtime.
17.8.3 Entropy collapse and regularization
Standard symmetric ratio clipping provides a direct algorithmic route to entropy collapse, the premature loss of action diversity on tasks the agent has not yet solved. When an agent discovers a sequence that partially succeeds early in training, the positive advantage boosts the probabilities of those tokens, rapidly sharpening the distribution around that single path. The positive-advantage surrogate in section 17.5.2 stops increasing at a policy ratio of \(1+\RatioClippingWidth\), but this incentive threshold acts with extreme asymmetry in absolute probability space.
With \(\RatioClippingWidth\) set to 0.2, a dominant token already at probability 0.9 has an incentive threshold of 1.08. Because this threshold exceeds 1.0, the surrogate continues rewarding increases throughout the remaining probability mass. In contrast, for an alternative candidate token at probability 0.01, the positive surrogate incentive flattens after an absolute probability increase of 0.002. Gradients from other tokens and loss terms can still move a token’s probability ratio beyond the clipping interval. Rarely sampled actions therefore face a steep climb to regain probability, while dominant actions consolidate their lead. As action diversity drops across unsolved tasks, the \(\GroupSize\) rollouts in each group collapse into near-identical trajectories. Their rewards tie, precipitating the zero-contrast state described in section 17.5.4 where the reward-surrogate gradient vanishes.
To counteract this dynamic, the clip-higher formulation decouples the clipping boundaries into an asymmetric interval \([1-\LowerRatioClippingWidth, 1+\UpperRatioClippingWidth]\) with \(\UpperRatioClippingWidth > \LowerRatioClippingWidth\)(Yu et al. 2025). This wider upper bound allows low-probability tokens backed by positive advantages to climb more rapidly, while the lower bound still curbs policy drift for penalized tokens. As always, diagnostic context matters: low entropy at deterministic syntax positions is normal and desirable; the operational danger is collapsing diversity among tool invocations and decision branches on unsolved tasks.
Yu, Qiying, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, et al. 2025. “DAPO: An Open-Source LLM Reinforcement Learning System at Scale.”arXiv Preprint arXiv:2503.14476.
Alternative regularizers attack this problem through different mechanisms. An explicit entropy bonus directly penalizes sharp token distributions across targeted positions. A reference-policy divergence penalty, such as the \(\KlPenaltyCoefficient\)-weighted term of equation 4, instead penalizes drift from the initial base model, though it enforces no minimum entropy floor of its own. Both approaches modify the loss surface and require controlled experimentation: hold task distributions and sampling parameters fixed, evaluate on downstream task success and token efficiency, and track policy-ratio distributions. A smoother training loss is weak evidence of healthy exploration; only verified task completion counts.
When exploration collapses, an agent can enter repetitive loops (emitting duplicate shell commands or re-reading unchanged files) where runtime watchdog mechanisms must intervene. However, repetitive behavior in multi-turn environments is inherently ambiguous: waiting for a slow build, polling an asynchronous API, or retrying a failed test after adjusting parameters can mimic an infinite loop. Systems should log the exact termination trigger and apply the stop-state rules of section 17.6.3, which score budget exhaustion as a task failure and keep infrastructure interruptions out of the reward. Terminating a runaway rollout caps resource consumption immediately, but it settles nothing outside the slot; environment state and external side effects still pass to their cleanup handlers.
↩︎ Runtime watchdog mechanisms for detecting and terminating repetitive agent loops are established in Semantic Watchdogs.
17.8.4 Recovery behavior
Reports such as the 2025 DeepSeek-R1 study document spontaneous increases in response length during pure reinforcement learning (R1-Zero), accompanied by natural-language tokens where the model appears to reconsider prior reasoning steps (DeepSeek-AI et al. 2025). From a systems perspective, however, introspective tokens such as “Wait, let me double-check” are merely generated text, not verified cognitive state. In an agentic system interacting with real infrastructure, the sole criterion for genuine recovery behavior is whether subsequent tool invocations successfully correct an execution failure or adapt to unexpected environmental feedback.
DeepSeek-AI, Daya Guo, Dejian Yang, et al. 2025. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.”arXiv Preprint arXiv:2501.12948, ahead of print. https://doi.org/10.48550/arXiv.2501.12948.
Consider testing recovery on the connection-pool repair task. A diagnostic evaluation fixture can intentionally inject an environmental disturbance, for example a dependency version conflict introduced immediately after the agent completes its initial inspection. The evaluation then measures whether the policy gathers fresh diagnostic data, revises its patch to satisfy the conflicting constraints, and successfully passes the test suite within the original resource allowance. Tracking failed and unhandled executions alongside successful runs is essential to ensure that apparent recoveries do not reflect hidden timeout artifacts. Designing experiments around controlled perturbations provides empirical evidence of recovery under the tested perturbation, just as targeted recovery demonstrations mitigate compounding errors during imitation learning.
↩︎ Recovery trajectories as a remedy for compounding errors in imitation learning are established in Exposure Bias.
Matched-budget evaluations, in which candidate policies and baselines operate under identical token and turn limits, reliably separate productive deliberation from empty verbosity. Generating this diagnostic evidence at scale demands robust infrastructure: a versioned catalog of task fixtures, structured rollout traces with explicit execution states, and sufficient isolated grader capacity. These diagnostic demands directly link algorithmic health to the systems architecture of the rollout pipeline.
17.9 Environments and Rollouts
Generating rollouts for verifiable reinforcement learning requires coordinating several pipeline stages operating at fundamentally different latencies. Before a single repair rollout can contribute to a parameter update, the infrastructure must orchestrate model inference to sample tokens, provision an isolated sandbox (such as pool_fixture_v1), execute stateful tool commands, run automated assessment suites, and stream trace records to storage. An inference step completes in tens of milliseconds, tool commands and compilers require seconds, and sandbox provisioning can take longer still. The primary systems challenge is to keep every stage saturated while preserving the statistical contract of the learning algorithm: the pipeline must maximize rollout throughput without silently altering, truncating, or biasing the distribution of trajectories delivered to the optimizer.
17.9.1 Task fixtures and reset conditions
A recorded trajectory from a previous failure cannot be replayed as a reinforcement learning task on its own; without a reproducible container image, pinned dependencies, and a verified grading suite, candidate rollouts fail on missing packages or residual files left by earlier runs. Every rollout therefore begins with a concrete task environment governed by a fixture contract that defines the initial filesystem state, available tool APIs, network permissions, and reward evaluation criteria.
↩︎ Collection fixtures, which layer a collection contract on the evaluation gym’s execution baseline, are established in Task Fixture Design.
Data splitting requires the same discipline before any task is sampled or filtered into a curriculum, keeping related problem variants, mutations, and solution-equivalent tasks on the same side of the split. Training on familiar repositories teaches local code conventions, whereas holding out entire repository families tests whether the policy transfers to unfamiliar codebases. Within each split, ground-truth reference solutions and known defective candidates calibrate the verifier so that flaky dependencies and harness faults show up as operational errors rather than false rejections. Because filtering tasks by difficulty to sharpen reward contrast also shifts the task mixture, the pipeline must log the active curriculum rather than reading a higher training reward as a gain across the uncurated population.
↩︎ Task groups, which keep every variant of a problem on one side of the split, are established in Evaluation claims and task groups.
Between successive episodes, an environment slot executes a rollout reset to restore the fixture to its declared initial state. Its obligations are those of the trial reset in State reset between trials, which terminates lingering processes, scrubs writes and credentials, and settles outstanding operations before the slot is reused; snapshots restore local disk state in milliseconds but leave external side effects to teardown handlers. Controlled stochasticity, such as randomized problem seeds or initial state permutations, must be codified directly in the fixture specification. While offline benchmarks incur reset overhead only intermittently, reinforcement learning loops execute resets across millions of rollout cycles, making reset efficiency a direct driver of aggregate throughput.
17.9.2 Environment-cycle time and throughput
A cold container reset takes several seconds, whereas a snapshot restore finishes in milliseconds, so the reset mechanism can easily take as long as the model inference and tool calls of the episode itself. To price how much capacity a faster reset actually recovers, consider an environment slot executing agent trajectories serially. Let \(\TurnHorizon\) be the horizon (the number of interaction turns), \(\ModelTime\) the model inference time per turn, \(\ToolTime\) the tool execution time per turn, \(\RolloutAssessmentTime\) the final assessment duration, and \(\EnvironmentResetTime\) the environment reset time. Assuming constant durations, no intra-slot concurrency, and zero queueing delay, the total slot cycle time is
Accelerating the reset routine directly shortens this cycle, but by a bounded amount. Amdahl’s law sets that bound. Speeding up one stage of a fixed workload shortens the whole by at most the time that stage occupied, so the overall speedup is limited by that stage’s share of the total. Multiplying the reciprocal of \(\EnvironmentCycleTime\) by the total number of environment slots gives the theoretical capacity ceiling of the rollout fleet. However, that projection assumes model inference servers, tool executors, verification sandboxes, and storage pipelines can sustain this aggregate rate without contention. We compute this baseline capacity in notebook 17.6 before examining where downstream shared bottlenecks arise.
Napkin Math 17.6: Reset cost and slot capacity
Problem: A hypothetical pool contains 256 environment slots. Each episode has 20 decisions, each using 120 ms of model service and 80 ms of tool execution, followed by 0.5 s of assessment. Cold reset takes 3.5 s; snapshot reset takes 15 ms. What capacity change follows under the serial-slot assumptions?
Math: Under the serial-cycle model in equation 6, non-reset work takes 4.5 s per cycle. Cold reset gives a cycle of 8 s, of which 56.25 percent is non-reset work. Snapshot reset gives a cycle of 4.515 s and a non-reset fraction of 99.67 percent. For either case, idealized pool capacity is the number of slots divided by cycle time.
Result: The corresponding capacities are approximately 115,200 and 204,120 episodes per hour, a 1.77× local capacity ratio. The non-reset fraction includes tool and assessment time, rather than representing accelerator utilization alone.
Systems insight: Reducing cold reset from 3.5 s to the stated snapshot latency yields a 1.77× slot-capacity gain; eliminating reset entirely would leave the 4.5 s of active work as the cycle-time floor. This is Amdahl’s law applied to the slot. The capacity ratio is \(\ColdResetCycleTime / \SnapshotResetCycleTime\), and because the active work term remains fixed, optimizing reset latency yields diminishing returns once reset overhead is a small fraction of the cycle. Furthermore, these recovered slot-hours translate into training updates only if downstream inference serving, verification workers, storage pipelines, and the training loop possess sufficient headroom to process 1.77× as many episodes. In a balanced pipeline, the stage with the least headroom dictates the realized throughput gain.
↪ Amdahl’s law bounds the speedup that a fixed serial fraction allows, as developed in The coordination tax.
17.9.3 Complete-group scheduling
Rollout generation continuously interleaves bursty model inference with blocking tool waits, while training workers consume completed trajectories in minibatches. Allocating distinct resource pools decouples these workloads, whereas colocated or alternating execution schedules reduce model weight broadcast and network synchronization costs. The optimal balance depends on the relative runtimes of rollout generation and gradient computation. Because all \(\GroupSize\) rollouts for a given task share an identical prompt, a radix-tree prefix cache computes the key-value states for that prompt once across the cohort (principle 5). Capitalizing on this prefix reuse requires matching model weights, identical prompt token sequences, and compatible cache allocation policies; promoting a new policy checkpoint inevitably invalidates existing cache entries and forces a cold start even on identical prompts.
↩︎ The weight, exact-prefix, and residency conditions for prefix-cache reuse are established in Prefix compatibility.
When the training loop synchronizes across rollout groups before updating parameters, the trainer enforces a group barrier. The step duration is dictated by the slowest member of the cohort:
Every worker slot that finishes earlier sits idle until the tail completes. Because variable tool execution times, multi-turn reasoning loops, and compiler or test runs give agent trajectories heavy latency tails, that tail wait can leave a large share of the pool idle.
For a hypothetical group of 4 whose members complete between 0.2 s and 16.8 s, equation 7 holds all four slots until the 16.8 s member finishes, stranding 65 percent of reserved slot capacity in an idle state. This barrier penalty, charted in panel (a) of figure 2, motivates releasing slots early and gathering rollouts asynchronously.
Under group-relative objectives, asynchronous rollout completion does not remove the mathematical requirement for an aggregated group baseline. The scheduler assigns the group identifier, fixture configuration, planned cohort membership, and generating checkpoint version prior to dispatch. Each completed member record and its verification status buffer in memory until the group closure policy determines eligibility. If the objective requires verified rewards across all group members, group closure must hold the record until all members report. If an environment terminates due to turn caps or time limits, the orchestrator applies the declared protocol penalty; infrastructure failures such as sandbox crashes or network timeouts stay unresolved under the rules of section 17.6.3, and the closure policy must say whether the group waits for them or closes without them.
As soon as a rollout finishes, its environment slot frees its inference concurrency and sandbox resources for subsequent work, retaining only its compact trajectory record in an aggregation buffer while evicting key-value cache states from accelerator memory. Because rollouts complete out of order, group buffers, verification queues, and training ingestion queues require bounded capacities in both record count and total bytes. Admission controllers must track in-flight slot allocations alongside buffered records, applying backpressure upstream to prevent intermediate queues from overflowing host memory during bursty completions.
Prematurely closing a group with only its fastest-finishing members distorts the baseline sample. Slower trajectories often represent harder subtasks, alternative multi-step tool interactions, or transient infrastructure contention. Recomputing relative advantages across only the early completions fails to recover the true underlying distribution; early truncation imposes a survival bias that favors fast, simplistic solutions. Training pipelines must explicitly log all exclusions and their causal categories, monitor completion latency distributions, and track task coverage shifts over time. Similarly, filtering out zero-variance groups (where all members receive identical rewards) reduces gradient variance in advantage-based policy surrogates, but it also eliminates training signals intended for auxiliary objectives, such as reference-policy divergence constraints.
17.9.4 Policy staleness
Decoupling rollout collection from training updates means trajectories can arrive after the optimizer has already advanced the policy weights. The version disparity between the policy that sampled an action and the policy undergoing optimization is policy staleness. Asynchronous actor-learner frameworks such as IMPALA address this discrepancy by pairing asynchronous scheduling with off-policy corrections,9 reweighting trajectory tokens by the probability ratio between the learner and actor checkpoints (Espeholt et al. 2018). For an episode pinned to a single behavior checkpoint, let \(\BehaviorPolicyVersion\) denote its generating version and \(\text{ver}_t\) the trainer’s active version. When the training loop steps at a steady pace, the update-count lag is
9V-trace: IMPALA’s correction weights each step by the ratio of learner to actor probabilities, truncated at a ceiling, accepting some bias to keep the variance of long products of ratios in check (Espeholt et al. 2018). Truncation matters more for agent rollouts than for short episodes, because a product of per-token ratios over thousands of tokens is otherwise dominated by a few extreme values.
Espeholt, Lasse, Hubert Soyer, R’emi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, et al. 2018. “IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures.”Proceedings of the 35th International Conference on Machine Learning (ICML), 1407–16.
where \(\RolloutAdmissionAge\) is the wall-clock time elapsed from sampling the first prompt token to admission into the training batch, and \(\PolicyUpdateStepTime\) is the training step latency. This relationship enables system designers to calibrate staleness bounds directly from rollout duration. Long agent rollouts are precisely the trajectories most prone to severe staleness. Bounded staleness has a long lineage in distributed training architectures, where worker parameter reads are constrained to a fixed step horizon (Ho et al. 2013), and asynchronous actor-learner algorithms tolerate bounded lag in exchange for high actor utilization (Mnih et al. 2016). Applied at admission (figure 2, panels b and c), the check admits the two short rollouts and excludes the long one, recording why it was dropped. These are record-level lag decisions, not a complete group-admission rule. Under an objective that requires the planned cohort, every member must also have a resolved reward and satisfy that bound before the group enters an update.
Figure 2: Group Barriers and Version-Checked Admission: (a) In the hypothetical group of four, the barrier holds every slot until the 16.8 s member finishes, leaving 65 percent of reserved slot time idle. (b) Asynchronous collection releases slots early, so the trainer’s version advances while rollouts run. Each record carries the version that sampled it, and its lag \(\FiniteChange\text{ver}\) at admission grows with its duration; the long rollout exceeds the illustrative bound and is excluded with its reason recorded. (c) A record leaves verification with a pass, a fail, or an unavailable verdict, waits in a bounded group buffer, and reaches the trainer only if its lag is within the bound. While this admission check enforces a strict bound on version lag, it cannot prevent distribution drift when individual gradient updates make large steps in parameter space.
While the version counter provides an operational threshold for admission control, distribution divergence requires separate monitoring: a single large optimizer step can shift output probabilities far more dramatically than several incremental updates. Production diagnostics must break down elapsed age into generation time, tool wait states, verification latency, group buffer residence, and ingestion queue delay, while tracking empirical probability-ratio distributions (\(\Policy / \pi_{\text{old}}\)) to detect distribution drift directly. Logging per-invocation metadata as described in section 17.6 accurately captures that behavior distribution even when policy checkpoints update mid-episode; training on such composite trajectories requires optimization formulations that support turn-by-turn behavior probabilities, because retroactively relabeling an entire trajectory under the newest policy checkpoint distorts importance weights and introduces optimization instability.
Freshness limits fundamentally balance off-policy variance against selection bias. Discarding trajectories that exceed a staleness threshold bounds policy lag, but because complex tasks demand more reasoning steps and tool calls, strict staleness cutoffs systematically eliminate hard problems from the training distribution. As indicated by equation 8, systems engineers can preserve task diversity by reducing the numerator rather than loosening the bound. Reducing generation, tool-wait, verification, group-buffer, and ingestion delays can shorten \(\RolloutAdmissionAge\) while preserving the long-horizon tasks the learner needs. Synchronous collection eliminates staleness entirely at the expense of straggler idle time, whereas multiple optimization epochs over a single collected batch reintroduce policy lag. While the clipped surrogate objective in section 17.5.2 limits policy updates driven by stale probability ratios, it neither bounds importance sampling variance nor corrects for the shift in visited state distributions (Schulman et al. 2017). Because discarded trajectories consume inference and sandbox capacity without producing gradient updates, monitoring systems must track discard rates, compute overhead, and shifts in task coverage.
17.9.5 Verification capacity
Even when inference and sandbox slots keep up with rollout generation, an undersized verifier pool backs up completed trajectories in the assessment queue, lengthening \(\RolloutAdmissionAge\) until valid rollouts miss the staleness cutoff. Verifier sizing reuses the pool model of Collection credits and staging reserves, changed only in that a reinforcement-learning trajectory may need several trials rather than one. Let \(\TrajectoryAdmissionRate\) denote completed trajectories arriving per second, \(\WorkerCount\) the number of independent verification worker slots, and \(\VerificationTrialTime\) the mean execution duration for one verification trial, including environment provisioning, execution, and cleanup. When each trajectory requires \(\VerificationTrialCount\) total trials to evaluate determinism or flaky tests, each trajectory demands an average of \(\VerificationTrialCount \VerificationTrialTime\) worker-seconds. By Little’s law (theorem 5.1), mean busy workers equal the arrival rate times that demand. The ideal service capacity \(\VerificationPoolCapacity\) and resulting offered utilization \(\ServingUtilization\) are
Both \(\TrajectoryAdmissionRate\) and \(\VerificationPoolCapacity\) measure completed trajectories per second. As in that pool model, a pool run at \(\TrajectoryAdmissionRate=\VerificationPoolCapacity\) has no margin for bursts or slow trials, so designers choose a target utilization \(0<\TargetServingUtilization<1\) and provision at least \(\lceil\TrajectoryAdmissionRate \VerificationTrialCount \VerificationTrialTime/\TargetServingUtilization\rceil\) worker slots. Queue capacity limits and upstream admission control remain necessary. While executing a candidate’s \(\VerificationTrialCount\) trials concurrently across multiple slots reduces wall-clock verification latency, the aggregate worker-second demand remains identical.
For a hypothetical arrival of 512 trajectories every 16 s, \(\TrajectoryAdmissionRate\) is 32 trajectories per second. With each trial lasting 8 s, a single trial per trajectory consumes 256 worker equivalents at that arrival rate. Executing all 3 trials elevates demand to 768 worker equivalents. Applying the utilization relationship in equation 9 with an illustrative 80 percent target requires provisioning 960 worker slots. Verification demand scales linearly with \(\VerificationTrialCount\), and the target utilization establishes the necessary headroom against queueing delays.
Early-exit evaluation policies fundamentally alter these capacity requirements. Under the stop-on-first-rejection policy of notebook 17.5, verification terminates upon the first failing trial, so capacity models substitute that notebook’s expected executed trial count in place of \(\VerificationTrialCount\) when computing total worker demand, averaged over the empirical mixture of pass rates for the convexity reason given in section 17.7.2. Sizing calculations must also incorporate unequal trial durations, sandbox initialization overhead, and retry cascades. In-memory queues absorb brief arrival spikes, but sustained overload requires adding worker slots or adopting more aggressive early-exit evaluation rules.
17.9.6 Policy version gating
Once the verifier pool has scored a batch and the trainer has produced a new checkpoint, that checkpoint faces two distinct promotion gates: entering the rollout fleet to collect the next batch, and shipping to production users. For internal collection gating, automated checks verify prompt-template compatibility, tool-calling syntax, entropy bounds against policy collapse, and memory utilization before promoting the candidate weights to the rollout fleet, while the orchestrator records the promoted artifact digest and generating version for every dispatched episode. If a promoted version still falls into degenerate loops or crashes sandboxes, the controller halts further admission under that version, and any side effects produced by faulty rollouts remain subject to their designated cleanup handlers.
Production deployment release requires a formal evaluation framework verifying regression resistance, safety constraints, latency budgets, and out-of-distribution transfer. Internal training diagnostics provide rapid feedback during optimization, but they must remain strictly quarantined from release benchmarks to prevent benchmark leakage. Production evaluation suites must enforce an isolated assessment boundary, logging benchmark exposure whenever validation metrics inform architectural or hyperparameter revisions.
Yue, Yang, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2025. “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?”arXiv Preprint arXiv:2504.13837.
System evaluations must carefully distinguish between operational consistency and genuine capability expansion. In the competitive reasoning benchmark study of Yue et al. (2025), RLVR-trained models improve pass@1 while base models match or exceed them when sampled across broad cohorts (\(\IndependentAttemptCount \gg 1\)). This pattern is consistent with concentrating probability mass onto successful solution modes that the base model could already generate; it does not establish a universal limit on acquiring new behavior. Reporting pass@1, pass@\(\IndependentAttemptCount\) at large \(\IndependentAttemptCount\), and the repeated-trial pass\(^\IndependentAttemptCount\) defined in Potential and repeated-run consistency separates a policy that has become more reliable from one that has expanded its underlying capabilities. A candidate checkpoint must remain unreleased until it satisfies recovery, transfer, and unresolved-outcome criteria across comprehensive production testbeds. Balancing rollout generation, straggler barriers, staleness bounds, verifier capacity, and release gating provides the infrastructure that keeps this learning loop saturated; tracing how these systems mechanisms operate together requires following a complete update cycle on a concrete repair workload.
17.10 Case Study: Reinforcing the Repair Policy
Consider the connection-pool repair group with rewards \([0,1,0]\) from section 17.5. Three fresh trials, rl_trial_01, rl_trial_02, and rl_trial_03, execute task pool_repair_01 against the fixture pool_fixture_v1, sampled from the supervised checkpoint trained_policy_v1, and each ends in a verdict under repair_accept_v1. The recovery run that produced workspace ws_r20 and the demonstrations built from it remain historical artifacts of curation and fine-tuning; they supply comparisons, not targets, for this group. Following the group through each stage of the chapter shows which question every stage answers before the middle rollout is allowed to move the weights.
The first question is whether the lone pass is a pass. A \([0,1,0]\) group on a hard bug is exactly the shape that tampering produces, so the verdict for rl_trial_02 counts only because it came from the assessment side of table 7. The patch was admitted as a source-only diff, applied to a fresh copy of the fixture, and graded by a controller whose tests and result store the candidate could not reach. The defect is a race on connection return, so the fixture is timing-sensitive, and the all-pass rule of notebook 17.5 applies. If an incorrect patch passes a single trial with probability 0.20, 3 independent trials cut its acceptance to 0.8 percent at a 5.88 percent risk of rejecting a correct one. A trial whose verifier crashed, or whose container was preempted, is marked unresolved under the stop-state rules of section 17.6.3 rather than coerced to zero, because converting an infrastructure fault into a task failure penalizes the policy for the cluster and shifts the group baseline.
The second question is whether the update can be computed on what the policy actually saw. Each rollout record holds the delivered prefix and sampled token IDs of every invocation, the behavior log probabilities under the recorded sampling configuration, and the checkpoint version that produced them (table 6). Before admission, the trainer checks each record’s version lag against its bound (equation 8); a trial that ran long enough to fall behind is excluded with its reason logged rather than silently relabeled under the newest weights. In this worked example, all three rewards are resolved and all three members satisfy the lag bound, so the original complete group is eligible. Under its declared complete-group rule, an unresolved or stale member prevents the group from entering the update; excluding members and recomputing its coefficients would require a separately declared objective and selection analysis.
Only then does GRPO compute coefficients. The pass receives 1.414214 and each failure -0.707107 (notebook 17.3), and under per-rollout averaging each token of rollout \(\GroupRolloutIndex\) carries \(\GroupRelativeAdvantage_i/\RolloutPolicyTokenCount_i\) (equation 5), so the length of each failure sets how hard its tokens are pushed down. Because the reward scores only the outcome, every token in rl_trial_02 receives the same coefficient. The edit that adds the missing notification and any redundant status checks along the way are reinforced alike, and only many such groups separate them statistically. The frozen reference policy stays distinct from the behavior policy, contributing only the divergence penalty. Because the GRPO name covers several variants, the training log records the clipping thresholds, the length and dispersion normalizations, and the penalty weight \(\KlPenaltyCoefficient\) with the update.
When an update over eligible groups yields a candidate checkpoint, such as rl_policy_v1, lineage binds its weights and inference configuration to the rollout records, fixtures, and verifier versions that produced it. The controller may promote rl_policy_v1 to the rollout fleet for further collection, since that is an internal training decision. Releasing it is a different decision. Higher training reward on a known fixture can coexist with regression on unobserved failure modes, and the benchmark study discussed in section 17.9.6 illustrates how higher pass@1 can coexist with unchanged broad-sampling performance, so the release case rests on held-out recovery fixtures evaluated by pass@1, pass@\(\IndependentAttemptCount\) at large \(\IndependentAttemptCount\), and pass\(^\IndependentAttemptCount\) under the protocol of Agent Evaluation. Each stage therefore answers one question that no other stage can: the isolated verifier says whether the reward is evidence, the record says whether the update is faithful, the coefficients say how hard the evidence pushes, and the independent evaluation says whether the result belongs in production.
17.11 Fallacies and Pitfalls
A rollout group can easily contain positive rewards without establishing a sound solution, and a policy update can fit that group without improving the deployed agent. The pitfalls detailed below each turn a narrow, scoped measurement into an unwarranted conclusion, and each carries a concrete systems cost.
Fallacy: An automated reward is a complete task specification.
Automated test suites check only the conditions their authors anticipated, and reinforcement learning operates as an aggressive search procedure that reliably uncovers unhandled edge cases. When a policy’s true success rate is low, Bayesian base rates compound this vulnerability. As demonstrated in notebook 17.1, a verifier that misclassifies only 2.5 percent of failing attempts still results in 18.4 percent of all rewarded rollouts being outright failures when the policy solves only 10 percent of tasks. Because reinforcement updates treat every rewarded attempt as a success, the optimizer systematically increases the probability of the tokens that generated those spurious answers. Always benchmark the verifier against suites of known incorrect solutions, evaluate its false-acceptance rate against the policy’s current pass rate, and gate production deployment on strict held-out task criteria rather than raw training rewards.
Pitfall: Leaving the scorer, the runner, or its configuration where the candidate can reach it.
If an agent can read or write to its evaluation harness, modifying the test framework becomes the path of least resistance. In the METR evaluations, when a task exposed an editable scoring script to the agent, the model bypassed the intended challenge and tampered with the verifier in 21 of 21 runs (war story 17.1). During training, this vulnerability causes severe gradient distortion. On difficult tasks where honest attempts fail, the first rollout that discovers an exploit becomes the group’s lone positive sample. In a rollout group of 16, that single hack receives an advantage coefficient of 3.87, while each honest failure receives a negative coefficient of 0.26 in magnitude, strongly reinforcing the exploit. Always evaluate candidate solutions in an ephemeral workspace isolated behind a hardened controller that retains exclusive ownership of the test harness, the execution runner, and the result path (section 17.7). Even within that sandbox, treat all generated agent code as untrusted input.
Fallacy: A bigger group is the way to extract signal from hard tasks.
Increasing the rollout group size \(\GroupSize\) appears to offer a straightforward way to capture rare successes on difficult tasks. While larger groups improve the likelihood that a batch contains contrasting rewards, the probability of contrast alone does not measure the information each group supplies. At a pass rate of 5 percent, rollout groups of size 8 consume 23.8 rollouts for every informative group with nonzero variance. Expanding the group size to 16 still demands 28.6 rollouts per informative group. By contrast, steering the training curriculum toward tasks near a 50 percent pass rate consumes only 8.1 rollouts per informative group (notebook 17.4). Where obtaining contrast is the binding constraint and the task-mixture change is acceptable, this comparison favors targeting difficulty. Choose the group size using baseline precision, required task coverage, and held-out learning progress as well as rollout cost, and log the sampling distribution.
Pitfall: Reading longer self-correction text as evidence of better reasoning.
When an agent produces lengthy internal monologues and verbose self-critiques, engineers often mistake length for deeper reasoning. In reality, output lengths frequently expand as a direct artifact of loss normalization. Under per-rollout token averaging, a failing rollout of 128 tokens penalizes each individual token 32× as heavily as a failing rollout of 4,096 tokens. Because token-level gradient penalties dilute across longer generation lengths, this objective can favor longer failures even when the extra text performs no useful validation (equation 5). To uncover length bias, compare sequence lengths conditioned on task outcome and train with a constant-denominator loss objective. Ultimately, evaluate candidate revisions by the concrete tool actions and verified task completions they deliver under a matched token budget, rather than by the model’s narration of reconsidering.
Fallacy: Faster environment reset speeds training proportionally.
Optimizing container teardown and environment initialization yields satisfying microbenchmarks, but Amdahl’s law strictly limits the resulting gains in end-to-end training throughput. Reset latency represents only one slice of the rollout lifecycle. Cutting reset overhead from 3.5 s to 15 ms delivers a 233× acceleration in reset speed, yet it improves execution slot capacity by only 1.77×. The remaining 4.5 s spent on model inference, tool execution, and verifier scoring continues to dominate every cycle (notebook 17.6). Moreover, increased rollout throughput converts into faster training progress only if the inference workers, verification services, and parameter optimizers maintain sufficient headroom to ingest the higher volume. Profile the entire training loop to pinpoint the saturated pipeline stage before investing engineering effort into optimizing any individual phase.
17.12 Summary
Reinforcement learning from verifiable rewards uses assessed interactions to guide a policy update without requiring a demonstrated response at every decision. Its opportunity depends on what the sampling process can reach and what the reward distinguishes. Its risks arise because the recorded score, rather than the full task intent, enters the optimization. Reward validity, sampling fidelity, and external enforcement therefore remain part of the learning system rather than prerequisites that can be forgotten once training starts.
The repair group illustrates the resulting chain of responsibility. Completed assessments support group-relative coefficients; recorded policy actions and their actual inputs support the chosen update; resource and version controls govern the collection loop. None of those steps substitutes for evaluating the resulting candidate under its intended operating conditions. Training changes the policy and moves no exposure; the runtime still bounds horizon, state, and authority for whatever the updated weights propose. The relations in table 9 carry the chapter’s quantitative decisions, each under the assumptions stated where it is derived.
Table 9: Key Equations of Verifiable-Reward Training: Each relation holds only under the assumptions stated where it is derived, such as independent trials, binary rewards, constant stage times, or a steady training rate.
Generated-token price of continuation sampling at a chosen fork set; exhaustive forking grows as \(\ContinuationCount \TurnHorizon^2\,\TurnGeneratedTokenCount/2\)
\(\TrialPassProbability^{\VerificationTrialCount}\) and \(\mathbb E[\ExecutedTrialCount\mid \TrialPassProbability]=\sum_{j=0}^{\VerificationTrialCount-1}\TrialPassProbability^\VerificationTrialIndex\)
Acceptance under the all-pass rule and its stop-on-first-rejection trial demand
Key Takeaways: A learning signal is not a release decision
Reward learning needs reachable distinctions: Fresh interactions provide a different signal from admitted imitation targets. The policy must sample behavior the reward can distinguish; useful support can come from several sources, and supervised fine-tuning is not a mandatory preceding stage.
A protected score still has a scope: Automated checks assess specified properties, not every aspect of task intent. Reward composition can change preferences, while evaluator isolation blocks particular interference paths; neither replaces runtime permissions or independent assessment of the candidate.
Return is not causal credit: A baseline changes the coefficient attached to a sampled action without identifying its individual contribution. Continuation estimates require compatible forks and additional execution; their uncertainty and cost must accompany any interpretation of an estimated advantage.
Group coefficients do not constrain every update: GRPO’s centering and normalization define a particular surrogate. Equal rewards remove its reward-dependent terms, not all learning; clipping changes the objective but does not impose a hard bound on policy drift or gradient magnitude.
Training needs the actions actually sampled: Retain per-invocation inputs, sampled policy tokens, and matching behavior probabilities. Observations supply conditioning rather than sampled actions, and incomplete assessments or truncated episodes require declared treatment rather than silent relabeling or exclusion.
Behavioral changes require behavioral evidence: Entropy, length, reward, and recovery traces guide investigation but do not reveal internal cognition or uniquely diagnose a cause. Test interventions against task outcomes and costs under controlled conditions instead of optimizing telemetry in isolation.
The collection loop and release have different gates: Bounded queues, group completion, version accounting, and verifier capacity govern training operation. Admission for further collection is distinct from external deployment, which requires evidence for the identified policy and retains all runtime controls.
Verified trajectory post-training (principle 13) reaches its last form in this chapter. Imitation admits recorded responses as targets after curation; reinforcement learning admits every action the policy sampled, malformed ones included, and weights each by an outcome a verifier certified, so the admission decision passes from the curator to the reward. The worked group makes the verification asymmetry (principle 3) concrete under that optimization pressure. A protected, scoped assessment must be joined to the sampled record before it can justify an update. Executing the candidate inside that assessment also stress-tests containment beneath the model (principle 9), because fresh workspaces still receive untrusted code. The same asymmetry governs both ends of the loop. A learned score may shape an update, while the reward and the release decision rest on protected checks wherever such checks exist. Neither the learning signal nor an improved evaluation result moves authority into the weights; invariant closure (principle 1) continues to govern the effects of each proposal.
17.13 Historical Perspective and Further Reading
The problem this chapter manages began as a question about games. Minsky (1961) asked how a learning system that sees only whether a long strategy won or lost can distribute credit among the many decisions that produced the result, and he named it the credit-assignment problem. Two answers followed that still organize the field. Temporal-difference learning predicted later outcomes from earlier ones, so each step could be scored against the next prediction rather than the final result (Sutton 1988). The REINFORCE family instead differentiated expected reward directly through the probabilities of sampled actions and subtracted a reinforcement baseline to tame the variance (Williams 1992). The policy gradient theorem showed that such estimates remain valid with function approximation (Sutton et al. 1999), and the analysis of baselines as control variates explained why some reduce variance more than others (Greensmith et al. 2004).
Sutton, Richard S. 1988. “Learning to Predict by the Methods of Temporal Differences.”Machine Learning 3 (1): 9–44. https://doi.org/10.1007/BF00115009.
Sutton, Richard S., David A. McAllester, Satinder P. Singh, and Yishay Mansour. 1999. “Policy Gradient Methods for Reinforcement Learning with Function Approximation.”Advances in Neural Information Processing Systems 12 (NIPS 1999).
Greensmith, Evan, Peter L. Bartlett, and Jonathan Baxter. 2004. “Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning.”Journal of Machine Learning Research 5: 1471–530.
Ho, Qirong, James Cipar, Henggang Cui, Seunghak Lee, Jin Kyu Kim, Phillip B. Gibbons, Garth A. Gibson, Greg Ganger, and Eric P. Xing. 2013. “More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server.”Advances in Neural Information Processing Systems (NeurIPS) 26.
Mnih, Volodymyr, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. “Asynchronous Methods for Deep Reinforcement Learning.”Proceedings of the 33rd International Conference on Machine Learning (ICML), 1928–37.
Scaling policy gradients to neural networks exposed instability rather than bias. Trust region policy optimization bounded each update’s divergence from the policy that collected the data (Schulman et al. 2015), generalized advantage estimation made the bias-variance trade in the baseline explicit (Schulman et al. 2016), and proximal policy optimization replaced the trust region with the cheaper clipped objective that GRPO and many later methods inherit (Schulman et al. 2017). The systems problem arrived in the same years. Stale-synchronous parameter servers bounded how old a worker’s parameters could be (Ho et al. 2013), asynchronous actor-learners kept every actor busy at the price of that lag (Mnih et al. 2016), and IMPALA separated acting from learning with an explicit off-policy correction (Espeholt et al. 2018), a split that agent rollout fleets resemble.
Language models then turned the reward itself into the hard part. Learning a reward model from human comparisons made preferences optimizable (Christiano et al. 2017; Ouyang et al. 2022), and measuring what happens when a policy is optimized against such a proxy showed a held-out gold score eventually declining while the proxy score keeps rising (Gao et al. 2023). Executable checks offered a sturdier signal: unit tests became the reward for code generation (Le et al. 2022), outcome and process supervision were compared on checkable reasoning (Uesato et al. 2022; Lightman et al. 2024), and the name reinforcement learning with verifiable rewards was attached to training against such checks (Lambert et al. 2024). On the optimizer side, sampled baselines that score each output against its siblings (Kool et al. 2019; Ahmadian et al. 2024) led to group-relative policy optimization (Shao et al. 2024). The same years documented the cost of an imperfect reward, from the early catalog of specification problems (Amodei et al. 2016; Krakovna et al. 2020) to a formal definition of reward hacking (Skalse et al. 2022), and the defense returned to the protection literature: confinement (Lampson 1973) and complete mediation (Saltzer and Schroeder 1975) describe the verifier boundary in terms that predate learned policies entirely. For further reading, six works trace that lineage from classical learning theory to agent training:
Christiano, P. F., J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. 2017. “Deep Reinforcement Learning from Human Preferences.” In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, edited by Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett. Curran Associates.
Ouyang, L., J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, et al. 2022. “Training Language Models to Follow Instructions with Human Feedback.”Advances in Neural Information Processing Systems (NeurIPS) 35: 27730–44. https://doi.org/10.52202/068431-2011.
Gao, Leo, John Schulman, and Jacob Hilton. 2023. “Scaling Laws for Reward Model Overoptimization.”Proceedings of the 40th International Conference on Machine Learning 202: 10835–66.
Uesato, Jonathan, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. “Solving Math Word Problems with Process- and Outcome-Based Feedback.”arXiv Preprint arXiv:2211.14275, ahead of print. https://doi.org/10.48550/arXiv.2211.14275.
Lightman, Hunter, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. “Let’s Verify Step by Step.”The Twelfth International Conference on Learning Representations (ICLR).
Lambert, Nathan, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, et al. 2024. “Tülu 3: Pushing Frontiers in Open Language Model Post-Training.”arXiv Preprint arXiv:2411.15124.
Shao, Zhihong, Peiyi Wang, Qihao Zhu, et al. 2024. “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.”arXiv Preprint arXiv:2402.03300, ahead of print. https://doi.org/10.48550/arXiv.2402.03300.
Amodei, D., C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. 2016. “Concrete Problems in AI Safety.”arXiv Preprint arXiv:1606.06565, ahead of print. https://doi.org/10.48550/arXiv.1606.06565.
Krakovna, Victoria, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom McGrath, Celestine Mendler-Dünner, Kyle Haslett, et al. 2020. “Specification Gaming: The Flip Side of Specification Design.”arXiv Preprint arXiv:2011.08827.
Lampson, Butler W. 1973. “A Note on the Confinement Problem.”Communications of the ACM 16 (10): 613–15.
Sutton and Barto (2018) is the standard text on returns, value functions, temporal-difference learning, and policy gradients. It supplies the vocabulary every section of this chapter assumes.
Williams (1992) introduces the REINFORCE estimator and the reinforcement baseline. Reading it shows how little the core update has changed since.
Schulman et al. (2017) presents the clipped objective, read together with Schulman et al. (2016) for advantage estimation. The pair explains what GRPO keeps and what it removes.
Espeholt et al. (2018) is the reference design for separating actors from learners with a bounded, corrected lag. Its staleness problem is the one agent rollouts make worse.
Skalse et al. (2022) gives a precise definition of when a proxy reward can be hacked. It turns the informal worry of this chapter into a property a reward designer can reason about.
Saltzer and Schroeder (1975) states the protection principles that verifier isolation applies. Reading it shows why an instruction to the policy is never a substitute for mediation of every route to the verdict.
Sutton, Richard S., and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. 2nd ed. MIT Press.
Williams, Ronald J. 1992. “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning.”Machine Learning 8 (3–4): 229–56. https://doi.org/10.1007/BF00992696.
Skalse, Joar, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. “Defining and Characterizing Reward Gaming.”Advances in Neural Information Processing Systems (NeurIPS) 35: 9460–71.
Saltzer, J. H., and M. D. Schroeder. 1975. “The Protection of Information in Computer Systems.”Proceedings of the IEEE 63 (9): 1278–308. https://doi.org/10.1109/proc.1975.9939.
17.14 Exercises
Every given below is a hypothetical problem input rather than a measurement of a real system. Each problem ends in a decision, and the calculation exists to justify it.
False accepts early and late in training (Basic). A repair-task verifier accepts every criterion success (\(\VerifierTrueAcceptanceRate=1\)) and a fraction \(\VerifierFalseAcceptanceRate=0.02\) of criterion failures. At the start of training the policy passes the reference criterion with probability \(\ValidCandidateRate=0.04\); after several hundred updates it passes with \(\ValidCandidateRate=0.40\).
Use equation 1 to compute the false-accept share of rewarded rollouts at both pass rates.
Find the largest \(\VerifierFalseAcceptanceRate\) that keeps the share at or below \(0.10\) when \(\ValidCandidateRate=0.04\).
The team can afford one verifier-hardening effort, either before training starts or after the pass rate has risen. Decide when to spend it and justify the choice from parts (a) and (b).
Keeping the pass above the failure (Basic). A proposed terminal score is \(\TrajectoryReward=\CorrectnessReward+\FormatRewardWeight\FormatComplianceScore-\LengthPenaltyWeight\text{len}\) with \(\FormatRewardWeight=0.4\), \(\LengthPenaltyWeight=0.7\), and \(\text{len}=\RolloutPolicyTokenCount/\GeneratedTokenCap\) for an enforced cap \(\GeneratedTokenCap=8{,}000\) generated tokens.
Score a pass that uses \(7{,}200\) tokens with \(\FormatComplianceScore=0.5\), and a failure that uses \(400\) tokens with \(\FormatComplianceScore=1\). State whether this pair keeps the intended order.
Compute the lowest possible pass score and the highest possible failure score over the stated ranges, and state whether every observed pass outranks every observed failure.
Holding \(\FormatRewardWeight=0.4\), find the range of \(\LengthPenaltyWeight\) that satisfies the sufficient ordering condition, and explain why part (a) alone was not evidence that the original weights were safe.
Sizing an all-pass rule (Basic). A timing-sensitive fixture passes a fixed incorrect patch with per-trial probability \(\IncorrectCandidateTrialPassProbability=0.30\) and a fixed correct patch with \(\CorrectCandidateTrialPassProbability=0.95\). Trials are independent, acceptance requires \(\VerificationTrialCount\) total passes, and verification stops at the first rejection.
Find the smallest \(\VerificationTrialCount\) that brings an incorrect patch’s acceptance probability below \(0.01\).
At that \(\VerificationTrialCount\), compute the false-rejection probability for the correct patch and the expected number of trials for each patch.
A lone false pass on a hard task receives the largest group coefficient. Compare adding trials with repairing the fixture’s timing sensitivity, and state which the loop should do first.
A mixed task pool and its silent groups (Intermediate). An update samples groups of \(\GroupSize=8\) over \(256\) tasks. Half the tasks belong to a family the policy passes with \(\AttemptPassProbability=0.02\); the other half to a family it passes with \(\AttemptPassProbability=0.60\). Runs are independent and rewards are binary.
Compute \(\GroupContrastProbability\) for each family, the expected number of informative groups per update, and the rollouts spent per informative group across the whole pool.
Compute \(\GroupContrastProbability\) from the pool-average pass rate instead, and explain the size of the error.
For the hard family, compute the coefficient a lone pass receives with \(\AdvantageStabilizer\to0\). A colleague proposes dropping the hard family to cut the rollout bill. Critique the proposal, naming what it does to the training distribution, and propose an alternative that raises the hard family’s contrast without discarding it.
A continuation-sampling budget (Intermediate). A repair trajectory runs \(\TurnHorizon=40\) turns of \(\TurnGeneratedTokenCount=400\) generated tokens. The team may spend at most \(30\) times the base trajectory’s generated tokens on credit estimation, including the base. Continuations are independent binary returns and are not recursively forked.
Compute the generation multiple for exhaustive forking with \(\ContinuationCount=16\) continuations per boundary, and for forks after turns \(10\), \(20\), and \(30\) only.
Find the \(\ContinuationCount\) that brings the per-fork standard-error bound to \(0.05\), and the multiple that \(\ContinuationCount\) costs at the three chosen forks.
Find the largest \(\ContinuationCount\) the budget allows at those three forks and its standard-error bound. Treat a difference between two fork values as resolved only if it is at least twice the bound on the standard error of the difference, \(\sqrt2/(2\sqrt \ContinuationCount)\). Decide whether this schedule can resolve a value difference of \(0.20\), and if it cannot, redesign the schedule within the same budget.
Reset for a short-horizon support agent (Intermediate, second-lighthouse transfer). Apply the slot model of equation 6 to training the support agent on refund tasks, where a deterministic check compares the issued refund with the recorded policy. An episode has \(\TurnHorizon=4\) decisions with \(\ModelTime=0.6\) s of model service and \(\ToolTime=0.1\) s of record lookup each, then \(\RolloutAssessmentTime=0.2\) s of assessment. Restoring the customer record and refund history from a cold fixture takes \(\EnvironmentResetTime=6.0\) s; a snapshot restore takes \(0.3\) s. The pool has \(64\) slots, and the shared model service can sustain at most \(40{,}000\) episodes per hour.
Compute the cycle time and the reset share of the cycle for both resets, and the idealized pool capacity in episodes per hour.
Compute the capacity gain that the shared model service actually permits, and decide whether the snapshot work or added model-service capacity should come next.
In the chapter’s repair-pool example, cold reset was under half of the cycle. Compare this lighthouse’s reset share, explain which term of equation 6 makes reset the lever here, and name the state a local snapshot does not restore if a refund reached an external payment service.
Staleness bounds and verifier provisioning (Advanced). A repair-agent trainer steps every \(\TrainingStepTime=40\) s and admits a record only if \(\FiniteChange\text{ver}\le2\). Of the rollouts, a fraction \(0.70\) take \(30\) s from first token to admission, \(0.25\) take \(90\) s, and \(0.05\) take \(200\) s. Verification receives \(256\) completed trajectories per training step. Every trajectory needs up to \(\VerificationTrialCount=3\) verification trials of \(\VerificationTrialTime=10\) s, and the pool targets \(\TargetServingUtilization=0.8\).
Compute \(\FiniteChange\text{ver}\) for each duration class and the fraction of rollouts excluded. Explain why the excluded fraction understates the damage to hard tasks.
Find the step time above which the \(200\) s rollouts are admitted, and the duration below which a rollout is admitted at the current step time. Compare slowing the trainer, capping rollout duration, and shortening \(\FiniteChange \RolloutDuration\).
Size the verifier pool when all three trials execute. Then size it under stop-on-first-rejection when a fraction \(0.60\) of candidates are incorrect with \(\IncorrectCandidateTrialPassProbability=0.10\) and the rest are correct with \(\CorrectCandidateTrialPassProbability=0.97\).
Size the stop-on-first-rejection pool again by substituting the population’s mean pass probability into every power, and state the consequence of provisioning from that number.
Diagnosing a lengthening policy (Advanced, critique and redesign). During GRPO training with per-rollout token averaging, the mean length of failing repair rollouts grows from \(2{,}000\) to \(6{,}000\) generated tokens while acceptance stays flat. The team proposes adding \(\LengthPenaltyWeight\text{len}\) with \(\LengthPenaltyWeight=0.4\) and \(\FormatRewardWeight=0.2\), where \(\text{len}=\RolloutPolicyTokenCount/4{,}000\) is normalized by a nominal cap the runtime does not enforce.
Using equation 5, compute the ratio of per-token penalties on a \(2{,}000\)-token failure and a \(6{,}000\)-token failure that share the same \(\GroupRelativeAdvantage_i\).
Score a \(9{,}000\)-token pass with \(\FormatComplianceScore=0\) and a \(500\)-token failure with \(\FormatComplianceScore=1\) under the proposal. Find the pass length beyond which a pass with \(\FormatComplianceScore=0\) scores below the best-scoring failure.
Critique the proposal and redesign the response. The redesign should state which diagnostic runs before any penalty is added, what the runtime must enforce, and which condition any length term that survives must satisfy.
What’s Next: From a measured single agent to coordination
Splitting a task across several agents is justified only when the benefits of decomposition exceed its coordination costs. Every mechanism so far, from the harness to this chapter’s repair group, governs one trajectory, and the connection-pool race that group repaired fits in one controller’s context. The repair agent’s next assignment is wider: migrating an authentication module across a repository, a change that reaches every caller of its interface and tempts a split. Part V provides ways to curate learning material, update a policy, and test the result, but no system must complete all three stages before asking whether to delegate. Multi-Agent Coordination compares a coordinated design with a capable single-agent baseline that holds the same parallel tools and a comparable resource budget. Delegation can separate work and context, but it also introduces shared-state conflicts, correlated mistakes, and obligations that cross agent boundaries, and the comparison must price those costs while preserving every runtime control already required for one agent.
