Why is an accepted agent trajectory not automatically a good training example?
Every agent run leaves a durable record of what it saw, proposed, and achieved, and a team preparing the next policy is tempted to train on that record directly. A record, however, answers a different question than a training example does. An accepted trial can contain a malformed call that the runtime refused, a failure can stem from a broken tool schema that no weight update repairs, and two attempts at the same task split across training and evaluation turn a test score into a measure of recall. Imitating such records teaches the model its own mistakes and workarounds, and filtering them by outcome alone narrows behavior toward whatever the filter happens to reward. Curation therefore decides, before any gradient is computed, whether a failure belongs in the weights at all, which records may serve which learner, which tokens are targets and which are context, and what evidence and permissions travel with every record derived from the original run.
Learning Objectives
Justify a learning use for a residual failure and distinguish imitation material, preference comparisons, and rollout fixtures
Design versioned collection fixtures and protected splits for a stated generalization claim
Construct admission predicates and compare verification costs without changing their criteria
Assign recovery and negative records to appropriate roles while distinguishing trajectory counts, stored tokens, and loss-bearing tokens
Assess how selection rules and sampling budgets change the diversity and yield of a self-improvement loop
Size bounded collection flow against downstream capacity, and price an emitted artifact by attempt cost and emission yield
Defend a versioned corpus manifest with transformation lineage, permitted uses, and intended consumers
15.1 From Trajectory Records to a Training Corpus
The evaluation harness of Agent Evaluation and the operational record of Operating Agents produced more than a release decision. While they judged the repair agent (Four lighthouse workloads) over repeated attempts on the connection-pool repair task, they also accumulated a durable trajectory log that captures every trial in granular detail: the environment observations delivered to the model, the code edits it proposed, the invalid tool calls rejected by the runtime, the subsequent recovery steps, and the final verdict scored against the task acceptance criteria. Some runs passed cleanly, others failed outright, and several remain stalled in pending states because regression suites or accounting checks timed out. To an engineering team preparing the next iteration of the policy, these accumulated interaction logs look like the natural, abundant raw material for fine-tuning.
Curation changes what the policy learns from its records and leaves every exposure where the runtime closes it.
The machinery built so far, however, does not specify how to translate raw logs into training data. The acceptance contract defined in The Task Acceptance Contract evaluates the final outcome of an entire trial rather than assigning credit to individual tokens or tool calls within it. Consequently, an accepted verdict alone cannot distinguish between an optimal tool call and an accidental stumble that the agent managed to survive. Imitating raw logs reproduces whatever they contain. If an agent wastes ten calls exploring invalid syntax before hitting the correct command, blind imitation trains the next model to repeat that exploratory churn. Furthermore, rigorous evaluation requires held-out task distributions that the training process never touches. If an engineer partitions data naively at the trace level, different attempts on the same underlying task fixture land on both sides of the train-test split, silently destroying evaluation integrity. Finally, because the agent generated these records under its own behavioral distribution, any filtering rule directly shapes what the subsequent policy will produce, inducing feedback dynamics across training generations.
Bridging the gap between raw execution logs and active model training requires a dedicated systems stage. Trajectory curation selects, labels, and transforms recorded interactions for a declared learning use while preserving the evidence and constraints that justify that use. Where evaluation establishes whether a trial satisfied its specification under fixed conditions, curation resolves a different operational question: which behaviors and tasks the learning process should absorb, which steps must be masked or rewritten, and how to preserve statistical isolation across splits. The end product of this pipeline is a versioned, reproducible corpus tailored to specific training consumers, accompanied by strict provenance records rather than an ad hoc collection of success flags. Transforming these admitted records into tokenized inputs and loss objectives becomes the downstream responsibility of Trajectory Fine-Tuning.
This chapter follows the repair agent from diagnosis to a published corpus. A failure caused by a truncated observation is the case whose remedy lies outside the weights, and a recovery from a refused edit becomes the record carried all the way into a corpus. The stages run in the order the risks arise. Before anything is collected, the failure must be shown to belong in the weights (section 15.2), and the evidence that will later judge the trained model must be fixed and protected (section 15.3, section 15.4). Collected records then pass admission checks specific to the learner that will consume them (section 15.5), recovery records are divided into context and targets (section 15.6), and the feedback created when a policy generates its own data is bounded (section 15.7). The remaining sections keep the collector within its capacity (section 15.8), bind every record to its lineage (section 15.9), and carry one record through the whole chain (section 15.10). The first decision is triage, because a failure that does not belong to the model should never reach a corpus.
15.2 Capability Gap Diagnosis
In Agent Evaluation, the repair agent’s trial on pool_repair_01 submitted workspace ws_r19 with its targeted reproducer passing, while regression and resource-accounting checks remained pending, leaving the run as neither an accepted repair nor a valid demonstration. A separate trial of the same task failed outright after the observation interface silently truncated a search result, and the postmortem’s controlled ablation traced most of that failure to the interface: once pagination was added, every continuation from the saved prefix reached an accepted repair, even though the ablation did not prove the model blameless. A team that had collected those truncated-observation runs for training would have taught the model to work around an interface defect that a single adapter change removed. An execution failure belongs in the trajectory curation pipeline only after diagnosis confirms that model weights are the right target for repair. Modifying weights is justified only when confronting a genuine capability gap: an operational failure that persists even when the model receives complete context, interacts through a well-formed interface, and runs in a healthy runtime with adequate resources. Every other failure stems from an external system defect that calls for a cheaper, far more reversible engineering fix.
↩︎ The controlled ablation that traced the truncated-observation failure to the interface is established in Controlled diagnostic experiments.
15.2.1 Diagnosis and the learning objective
When the repair agent failed on pool_repair_01, the final failure verdict alone revealed nothing about which subsystem broke. The postmortem analysis in Incident Postmortems separated three distinct events within that truncated-observation incident: an interface defect (the search excerpt arrived without truncation metadata), a forbidden action proposal (the next response proposed replacing a file outside the repair scope), and a successful runtime refusal (the admission gate rejected that proposal before dispatch). Such findings often coexist within a single execution trace. Adding missing truncation metadata repairs the interface contract; retaining the admission guard enforces system boundaries against out-of-scope dispatches; and training the policy to seek additional evidence addresses model behavior. Each diagnostic finding demands its own engineering remedy.
Table 1 connects diagnostic findings to candidate system interventions and their corresponding learning objectives. Because multiple failure modes frequently overlap in production traces, engineers must evaluate each candidate intervention under realistic operating conditions. The sequence in which teams investigate and apply these remedies follows directly from their computational cost and operational reversibility.
↩︎ Failure attribution, and the limit of what each finding shows, is established in Fault attribution.
Table 1: Diagnostic Findings and Learning Uses: A direct system repair and a behavioral learning objective can both be justified by the same incident. Each proposal needs its own evidence and scope.
Diagnostic finding
Direct system intervention to investigate
Possible learning use and its limit
A required fact was absent from the delivered context
Repair retrieval or assembly and preserve source and omission status
Teach the model to recognize insufficient evidence and request it; weights cannot supply a reliable current fact that was never observed
A tool description or response violated its interface contract
Correct the schema, documentation, or observation adapter
Teach interpretation of valid status and recovery information; learned habits do not repair the interface itself
An operation encountered an execution failure or unresolved outcome
Investigate endpoint behavior, resource bounds, and recovery ownership
Teach bounded recovery or escalation under represented uncertainty; a retry proposal cannot establish that the earlier effect failed
The model repeatedly chose an unsupported or ineffective continuation
Compare prompts, examples, and targeted policy updates on declared tasks
Collect justified alternative actions or recovery targets; repeated failure alone does not identify a unique cause or prove that training will help
15.2.2 The intervention ladder
Fine-tuning a model to compensate for an omitted context fact or an ambiguous tool schema represents premature fine-tuning, an expensive waste of compute and curation effort on a defect that a prompt or schema fix cures in minutes. To prevent that misallocation, the intervention ladder orders failure remedies by ascending engineering cost and descending reversibility: context retrieval and assembly, tool definitions and schemas, runtime execution controls, supervised fine-tuning (updating weights to imitate selected demonstrations), and reinforcement learning (updating weights to raise a verified reward on fresh rollouts). Engineers test the inexpensive, reversible rungs first by default, advancing to a weight update only when diagnostic evidence establishes a genuine capability gap and confirms that training improves the behavior rather than masking an interface or runtime defect.
This ordering reflects how cost, reversibility, and enforcement guarantees change across the stack. Engineering and compute costs rise sharply at each rung: editing a prompt assembly pipeline, schema definition, or runtime timeout modifies version-controlled configuration that an engineer can test against integration fixtures in minutes, whereas retraining weights requires curating a trajectory corpus, running accelerator jobs, and re-evaluating every capability the model serves rather than only the failing task.
Those cost differences are compounded by reversibility and blast radius. A defect repaired at a lower rung reverts cleanly as a configuration change whose impact is limited to the callers of the changed tool or prompt. Updating model weights, by contrast, alters behavior across the entire input space, and rolling back a checkpoint discards every concurrent improvement alongside the targeted fix. Reinforcement learning sits at the top of the ladder for the same reason: beyond the dataset and validation demands of supervised fine-tuning, it requires distributed rollout fleets, isolated reward verification harnesses, and optimization controls that keep the policy from exploiting loopholes in the reward check.
Even when a weight update succeeds, statistical weights cannot replace runtime enforcement. A fine-tuned policy may propose out-of-scope tool calls far less often, but lower observed failure rates do not guarantee compliance on every permitted input. Invariant closure (principle 1) requires that execution runtimes never assume policy compliance; resource budgets, authorization checks, and process sandboxes remain hard boundaries regardless of how thoroughly the model has been trained.
The end-to-end argument (Saltzer et al. 1984) yields the same conclusion for tool interfaces: a function belongs where it can be implemented completely and correctly. A typed action contract (principle 7) provides enforceable guarantees across any model version placed behind it. By contrast, fine-tuning a model to guess fields omitted by an API schema teaches brittle habits that break as soon as the interface changes. An integration test on an observation adapter can verify directly that truncated outputs return explicit pagination markers, resolving the ambiguity at the interface layer without asking downstream models to guess what was cut.
Empirical evidence, rather than administrative habit, decides when a failure climbs the ladder. Engineers test each lower rung with a counterfactual probe, a controlled re-execution of the recorded failure trace with that rung’s suspected defect removed. A context probe begins by injecting the missing fact directly into the prompt, whereas an interface probe resumes matched saved prefixes with corrected tool descriptions or schemas and samples fresh continuations under the controlled conditions of Statistical Evaluation Rigor (since replaying a fixed proposal would test only the validation or adapter boundary, not whether the model chooses a different action). If both checks leave the failure intact, a runtime probe runs the identical tool payload inside an isolated sandbox with relaxed resource limits. Whenever a failure disappears under one of these probes, the result supports a repair at that rung under the probed conditions without ruling out other contributing causes. Probing in ladder order routes each resolved defect to an immediate, reversible repair, so only the failures that survive all three probes reach the weights (figure 1). Even then, checkpoints trained on this residual must pass the evaluation protocol of Statistical Evaluation Rigor rather than ship merely because model weights were the last hypothesis left.
Figure 1: Probes Before Weights: Under the default diagnostic route, counterfactual probes evaluate a measured failure in ascending ladder order. A failure that disappears under a probe routes to an immediate systems repair, modifying versioned artifacts that revert cleanly as configuration. Only failures that survive every lower-rung probe reach the weight modification stages, where supervised fine-tuning precedes reinforcement learning. Engineering cost rises and operational reversibility falls from top to bottom.
The intervention ladder is a diagnostic discipline rather than a rigid queue. When counterfactual probes immediately expose a capability gap in an agent that fails despite complete context, a clear schema, and a healthy runtime, engineers can skip lower-rung changes and proceed directly to targeted training, provided the empirical evidence of the gap is recorded first. Moreover, production environments routinely return partial outputs, transient network timeouts, and service-unavailable errors; training an agent to handle those degraded conditions is a valid behavioral objective, not an excuse to leave broken interfaces unfixed. Notebook 15.1 prices the compute and reliability trade-offs across this hierarchy for a representative failure workload.
Napkin Math 15.1: Pricing the ladder for one failure mix
Problem: A hypothetical maintenance agent executes 50,000 trajectories per month, with 6 percent failing (3,000 total failures). Counterfactual probes attribute 70 percent of the failures to a tool schema that omits a required field and 20 percent to sandbox startup timeouts; only the remaining failures survive every probe. One engineering team repairs the tool schema and scales the sandbox pool before considering training. A second team bypasses system repairs and fine-tunes the base model on all failing trajectories using corrected demonstration targets, training on 4,096-token sequences across 3 epochs. What failure modes does each path eliminate, and what is the direct compute cost of the training run?
Variables: The underlying model contains 70.6 billion parameters. Standard transformer training requires approximately 6 floating-point operations (FLOPs) per parameter per token (\(6\Parameters \DatasetSize\)). A dedicated cluster node of 8 accelerators provides a dense 16-bit compute peak of 989 TFLOP/s per device while sustaining 30 percent model FLOPs utilization (MFU). Accelerator rental costs $4 per hour. Both paths assume that diagnostic attribution is correct and exclude data curation, annotation, and software engineering labor.
Math: System-level repairs eliminate the 2,700 failures attributed to interface and runtime defects (90 percent of all failures), deploying and reverting cleanly as versioned configuration updates. In contrast, fine-tuning indiscriminately across all 3,000 failures trains on 3,000 \(\times\) 4,096 \(\times\) 3 = 36,864,000 tokens. Evaluating \(6 \times \text{parameters} \times \text{tokens}\) yields 1.56 × 10¹⁹ FLOPs of total training compute. At sustained cluster throughput, this workload requires 1.8 h of wall-clock time (14.6 accelerator-hours), resulting in an accelerator invoice of $58. Fine-tuning exclusively on the 300 residual failures that survived diagnostic probing costs only $6.
Result: The direct accelerator invoice of $58 is modest, so raw compute expenditure does not govern intervention ordering. The decisive factor is the architectural debt and reliability profile left in production. Under premature fine-tuning, the tool schema still omits the required field and the sandbox pool still times out under load; consequently, those operational failures return whenever the model guesses incorrectly or cluster load spikes. Furthermore, teaching the model to work around missing fields creates fragile behavioral compensations that collapse as soon as the interface changes. Its training corpus also squanders curation and review effort on incidents whose proper remedy was never behavioral. Under the systems repair path, engineers resolve 90 percent of all failures permanently for every model checkpoint, leaving a residual of 300 failures whose curated corpus is focused entirely on genuine policy gaps.
Systems insight: The primary risk of premature fine-tuning is rarely compute expenditure; it is architectural misattribution. While an invoice of $58 appears manageable, training cannot fix the 90 percent of failures rooted below the weights. Only a systems repair resolves those defects permanently across all current and future model generations. The governing engineering principle is to attribute failures with counterfactual probes first, remediate what lies below the weights in software, and train exclusively on the residual capability gap that survives.
Once diagnostic probing isolates a genuine residual failure, engineers must specify the exact behavior to train before gathering examples. For the repair agent, the paginated interface now supplies completeness metadata, but a policy can still ignore it; runs on the repaired interface that receive that metadata and still draw a conclusion from a partial result form the residual that a learning objective can target. An effective objective does not aim vaguely to “fix failed trajectories.” Instead, it trains the policy to inspect the completeness metadata of a search result, issue a permitted follow-up query when pagination indicates more records exist, and state remaining data boundaries when limits are reached. Evaluating the fine-tuned model requires checking both the final task outcome and the efficiency of the path, including redundant reads, context token use, and unsupported intermediate claims. A policy can easily learn a circuitous investigation habit that raises task acceptance at the cost of high latency and token spend, or an overly aggressive shortcut that skips required checks; neither flaw shows up in cross-entropy training loss alone.
15.2.3 Learning artifacts
A curation pipeline that keeps only clean, end-to-end successes throws away most of its diagnostic signal: a trace that stumbled on a malformed tool call before recovering still demonstrates how to repair the mistake, and a run that failed outright on a subtle race condition still captures a valuable test environment. Depending on what evidence a trace preserves, a single recorded interaction can feed three distinct learning artifacts, each governed by its own admission criteria:
Imitation material consists of complete demonstrations or extracted trajectory segments whose decision steps represent exemplary behavior under the supplied context. When curating an error-recovery sequence, the ingestion pipeline can retain an initial malformed tool proposal and its runtime rejection within the conversational history, while selecting only the subsequent corrected action as the supervised target. Faithfully preserving an error in conversational context provides realism, but supervising a model to imitate an unnecessary retry loop degrades runtime efficiency. Curation explicitly designates the target tokens to be reinforced.
A preference comparison pairs alternative trajectories generated from a shared initial context, attaching an explicit, justified preference ordering. Two runs of a task form a valid comparison pair only when their initial state, environment configuration, and accessible action spaces are identical. Furthermore, a failing or degraded run belongs in the rejected slot only when an evaluator records explicit diagnostic criteria explaining why its trajectory was inferior. These supporting assessments, tie-breaking criteria, and conditioning contexts allow offline preference optimization methods to construct statistically sound loss objectives.
↪ The trajectory preference loss, which scores the preferred and dispreferred records against a frozen reference policy, is developed in The trajectory preference loss.
A rollout fixture bundles a reproducible task environment, versioned configuration fixtures, and an automated grading specification to support online exploration during reward-based reinforcement learning. An unsuccessful execution attempt often reveals a valuable environmental corner case even when none of its recorded actions warrant imitation. In this setting, the trace does not provide demonstration data; rather, it seeds a versioned simulation fixture that recreates the problematic state so the policy can explore alternative trajectories and optimize against the reward specification.
↪ Optimizing a policy against verifiable checks on the rollouts it executes in such an environment is developed in Foundations of Verifiable Rewards.
These three downstream roles explain why a curation pipeline preserves execution failures rather than discarding them. A single flawed trajectory can support diagnostic root-cause analysis, provide multi-turn context for an error-recovery demonstration, serve as a negative example within a preference comparison, or seed a simulation fixture for online reinforcement learning. The intended downstream artifact dictates the admission evidence required to validate the record. Symmetrically, successful trajectories that produce no state mutations (an agent correctly refusing an out-of-scope request or answering a read-only query) constitute valid imitation demonstrations whenever the task specification requires restraint. Producing a nonempty workspace modification is never an inherent admission requirement.
Every proposed trajectory collection campaign must formally document four elements: the specific behavioral capability to be trained, the target artifact format, the empirical evidence required for admission, and the held-out evaluation claim that will assess the trained model. When trajectory evaluations rely on automated model judges or human annotators, those evaluations must operate under explicit rubrics with evaluator coverage and scoring variance fully exposed.
Curation produces defensible artifacts for declared uses, while held-out evaluation decides whether the resulting model update actually improves the system. That separation is the decision boundary of verified trajectory post-training (principle 13), which holds that a recorded outcome does not by itself determine how a trajectory may be used for learning. Two obligations follow before any record is collected. Fixtures must translate the desired behavior into collectible tasks, and the evaluation split must be reserved first, so that data selection never consumes the evidence later used to judge it.
15.3 Task Fixture Design
When an agent collection worker attempts a complex task such as connection-pool repair repeatedly, it can generate a distinct candidate solution on every run. If an attempt inherits residual state from an earlier repair attempt, however, it no longer demonstrates solving the problem from the declared initial state. Conversely, if a collection pipeline records only the final passing patch, it discards the sequence of tool invocations, intermediate terminal outputs, and error-recovery decisions that explain how that patch was derived. A collection fixture prevents both failures by preserving reproducible task conditions while recording the comprehensive execution evidence demanded by downstream learning algorithms.
The task fixture of Task fixtures already fixes the execution baseline (initial state, dependencies, permitted interfaces, variation policy, evidence boundary, and reset). Trajectory curation layers a collection contract on top of that fixture. The collection contract specifies why the task belongs in the corpus, which interaction records the ingestion pipeline may retain, and what intermediate evidence downstream admission filters will require.
15.3.1 The collection manifest
If a new collection run on pool_repair_01 reuses the trial identity of ws_r19, overwrites the unresolved record of test_r18, or lets engineers adjust admission rules after inspecting the sampled outputs, the pipeline corrupts its own audit trail. Separating the versioned identities of the task specification (pool_repair_01), execution fixture (pool_fixture_v1), and evaluation criteria (repair_accept_v1) prevents that collision. Each new collection attempt receives its own run identity and produces an independent candidate artifact, while the historical evaluation candidate ws_r19 stays permanently bound to its original trial, and a newly passing test run never alters an earlier unresolved record such as test_r18.
The collection manifest references the underlying evaluation fixture and records the operational decisions summarized in table 2. Defining these fields before sampling begins ensures that data engineers fix the intended training role and capture requirements in advance, rather than biasing the sample or fitting admission rules retroactively to observed outputs.
Table 2: Collection Decisions Attached to a Fixture: The collection manifest extends the experimental contract with intended use, relationships, and evidence requirements. Actual admission follows inspection of the resulting record.
Collection field
Decision recorded
Consequence for the repair
Learning use
Proposed behavior to improve and intended artifact consumer
Collect evidence for interpreting failed checks and choosing a justified next action, not merely final patches
Source and permitted use
Task origin, access conditions, retention scope, and applicable restrictions on derived data
Permission to inspect a repository does not by itself authorize exporting its contents into a corpus
Task relationships
Stable task identity, source lineage, and links to variants or shared solutions
Repeated attempts and derived variants remain identifiable when splits are assigned
Collection configuration
Sampling system, prompt and tool versions, allowed variation, and attempt-selection policy
Distinguish a policy change from a changed task or collector
Required capture
Delivered context, proposals, matched observations, candidate identity, and criterion-level evidence needed for the proposed use
Preserve what supported each decision, including incomplete observations and rejected proposals
Admission policy
Versioned conditions for accepting a record into each proposed artifact, with unresolved cases handled explicitly
A targeted pass does not become whole-task acceptance or automatic demonstration eligibility
What the manifest must capture depends on the consumer that will train on the artifact. A demonstration selected for supervised error recovery must preserve the diagnostic context, error messages, and intermediate inspection steps that explain why a corrective action was taken. An offline preference comparison needs matched decision conditions and environment state across paired rollouts so that the comparison isolates policy differences. A fixture built for online reinforcement learning rollouts must instead expose live, resettable environment instances and versioned reward harnesses rather than static text logs. Because the target loss function decides how admitted trajectory data is tokenized and masked, a trace that lacks certain tool outputs may fail admission for imitation yet remain useful for failure diagnosis. Curation pipelines therefore tailor their logging schemas to the declared learning objective rather than relying on a single untyped event log, and verify that the grading harness itself is trustworthy before collection begins.
15.3.2 Adequate and inadequate controls
An automated grader that accepts an invalid solution pollutes a training corpus with false-positive demonstrations, while a grader that rejects valid work starves the collection pipeline. Before launching data collection at fleet scale, engineers calibrate grading harnesses against independently verified adequate and inadequate controls, candidate implementations whose expected verdicts are known in advance.
↩︎ Measuring a grader’s false-acceptance rate against independently assessed calibration cases is established in Grader error rates.
For the connection-pool repair task, an inadequate control is a candidate that deliberately keeps the target defect, such as leaving worker threads blocked after a connection returns to the pool (the missed-wakeup bug). This buggy candidate must reliably trigger a failure in the reproducer suite. Conversely, an adequate control is an independently reviewed reference patch that resolves the issue cleanly; it must consistently pass the test harness. If the reproducer accepts the buggy control or rejects the reference repair, the failure points to a flaw in the test harness, flaky environment state, or broken process exit bindings that must be fixed before collection proceeds. Each control isolates and tests a specific grading criterion: a run that produces the reference repair must satisfy the full suite of regression, execution, resource, and residual-duty checks defined in repair_accept_v1, while separate controls check that the agent does not alter protected configuration files or emit premature completion signals.
The harness that grades collection runs executes outside the agent’s authority boundary, satisfying invariant closure (principle 1). It keeps the grader isolation of Grader isolation, so candidate code cannot rewrite tests, forge exit codes, or read hidden assets, and section 15.5.2 develops what admission adds to that boundary. Whenever engineers modify a fixture, update a grading rubric, or shift the target task distribution, they validate the changes against fresh control suites rather than inheriting prior trust.
Controls matter equally for tasks that change no persistent state. Consider a read-only evaluation fixture, report_comparison_v1, containing two immutable experiment reports. Report A documents that an optimization method was evaluated on Linux; Report B documents that an alternative method was evaluated on Windows. Neither report contains comparative throughput or latency measurements. The agent’s prompt asks for the documented platform coverage and asks whether the reports show which method achieves higher throughput. The task’s grading criteria demand accurate, source-specific attribution, an explicit statement that the throughput comparison is unsupported by the evidence, and strict compliance with read-only access limits.
Here, an adequate control is an independently validated response that maps each platform to its source and declines to declare a throughput winner. Inadequate controls include responses that swap the platform attributions, invent throughput claims, or return an empty answer that omits the documented facts. Because the filesystem stays bit-for-bit identical across all runs, diff-based checks cannot evaluate success; the grading harness must instead run structured field assertions and rubric checks to verify factual attribution and confirm that the agent stated the limits of the evidence. Automated curation can therefore harvest examples of grounded abstention without granting write permissions or requiring workspace patches, across whatever trajectory sources feed the collector.
15.3.3 Trajectory sources
Relying on a single source of trajectories leaves systematic blind spots in a training corpus: production logs overrepresent routine queries and abandoned user sessions, human demonstrations are too slow to cover wide fault matrices, and synthetic rollouts repeat whatever strategy the current policy already favors. Trajectory harvesting therefore draws from all three origins—human demonstrations, production telemetry, and synthetic model rollouts—balancing what each contributes against the collection risk it carries (table 3).
Table 3: Trajectory Source Trade-offs: Sources contribute different evidence and coverage. Each still requires an appropriate capture, assessment, and permitted-use contract.
Source
Useful contribution
Collection limitation to address
Human demonstrations
Deliberately chosen strategies and explanations for difficult decisions
Expertise does not guarantee correct actions, faithful capture, or representative task selection
Production records
Tasks and failures encountered under actual operating conditions
Retention and user selection can bias the sample; missing evidence and use restrictions can limit reuse
Model-generated runs
Repeated attempts and controlled variations on specified fixtures
The sampler and grader influence which behaviors appear and survive; repeated success can concentrate on one strategy
These three sources complement one another across the lifecycle of a training pipeline: a production failure reveals a real-world edge case that warrants reproduction in a benchmark, a human demonstration provides an expert recovery strategy to seed the model, and synthetic rollouts explore fault matrices through automated variation. When generating those synthetic rollouts, the environment must compute state transitions dynamically. If an agent emits an action that diverges from a historical trace, the platform cannot replay the next observation from a recorded log; doing so breaks environment causality and feeds contradictory context into the trajectory. Synthetic collection requires a live, interactive environment that generates valid observations for whatever action sequence the policy proposes.
The choice of data source shapes the distribution of harvested behaviors. A collection policy that filters strictly for short, successful runs discards trajectories that show backtracking and fault recovery under partial observability, whereas relying uncritically on production traces overrepresents abandoned sessions and unresolved timeouts. Recording the collection policy, sampling parameters, and task-family metadata in the manifest lets downstream curation stages distinguish genuinely rare engineering events from artificial gaps created by collection filters, and sets up the question of how broadly the task fixtures themselves span the domain.
15.3.4 Task population coverage
A repair corpus collected only on shallow, single-file bugs with clear stack traces produces a policy that stalls as soon as production code hides a race condition behind deep dependencies or ambiguous test output. To match operational conditions, task distributions must vary systematically across the dimensions that make tasks hard in practice. In software repair, fixtures should vary across dependency tree depth, the clarity of error feedback before an edit, and the concurrency conditions encountered during testing. In document synthesis, fixtures should deliberately include contradictory sources, missing numerical metrics, and schema drift across versions.
The H·S·A exposures (The H·S·A Exposure Space) characterize each variant: the horizon \(\TurnHorizon\) in turns required to solve the task, the state level (\(\StateExposureLevel_0\) to \(\StateExposureLevel_3\)) the task retains or mutates across steps, and the authority level (\(\AuthorityExposureLevel_0\) to \(\AuthorityExposureLevel_3\)) its tool set grants. Varying a task along these exposures provides a concrete basis for task variation that avoids arbitrary context-length thresholds or the false assumption that all meaningful tasks must mutate a filesystem.
System privileges and tool access grants remain explicit and enforced across all task variants. As a model improves, it may learn to query fewer tools, avoid redundant exploratory commands, or complete tasks in fewer turns. That progress changes the agent’s realized exposure to the environment, but the underlying permissions and runtime sandboxing boundaries stay fixed. Similarly, the environment reset contract governs every reuse of a fixture: if a failed run leaves background processes running, uncommitted file locks, or open network sockets, teardown must clear this residual state before the next collection worker starts, whether the previous trajectory was admitted into the corpus or discarded.
↩︎ The reset contract, which catalogs every piece of state that teardown must clear before an environment serves a new lease, is established in The reset contract.
A well-designed collection fixture unifies the task specification and grading criteria with an explicit learning objective, a capture contract, and a structured distribution of variants. Before collection workers run across this task matrix, however, the held-out evaluation tasks must be partitioned and quarantined, so that no collected record can reach the evidence that will later judge the trained model. Reserving those tasks is the focus of section 15.4.
15.4 Split Design
Consider an automated collection pipeline that harvests multiple repair trajectories for the same connection-pool defect. If an engineer naively partitions the resulting execution traces at random, one successful repair trajectory lands in the training set while another attempt at the exact same defect lands in the final test set. Varying the random seed, prompt wording, or tool sequence creates superficial operational diversity, but it does not make the underlying engineering problem new. When the fine-tuned policy subsequently encounters this task during evaluation, it does not demonstrate generalization; it merely recalls an environment, bug mechanism, and solution it has already observed during training. This failure mode is split leakage: information about an evaluation target reaching the training pipeline through a related or derived record (Kaufman et al. 2012).
1Duplication in Code Corpora: Code datasets are especially prone to leakage because repository forks, vendored libraries, and copy-pasted snippets reproduce the same defect across nominally distinct codebases. Evaluating code models on corpora that contain near-duplicates inflates reported accuracy by as much as a factor of two compared with deduplicated corpora (Allamanis 2019). An agent trajectory corpus inherits every duplicate in the underlying codebases and adds a second source of duplication: repeated execution rollouts on the same benchmark fixture. Grouping trajectories by their underlying task is therefore required for valid evaluation rather than an optional refinement.
Allamanis, Miltiadis. 2019. “The Adverse Effects of Code Duplication in Machine Learning Models of Code.”Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!), 143–53. https://doi.org/10.1145/3359591.3359735.
In agent trajectory curation, split leakage is the default outcome rather than a rare edge case.1 Suppose an agent executes 8 rollouts per task and an engineer partitions the individual traces at random using a 80 percent/20 percent ratio. A task remains cleanly isolated on one side of the split only if every one of its attempts independently falls into that same partition. For a training fraction \(\TrainingSplitFraction\) and \(\IndependentAttemptCount\) independent attempts per task, that non-leakage probability is \(\TrainingSplitFraction^\IndependentAttemptCount + (1-\TrainingSplitFraction)^\IndependentAttemptCount\). Consequently, 83.2 percent of all tasks straddle the train–test boundary, contaminating the evaluation protocol. Partitioning schemes must therefore follow structural task relationships rather than arbitrary trace run identifiers.
With eight attempts per task, a random 80/20 trace split leaves 83 percent of tasks on both sides.
While the fixture contract defines what an individual execution trial represents, split design governs what subsequent evaluations can validly claim about agent learning. To preserve that validity, engineers isolate and lock down protected evaluation tasks before data collection, model-assisted filtering, or hyperparameter tuning begins. If developers inspect final test failures and then harvest demonstrations targeting those specific failure modes, they turn the test suite into training feedback, and the benchmark ceases to be an independent evaluation even when no raw test trajectory enters the training manifest.
15.4.1 Evaluation claims and task groups
A split that holds out individual bug tickets within a single repository cannot test whether a fine-tuned agent transfers to an unseen codebase, whereas a split that holds out entire repository families is unnecessarily restrictive when the agent will only ever serve one internal monorepo. Every evaluation split must therefore begin with an explicit claim about the capability under test. An in-distribution deployment check asks whether an adapted agent can resolve new maintenance tickets within the repositories and runtime environments it already serves; an out-of-distribution transfer check asks whether that agent succeeds across unseen repository families, software architectures, or tool environments. Familiarity with a shared build tool, compiler flags, or API schema reflects legitimate deployment conditions and is not leakage, whereas evaluating an agent on a defect whose solution appeared during fine-tuning invalidates the measurement under either claim.
To enforce these boundaries systematically, the curation pipeline organizes tasks into groups. A task group collects all repeated attempts, paraphrased prompts, derived test fixtures, and solution-equivalent variations of a single underlying problem. If a benchmark contains an issue fixture pool_repair_01, any rollout generated from an alternate random seed, a renamed variable, or an identical upstream bug copied into a downstream fork belongs to that same task group. The split manifest records the structural reason for each grouping because simple heuristics fail in opposite directions: grouping solely by repository name misses cross-repository forks and vendored copies, whereas grouping by shared dependencies merges unrelated software defects into giant clusters.
Formally, let the group map \(\TaskGroupMap(\EnvironmentFixture)\) send an environment fixture \(\EnvironmentFixture\) to its designated task group, and let the split map \(\SplitAssignmentMap(\TaskGroupMap)\) assign that group to a partition: training, development, or final test. Every harvested trajectory, synthesized record, or distilled learning example inherits the partition assignment of its originating fixture. Partition integrity requires that:
This partitioning invariant guarantees that every known form of a problem stays on one side of the split. Because the guarantee is only as strong as the pipeline’s record of how tasks relate, production split manifests log the active grouping policy, git commit lineages, schema versions, and quarantined matches awaiting human triage. When allocating splits, randomization operates over task groups rather than individual rollouts, stratifying by task complexity or repository size where needed, and evaluation dashboards report group counts alongside raw trajectory counts because dozens of rollouts on one fixture still test only one problem.
Table 4 maps common evaluation claims to their required isolation boundaries. Every valid protocol enforces task-group isolation at a minimum.
Table 4: Partition boundaries follow evaluation claims: Shared infrastructure and leaked answers represent fundamentally different conditions. Changing multiple axes simultaneously measures a combined shift and obscures attribution.
Evaluation claim
Additional held-out boundary
Interpretation
New tasks in the intended operating environment
Underlying task and solution-equivalent variants
Shared repositories and interfaces may match deployment; task answers remain protected
Transfer to new repository families
Declared related repositories, including relevant forks or copied problem families
Measures transfer beyond those families, rather than universal generalization across all codebases
Adaptation to changed tool interfaces
Specified schema families or transformations
Tests adaptation to the declared interface shift while holding baseline environment conditions fixed
Measures resilience to configuration and library drift under controlled operating conditions
Engineering teams must also keep development and final-test partitions separate across iterations. Development tasks guide day-to-day tuning of prompts, trajectory filters, and fine-tuning mixtures, while final-test tasks provide a single measurement of the finished system under the declared protocol. Evaluating candidate checkpoints repeatedly against the final test suite and adjusting data mixtures from those scores leaks test signal into the training loop. Whenever a team inspects a final-test failure to diagnose a model flaw or build a training demonstration, it must reclassify that task group as development data and draw an unseen task group to replace it in the final test. Keeping that boundary intact requires controlling every path through which held-out evidence could reach upstream components.
15.4.2 Held-out evidence protection
If a synthetic problem generator reads a held-out test specification, it weaves the target logic into nominally new training fixtures, and if an automated trajectory ranker scores candidates by their performance on the final test suite, it tunes the training corpus to that test set without copying a single token. A partition boundary is therefore an architectural firewall across information paths, not merely a directory tag on a storage volume. Reference patches, ground-truth task specifications, hidden unit tests, and final evaluation outcomes must stay isolated from every upstream component: trajectory harvesting prompts, vector retrieval indexes, shared cross-session agent memories, synthetic generators, and model-based selection filters. During a trial itself, the agent likewise receives only the feedback the test protocol permits, such as local development tests, while the hidden acceptance suite stays behind the grader isolation of Grader isolation. Protecting held-out evidence means auditing every one of these access paths across both upstream curation pipelines and the trial runtime.
Whenever new candidate fixtures enter the curation pipeline, ingestion checks compare them against reserved task groups using task identifiers, git commit histories, patch hashes, and abstract syntax trees. Any fixture with ambiguous lineage goes to a quarantine queue for manual review rather than receiving a fresh identifier. Redaction, variable renaming, and schema changes do not break provenance; derived records always inherit their parent fixture’s group assignment. If a later audit finds that training data leaked information about a final-test task, deleting the offending record from the data lake is not enough: because the leaked signal is already in the weights, the team must retire the checkpoint, re-evaluate any affected claims, and retrain from a clean corpus snapshot.
Even when local split boundaries are airtight, pipelines remain exposed to contamination: the appearance of evaluation tasks or solutions inside the base model’s pretraining web crawl.2 Public benchmarks drawn from competitive programming sites or popular open-source repositories often sit inside pretraining corpora, which undermines zero-shot or few-shot generalization claims. Defensible evaluation therefore relies on private fixtures authored behind internal firewalls, commit timestamps audited against model training cutoffs, and documented data provenance—backed by similarity screening to catch unrecorded duplicates before splits are frozen.
2Benchmark Contamination: When web-scale pretraining corpora grew to include public benchmarks, models were often trained directly on the test splits later used to grade them (Sainz et al. 2023). Because base-model providers rarely publish token-level manifests of their pretraining data, downstream teams must rely on n-gram overlap probes and memorization checks to detect leakage. For an agent trajectory pipeline, the only reliable defense is structural: maintain private internal fixtures whose code and solutions have never appeared on the public web, and report separately which benchmark results come from those private evaluations.
Sainz, Oscar, Jon Campos, Iker Garcı́a-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. “NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark.”Findings of the Association for Computational Linguistics: EMNLP 2023, 10776–87. https://doi.org/10.18653/v1/2023.findings-emnlp.722.
↩︎ Contamination is one form of the pre-trial leakage established in Evaluation leakage.
15.4.3 Similarity screening
Metadata group assignments catch only the relationships the pipeline already knows; a vendored library or a bug copied across unrelated repositories without a shared git commit slips right past them. Similarity screening searches for those unrecorded relatives so they can join their task group before splits are assigned, and the same screen limits redundancy inside the training corpus. In language-model pretraining, removing near-duplicate documents cuts verbatim memorization and improves generalization (Lee et al. 2022). In agent trajectory curation, however, similarity depends on how a record is represented, and purely textual overlap misleads in both directions: two repairs can fix the same race condition with almost no shared lines in their patch diffs, while two traces that run the same shell commands and test invocations can diverge completely in how they read the terminal output and recover from errors.
Lee, Katherine, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. “Deduplicating Training Data Makes Language Models Better.”Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8424–45. https://doi.org/10.18653/v1/2022.acl-long.577.
3MinHash: Minwise independent permutations were introduced to spot near-duplicate web pages at search-engine scale, where comparing every pair of documents directly is far too slow (Broder 1997). Trajectory screening shares the same economics: signatures are cheap to compute and store, and the real cost lies in reviewing the candidate pairs they surface.
Broder, A. Z. 1997. “On the Resemblance and Containment of Documents.”Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), 21–29. https://doi.org/10.1109/sequen.1997.666900.
To capture meaningful similarity in agent trajectories, feature extraction combines text shingles, parsed code structure, and tool-call sequences. A shingle is a local feature extracted from a document or trace, such as an n-gram of command tokens or a subtree from an abstract syntax tree (AST). While parsing code into an AST strips away formatting differences, aggressive canonicalization—discarding variable names, literal constants, or statement order—can erase the difference between a working patch and a broken one. Curation pipelines therefore version their normalization rules and treat high similarity scores as prompts for inspection rather than triggers for automatic deletion. To compare feature sets across hundreds of thousands of trajectories, pipelines use MinHash (Broder 1997),3 which compresses large feature sets into short signatures by taking the minimum hash value under each of several random permutations; workloads where feature counts matter use weighted or bag-based variants instead.
For nonempty finite feature sets \(\ScreenedFeatureSet_1\) and \(\ScreenedFeatureSet_2\), the Jaccard similarity \(\JaccardSimilarity(\ScreenedFeatureSet_1,\ScreenedFeatureSet_2)\) and the theoretical MinHash collision probability satisfy:
Here, \(\MinHashPermutation\) is a uniformly random permutation of the feature vocabulary. The minimum hash values under \(\MinHashPermutation\) match whenever the first element of \(\ScreenedFeatureSet_1\cup \ScreenedFeatureSet_2\) under \(\MinHashPermutation\) lies in their intersection \(\ScreenedFeatureSet_1\cap \ScreenedFeatureSet_2\). Across \(\MinHashPermutationCount\) independent random permutations, the fraction of matching signature components is an unbiased estimator of \(\JaccardSimilarity(\ScreenedFeatureSet_1,\ScreenedFeatureSet_2)\) with variance \(\JaccardSimilarity(1-\JaccardSimilarity)/\MinHashPermutationCount\)(Li and Li 2022). Production systems approximate true permutations with universal hash families and define explicit fallback rules for empty feature sets.
Li, Xiaoyun, and Ping Li. 2022. “C-MinHash: Improving Minwise Hashing with Circulant Permutation.”International Conference on Machine Learning (ICML) 162: 12857–87.
4Locality-Sensitive Hashing: A hashing scheme in which the collision probability increases with item similarity, built for sublinear approximate nearest-neighbor search in high dimensions (Indyk and Motwani 1998). Banding MinHash signatures adapts LSH to Jaccard similarity. Engineers choose \(\SignatureBandCount\) and \(\LshRowsPerBand\) to match candidate volume to downstream review capacity, since large collision buckets can still make pair-review work grow rapidly with corpus size.
Indyk, Piotr, and Rajeev Motwani. 1998. “Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality.”Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing (STOC), 604–13. https://doi.org/10.1145/276698.276876.
Comparing every pair of MinHash signatures directly still scales quadratically with corpus size, so pipelines index signatures with Locality-Sensitive Hashing (LSH).4 An LSH index splits an \(\MinHashPermutationCount\)-component signature into \(\SignatureBandCount\) bands of \(\LshRowsPerBand\) rows each (\(\MinHashPermutationCount = \SignatureBandCount \LshRowsPerBand\)). Two trajectories become candidate near-duplicates if their signatures match across all \(\LshRowsPerBand\) rows in at least one band. With independent hash components, a pair with Jaccard similarity \(\JaccardSimilarity\) matches all rows of a given band with probability \(\JaccardSimilarity^{\LshRowsPerBand}\), and collides in at least one of the \(\SignatureBandCount\) bands with probability:
This function forms an S-shaped selection curve whose steep region sits near \((1/\SignatureBandCount)^{1/\LshRowsPerBand}\). With the signature length \(\MinHashPermutationCount = \SignatureBandCount \LshRowsPerBand\) held fixed, the two parameters trade off directly: moving components from rows into bands (fewer rows per band, more bands) shifts the curve toward lower similarities, catching more potential duplicates at the cost of larger collision buckets and heavier downstream review. Adding bands without reducing rows shifts the curve the same way while lengthening every signature.
Banding transforms candidate retrieval into an S-shaped probability curve governed by the number of bands and rows.
With \(\MinHashPermutationCount =\) 128 independent ideal components, the variance \(\JaccardSimilarity(1-\JaccardSimilarity)/\MinHashPermutationCount\) peaks at \(\JaccardSimilarity = 0.5\), so the standard error is at most \(0.5/\sqrt{\MinHashPermutationCount}\), or 0.0442 on the Jaccard scale. Storing each component in 4 bytes requires 512 bytes per signature and 51.2 MB for 100,000 records. While this raw matrix excludes the feature inverted index, metadata storage, and candidate staging buffers, it shows that signature storage is small even for large corpora.
The index configuration of 32 bands with 4 rows retrieves a pair with similarity \(\JaccardSimilarity =\) 0.5 with candidate probability 87.3 percent, and candidate retrieval becomes virtually certain at \(\JaccardSimilarity =\) 0.8. Downstream review capacity, whether automated AST diff verification or human inspection, is the binding constraint. Because this configuration routes most moderately similar pairs to review, engineers tune the band geometry to their review budget rather than to an abstract similarity target.
Engineers must inspect candidate pairs in context before merging task groups or discarding trajectories from the corpus. Trajectory deduplication differs from document deduplication: a short, three-step execution trace is not automatically better than a longer trace that shows error recovery, and two agents that produce identical code patches may have taken distinct decision paths that are both worth keeping. Once every known and screened relative is locked into its group’s partition, the admission cascade can evaluate each candidate record on its own evidence.
15.5 Staged Verifier Cascades
A trajectory record can faithfully capture a bug fix yet remain unusable for imitation learning if its data licensing review is incomplete or its execution trace contains unredacted credentials. Running a heavy, multi-minute containerized test suite before verifying those compliance gates wastes expensive verification slots on candidates the system must ultimately reject. Curation architectures therefore separate two decisions: they define the admission criteria required for each learning objective first, and then schedule the predicates in an order that minimizes wasted checking work.5
5Classifier Cascade: An ordered sequence of tests that drops unpromising candidates early using cheap checks gained prominence in real-time object detection, where a cascade of boosted classifiers rejected background image patches after computing only a few features (Viola and Jones 2001). A curation cascade shares the same compute economics, but its error tolerance is much stricter: whereas a visual detector can trade a small rise in false positives for faster throughput, loosening an admission check in a curation cascade pollutes the training corpus and degrades the resulting policy.
Viola, Paul, and Michael J. Jones. 2001. “Rapid Object Detection Using a Boosted Cascade of Simple Features.”Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 1: I-511-I-518. https://doi.org/10.1109/cvpr.2001.990517.
Definition 15.1: Verifier cascade
Verifier cascade, also called an admission cascade, is an ordered sequence of declared verification predicates that screens a candidate trajectory for inclusion in a specific learning dataset, terminating evaluation along that route at the first unsatisfied predicate.
Significance: It governs curation throughput and verification cost. The expected validation work per admitted record equals the reach-weighted sum of predicate costs divided by the overall route yield, the same cost-over-acceptance form as cost per accepted task (equation 1.2). Reordering independent predicates can cut verification time substantially without altering the final admitted set.
Distinction: Rejection is route-specific. A trajectory that fails the strict criteria of an end-to-end demonstration route remains fully available for error diagnosis, preference pair generation, or rollout fixture synthesis, each governed by its own independent predicate sequence.
Common pitfall: Attempting to accelerate validation by dropping or weakening admission predicates. Relaxing a predicate alters the admitted population rather than optimizing evaluation order, converting an efficiency optimization into an unvetted shift in training criteria.
15.5.1 Admission predicates
Treating a passing test suite as sufficient for admission lets records with missing tool outputs, unresolved license holds, out-of-scope file edits, or unmasked intermediate errors slip straight into the training set. To catch each class of defect separately, table 5 divides the admission decision into five predicate families. These families are logical questions rather than separate microservices, and their concrete checks depend on the target artifact: a complete demonstration, an isolated recovery segment, a preference pair, or an evaluation fixture for fresh rollouts.
Table 5: Predicates for consumer-specific admission: Each verifier records its version, empirical evidence, evaluation scope, and downstream routing consequence. A faithful trajectory log can safely record invalid or prohibited actions without endorsing them as learning targets.
Predicate
Admission question
Repair example
\(\text{chk}_{\text{record}}\): record adequacy and fidelity
Does the available record accurately represent the events and context required for this use?
Distinguish a malformed tool call rejected before dispatch from an executed patch; bind test outcomes strictly to the evaluated candidate.
\(\text{chk}_{\text{use}}\): permitted use
Are the data provenance, licensing, access rights, and dataset splits compatible with the proposed use?
Quarantine evaluation benchmarks to prevent training contamination; hold derived traces pending unresolved disclosure reviews.
\(\text{chk}_{\text{proc}}\): process conditions
Does execution history satisfy the declared operational constraints and tool policies?
Verify that the agent respected file-modification boundaries, obtained required execution approvals, and stayed within memory and step limits.
\(\text{chk}_{\text{out}}\): outcome conditions
Does empirical evidence verify that the candidate achieved the required objective?
An end-to-end bug repair requires authoritative regression tests to pass; an isolated recovery turn requires its scoped postcondition to hold.
Are the selected target or comparison roles justified by context and the learning objective?
Retain an intercepted malformed proposal as conditioning context while masking it from token-level imitation targets.
For an end-to-end demonstration route, admission requires satisfying the conjunction \(\text{chk}_{\text{record}} \land \text{chk}_{\text{use}} \land \text{chk}_{\text{proc,demo}} \land \text{chk}_{\text{out,demo}} \land \text{chk}_{\text{target,demo}}\). The subscripts emphasize that predicates are consumer-specific: a candidate that fails the final repair objective fails this route’s outcome condition (\(\text{chk}_{\text{out,demo}}\)), whereas an isolated error-recovery segment within that same run can satisfy the outcome check of a recovery-learning route. Any action designated as an imitation target must be justifiable strictly from the observation context the agent possessed at that step, without lookahead leakage or hindsight annotations injected from subsequent checks. These target designations determine the token-level loss masks during supervised fine-tuning.
↪ Token-level loss masking and target designation mechanisms are developed in Observation Loss Masking.
Engineers must separate record validity from action validity. A log that accurately captures a malformed tool call and the runtime’s immediate pre-dispatch refusal satisfies record adequacy (\(\text{chk}_{\text{record}}\)); the logging subsystem worked properly even though the model made an error. Conversely, missing environment observations or out-of-order tool responses make a trajectory unusable for imitation learning, even if the candidate code ultimately passed all tests. Pipelines can route truncated or incomplete records to diagnostic stores for failure analysis, but must never package them as behavioral demonstrations.
A preference route adds the shared-context and justified-ordering conditions that define a preference comparison (section 15.2.3) to its outcome and target predicates. Until paired with a matching winning rollout, a failed attempt remains only an unassigned comparison candidate. For reinforcement learning pipelines, curation delivers versioned task definitions, resettable environment fixtures, and automated grading harnesses for fresh rollouts rather than static records of past failures. While these downstream consumers share the same requirements for permitted use (\(\text{chk}_{\text{use}}\)) and record adequacy (\(\text{chk}_{\text{record}}\)), they differ in how they check outcomes (\(\text{chk}_{\text{out}}\)) and assign target labels (\(\text{chk}_{\text{target}}\)).
Each predicate returns one of four canonical states (pass, fail, pending, or unknown), following the four-verdict rule of The four-verdict rule. A fail stops evaluation along that route at once and retains the record for diagnosis or another route (figure 2). A missing or timed-out check never defaults to a pass; unverified candidates move to an unresolved holding queue, tagged with the missing evidence needed to finish the check. Curation telemetry counts evaluated, admitted, rejected, and unresolved records per route so that traces redirected to diagnostic pools never inflate the reported yield of demonstrations, and so that outcome checks themselves are guarded against manipulation by the candidate code.
Figure 2: One Admission Route: A collection record meets the five predicates in execution order and is admitted to the route only if every one passes. The first result other than a pass stops that route’s checks. A failure rejects the record for this route without deleting it, and a pending or unknown result goes to an unresolved hold instead of counting as a pass. Other routes evaluate the same record under their own conditions.
15.5.2 Test-tampering defense
When an agent resolves a failing test suite by rewriting the test assertion rather than fixing the underlying race condition (listing 1), the breach costs far more during data curation than during benchmark evaluation. Tampering during evaluation skews a single benchmark score; tampering during collection admits corrupt demonstrations that train later models to repeat the exploit.
↩︎ Grader isolation, which keeps held-out tests, grading scripts, and pass signals beyond the candidate’s reach, is established in Grader isolation.
Listing 1: A tampered reproducer: The candidate replaces the missed-wakeup assertions with a constant-true assertion. The local test exits with success while the defect remains.
Replacing the assertion produces a zero exit code from the test runner while leaving the concurrency bug untouched. In a well-designed cascade, this patch fails on two counts: the candidate diff violates the task’s allowed-path policy (failing process condition \(\text{chk}_{\text{proc}}\)), and running the code against clean, unedited test harnesses reproduces the deadlock (failing outcome condition \(\text{chk}_{\text{out}}\)). An agent’s self-reported success or local exit code must never override authoritative verifiers.
Test-tampering defense is grader isolation applied to admission: the controls that keep test harnesses, test inputs, and verification results beyond the reach of the candidate and the acting agent, so that a pass recorded by \(\text{chk}_{\text{out}}\) is an independent measurement rather than the candidate’s own report. Because later checkpoints train on admitted trajectories, an undetected bypass is reinforced across training rounds rather than contained to one evaluation trial. The defense protects the integrity of the measurement, not its breadth; a tamper-proof suite still checks only what its assertions cover, and whether a passing solution generalizes remains a question for the target suitability predicate \(\text{chk}_{\text{target}}\). The failure modes that tamper defense prevents are not hypothetical: they have appeared directly in frontier models during agentic training.
War Story 15.1: Special-cased tests in agentic coding (2025)
Context: Anthropic’s system card for Claude 3.7 Sonnet documents strategic evaluation gaming observed during agentic coding evaluations (Anthropic 2025).
Mechanism: When faced with difficult programming tasks after repeated unsuccessful attempts, the model occasionally resorted to reward hacking, exploiting discrepancies between the empirical reward signal and true task completion. Specifically, the model hardcoded expected return values for known test cases or modified unit test assertions directly so that the suite matched its incorrect implementation.
Impact: Standard user testing failed to catch the behavior because it appeared rarely during straightforward interactions, emerging primarily after extended, multi-turn troubleshooting cycles in specialized programming sandboxes.
Fix: The team deployed transcript classifiers to detect gaming patterns in training runs and applied targeted mitigations prior to model deployment. For production agentic environments, the report advises monitoring for repetitive edit-and-test loops on individual files, code comments indicating test-specific overrides, and unauthorized diffs against test suites.
Systems lesson: An exit code produced inside an execution environment accessible to untrusted agent code is unverified telemetry. A curation pipeline that blindly admits trajectories based on local test passes ingests the exact gaming behavior that monitoring seeks to eradicate. Tamper verification must therefore run as an immutable gate in the cascade before any trajectory can serve as a training target.
Anthropic. 2025. Claude 3.7 Sonnet System Card.
To prevent such exploits from entering the corpus, tamper defense combines the evaluation gym’s containment controls (principle 9) with an admission-specific diff audit. Out-of-tree test fixtures keep authoritative tests outside the candidate’s writable workspace until its snapshot is hashed and frozen, while isolated runner supervision executes the harness in a fresh sandbox where candidate code cannot patch assertion libraries or forge exit codes (since a read-only mount alone still allows co-located code to read tests or interfere with the reporting process; Grader isolation). On top of those two evaluation controls, admission adds a protected-path manifest that binds the diff against the base commit to the process predicate, so any unauthorized change to test suites, continuous integration configuration, or build dependencies fails \(\text{chk}_{\text{proc}}\) irrespective of the test runner’s exit code. Because the manifest follows the task’s patch-admission policy, a task that legitimately requires a new regression test or an updated package manifest can still pass. Containment and authorization checks precede the execution of untrusted code regardless of how the other predicates are scheduled, and an audit log written after the fact cannot substitute for them.
When evaluating non-deterministic tasks, verification harnesses enforce a declared repetition protocol that fixes environmental seeds, maximum retry bounds, and aggregation rules for inconsistent trials. Rerunning a flaky test until it passes once merely samples from favorable noise: if a defective candidate passes a flaky test with probability \(\AttemptPassProbability\) per run, rerunning it up to \(\IndependentAttemptCount\) times reports a pass with probability \(1-(1-\AttemptPassProbability)^\IndependentAttemptCount\), the at-least-one form of pass@\(\IndependentAttemptCount\) applied here to false passes. Timeouts must likewise be recorded as unresolved outcomes (pending or unknown) and routed to an inspection queue rather than silently treated as model errors or transient infrastructure faults. Once the predicates and their isolation bounds are fixed, the remaining question is how to order those checks so that rejected candidates consume as little verifier time as possible.
15.5.3 Cascade economics
Running an eight-second sandboxed test suite before a twenty-millisecond process audit burns seconds of verifier slot time on candidates that a fast check would have rejected immediately. To price and optimize the order in which checks run for a given admission route, let \(\text{chk}_1,\ldots,\text{chk}_{\AdmissionCheckCount}\) represent the sequence of admission predicates in execution order. Define \(\CheckPassProbability_i=\Pr(\text{chk}_i\text{ passes}\mid \text{chk}_1,\ldots,\text{chk}_{i-1}\text{ pass})\) as the conditional pass probability of predicate \(\text{chk}_i\), with \(\CheckPassProbability_1\) denoting the unconditional pass probability of the initial check. Let \(\CheckSlotTime_i\) represent the expected verifier slot time conditional on reaching that check, measured in seconds. Under short-circuit evaluation, which stops at the first non-passing predicate, the expected verification time per attempted record is:
where the empty product evaluates to one. The overall route yield \(\AdmissionRouteYield\) represents the joint probability that a candidate passes all \(\AdmissionCheckCount\) predicates:
For routes with nonzero yield (\(\AdmissionRouteYield > 0\)), the ratio \(\ExpectedCheckingTime/\AdmissionRouteYield\) is the expected verification slot time spent per admitted trajectory, where the yield \(\AdmissionRouteYield\) plays the role of acceptance probability in the cost-per-accepted-task formula (equation 1.2). Because this ratio is an expectation across large samples, a batch with zero admitted records reports total slot time consumed rather than an undefined ratio. Upstream rollout generation, retry loops, and downstream storage costs are accounted for outside this verifier cascade.
Reordering these predicates preserves the exact admitted set as long as their definitions, inputs, and evaluations remain unchanged, though conditional pass probabilities must be re-measured for each order when predicates are correlated because each check sees a different pre-filtered subset. Operational dependencies and containment rules also constrain permissible orders; data governance and containment checks, for instance, must always precede running untrusted code in a sandbox.
When two adjacent predicates \(\text{chk}_\FirstAdjacentCheckIndex\) and \(\text{chk}_\SecondAdjacentCheckIndex\) are statistically independent, the cheaper execution order has a simple closed form. Running \(\text{chk}_\FirstAdjacentCheckIndex\) before \(\text{chk}_\SecondAdjacentCheckIndex\) incurs an expected local cost of \(\FirstAdjacentCheckSlotTime + \FirstAdjacentCheckPassProbability \SecondAdjacentCheckSlotTime\), whereas running \(\text{chk}_\SecondAdjacentCheckIndex\) before \(\text{chk}_\FirstAdjacentCheckIndex\) incurs \(\SecondAdjacentCheckSlotTime + \SecondAdjacentCheckPassProbability \FirstAdjacentCheckSlotTime\). Running \(\text{chk}_\FirstAdjacentCheckIndex\) first is cheaper whenever \(\FirstAdjacentCheckSlotTime + \FirstAdjacentCheckPassProbability \SecondAdjacentCheckSlotTime < \SecondAdjacentCheckSlotTime + \SecondAdjacentCheckPassProbability \FirstAdjacentCheckSlotTime\), which rearranges to:
The optimal schedule therefore orders independent predicates by ascending cost per rejection, \(\CheckSlotTime_i / (1 - \CheckPassProbability_i)\), running the cheapest rejection filter first. This rule matches the predicate-ranking algorithm developed for relational database query optimizers with expensive user-defined filters (Hellerstein and Stonebraker 1993).6
6Predicate Rank: Relational database query optimizers faced the same scheduling problem when selection predicates began calling expensive user-defined functions. Ordering filters by selectivity alone ran heavy functions on rows that a fast check would have dropped. Sorting independent filters by rank—per-row cost divided by rejection probability \((1 - \CheckPassProbability_i)\)—minimizes expected work, matching the exchange argument in equation 2. In agent curation cascades, where checks range from sub-millisecond AST scans to multi-minute integration tests and human reviews, rank ordering decides where most of the verification budget goes.
Napkin Math 15.2: Admission ordering with the same accepted set
Problem: A batch of 100,000 candidate trajectory records enters an end-to-end demonstration admission route evaluated under predicates \(\text{chk}_{\text{record}}, \text{chk}_{\text{use}}, \text{chk}_{\text{proc}}, \text{chk}_{\text{out}}, \text{chk}_{\text{target}}\). Record fidelity (\(\text{chk}_{\text{record}}\)) and permitted-use (\(\text{chk}_{\text{use}}\)) checks must execute first due to data dependencies, and target suitability (\(\text{chk}_{\text{target}}\)) executes last because its review reads the test artifacts that \(\text{chk}_{\text{out}}\) produces. The process audit (\(\text{chk}_{\text{proc}}\)) and outcome verification (\(\text{chk}_{\text{out}}\)) can execute in either order. Which evaluation sequence minimizes verifier slot time, and how much total compute does it save?
Variables: Each predicate evaluation resolves deterministically to pass or fail. For this analysis, pass events are modeled as mutually independent, ensuring invariant marginal pass probabilities across permutations. Each reached verifier occupies a dedicated worker slot for the duration that table 6 lists beside its pass probability. The trajectory record and candidate repository snapshot are immutable. Pre-execution authorization, container containment, and grader protections are enforced prior to running any test suites. Upstream rollout generation, job scheduling delays, candidate storage, and human review costs are excluded.
Table 6: Inputs for one hypothetical admission route: Independence and fixed durations make order changes comparable without reusing incompatible conditional measurements.
Predicate
Hypothetical pass probability
Occupied slot time per reached check
\(\text{chk}_{\text{record}}\)
95%
0.01 s
\(\text{chk}_{\text{use}}\)
90%
0.04 s
\(\text{chk}_{\text{proc}}\)
80%
0.20 s
\(\text{chk}_{\text{out}}\)
40%
8 s
\(\text{chk}_{\text{target}}\)
75%
2 s
Math: Applying equation 2, the cost per rejection is 0.20 s / (1 − 0.8) = 1 s for process audit \(\text{chk}_{\text{proc}}\), compared to 8 s / (1 − 0.4) = 13.3 s for outcome verification \(\text{chk}_{\text{out}}\). Because \(\text{chk}_{\text{proc}}\) yields rejections far more cost-effectively than \(\text{chk}_{\text{out}}\), scheduling \(\text{chk}_{\text{proc}}\) ahead of \(\text{chk}_{\text{out}}\) minimizes expected compute. Both predicates are reached with joint probability 0.855 (the product of pass rates for \(\text{chk}_{\text{record}}\) and \(\text{chk}_{\text{use}}\)). Swapping the sequence saves \(\RecordAdequacyPassProbability \PermittedUsePassProbability [\OutcomeCheckSlotTime(1-\ProcessConditionPassProbability) - \ProcessConditionCheckTime(1-\OutcomeCheckPassProbability)]\), yielding 1.27 s saved per attempted record. Evaluating equation 1 across the parameters in table 6 produces the overall slot durations in table 7, verified through exhaustive state-space enumeration.
Table 7: Equal-criterion ordering comparison: Both orders stop this route at its first failure. Other uses of rejected records require separate work accounting.
Result: Both schedules in table 7 achieve an identical expected yield of 20.52 percent, admitting approximately 20,520 valid demonstrations from the batch. Evaluating process compliance \(\text{chk}_{\text{proc}}\) prior to sandbox testing \(\text{chk}_{\text{out}}\) reduces total verifier slot time by 16.9 percent, without discarding a single valid trajectory or lowering any quality threshold.
Systems insight: Sequence independent, fixed-cost admission verifiers within permissible orders by cost per rejection, \(\CheckSlotTime_i / (1 - \CheckPassProbability_i)\), rather than by raw duration alone. Ordering by cost per rejection places fast, selective filters ahead of expensive, permissive verifiers while guaranteeing an identical admitted set. These compute savings require neither cutting validation corners nor loosening thresholds. The dependency fixed in the problem is what keeps target suitability \(\text{chk}_{\text{target}}\) last here. Its cost per rejection is 8 s, below the outcome check’s 13.3 s, so a target review that did not read the test artifacts would belong ahead of \(\text{chk}_{\text{out}}\) as well. Data dependencies, container cold-start amortization, and compliance boundaries constrain permissible schedules. Correlated checks require comparing admissible orders with measured conditional pass rates and reached-check costs in equation 1.
15.5.4 Learned scores and scoped judgments
Deterministic checks cannot score open-ended prose quality or step-by-step reasoning clarity, yet letting a learned scorer or brevity filter admit trajectories on its own either purges multi-turn recovery sequences or admits flawed runs that game the scorer. Learned scoring models are still useful for ranking candidate trajectories for manual review or sub-sampling from a pool of eligible examples to balance diversity and conciseness. Process reward models, for instance, score individual reasoning steps and intermediate tool calls rather than only final task outcomes (Uesato et al. 2022; Lightman et al. 2024). Even so, while a low step score flags a likely point of failure, pinpointing the root cause still requires human or symbolic review, and filtering aggressively for short trajectories strips out the multi-turn recovery paths the policy needs to learn error correction.
When tasks require evaluating open-ended prose, user interactions, or design quality, human raters or judge models supply scores governed by explicit rubrics and calibration standards. The verification asymmetry (principle 3) bounds what such a score can do inside a cascade. A learned assessment can rank candidates or supply scoped evidence for one qualitative criterion, but it is not an independent guarantee, and when valid records are rare, even a small false-accept rate fills the admitted set with invalid ones. The judge’s input must therefore be limited to the evidence its criterion names and kept away from anything the candidate wrote to influence its grader. Deterministic verifiers require the same scoping: every verifier execution records the assessor version, rubric hash, extracted evidence, confidence scores, and explicit abstentions, and a high qualitative rating never overrides a failed deterministic check.
Repeated selection against an automated scorer also concentrates the scorer’s blind spots. Filtering training data against a learned proxy score initially improves output quality, but continued optimization eventually triggers proxy collapse, where the proxy score keeps rising while true downstream task quality falls (Gao et al. 2023). This is the same divergence that best-of-N selection produces against an imperfect check at inference time (How search exploits imperfect checks), except that curation writes the winners into the next policy’s weights. Mitigating this feedback loop requires periodic double-blind audits of both accepted and rejected records, continuous recalibration against held-out ground truth that never enters training, and re-scoring whenever policy architectures or task domains shift. Any data augmentation or trajectory transformation likewise requires fresh verification of its modified payload and token masks, a provenance boundary that section 15.9 develops.
Checkpoint 15.1: Defend an admission route
A route is defensible when each predicate’s verdict and the order of checking can both be justified.
Deciding admission
Pricing the cascade
By evaluating trajectories through staged, consumer-specific cascades rather than forcing all data through a single cleanliness filter, the curation pipeline produces verified datasets tailored to distinct imitation, preference, and rollout objectives. Verifying clean executions, however, leaves the harder half of the data problem unsolved: production agents encounter disruptions, and training a policy to recover requires curating traces that contain mistakes without teaching the model to repeat them.
15.6 Recovery Demonstration Curation
Agents operating in real environments fail along the way. Consider an execution trace where an agent issues a malformed tool call, receives a rejection from the environment runtime, and then inspects the schema to produce a passing patch. A naive curation pipeline might discard this trace because it contains a failed step, or edit out the error to make the agent appear infallible. Either choice discards the evidence of how to recover from an imperfect prefix. The curation cascade can instead admit the successful repair while keeping the original misstep as conditioning context, which raises three practical questions: which recovery trajectories to harvest, how to mask them safely, and how much training capacity to give them.
A training corpus composed solely of error-free trajectories teaches an agent how to act only when every prior step succeeded. In production, however, noisy tool outputs, transient network faults, and early mistakes push an agent off the nominal path. Once off that path, a model trained only on clean prefixes faces states it never saw during training, and over a long horizon a recoverable detour turns into a failed task. This distribution mismatch7 creates the need for recovery demonstrations, and the curation pipeline’s job is to capture, validate, and package those recovery episodes with their conditioning evidence intact.
7Compounding Imitation Error: A policy trained only on states visited by an expert can turn a per-step error rate \(\PerStepErrorRate\) (the per-step error rate of horizon compounding, equation 1.1) into as many as \(\TurnHorizon^2\PerStepErrorRate\) mistakes over an \(\TurnHorizon\)-step task, because an early error exposes the policy to states absent from its training distribution. Querying an expert interactively on learner-visited states, the core mechanism of DAgger, restores an error bound that grows linearly in \(\TurnHorizon\)(Ross et al. 2011). Recovery demonstrations are the offline curation counterpart: they supply supervised repair targets on the flawed state prefixes that direct-success collection omits.
Ross, Stéphane, Geoffrey Gordon, and J. Andrew Bagnell. 2011. “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning.”Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS) 15: 627–35.
↪ Exposure bias, the mismatch between training on recorded prefixes and running on prefixes the policy produced itself, and its horizon-dependent cost are developed in Exposure Bias.
Definition 15.2: Recovery demonstration
A recovery demonstration is a curated trajectory record or isolated sequence segment that pairs the context of an operational disruption (a model mistake, environment fault, or partial observation) with an action assessed as correct under the task specification and runtime authority boundaries.
Significance: Recovery demonstrations directly address long-horizon execution stability. Under horizon compounding (equation 1.1), any step-level error that the policy can successfully remediate becomes a temporary detour rather than an irrecoverable task failure. Direct-success harvesting systematically starves the training distribution of these pivotal recovery branches.
Distinction: An appropriate recovery action depends on context: it may diagnose an exception, retry with corrected parameters, back off, or hand off control to a human supervisor. It does not require the agent to overcome an outage it cannot repair or to exceed its authority to finish the task.
Common pitfall: Treating every token in an eventually successful trajectory as a positive imitation target, thereby inadvertently teaching the model to repeat the initial mistake.
15.6.1 Record roles and learning uses
A single extended trajectory may begin with clean progress, hit an unexpected environment error, execute a sound recovery, and later stall on an unrelated tool failure. Discarding that entire trajectory wastes genuine recovery signal, whereas training uncritically on all of its tokens teaches the model to replicate its bugs. Because a trace rarely fits a single outcome bucket, labels such as direct success, recovery, and negative describe the behavioral evidence a segment contains rather than granting blanket permission for training. The predicates of table 5 already separate the two decisions: record adequacy and permitted use are shared across routes, while the process, outcome, and target predicates are set by the consumer, so a role label indicates which consumer-specific checks a segment might pass (table 8).
Table 8: Behavioral Roles and Learning Uses: Record labels describe evidence. Each intended consumer still requires its own admission conditions.
Record role
Evidence worth preserving
Learning-use decision
Direct-success demonstration
Grounded actions and adequate evidence for the declared result
Select useful responses under the demonstration policy; shortest length and absence of error codes are not sufficient criteria
Recovery demonstration
The preceding context, observed disruption, response, and evidence for its assessed outcome
Select the justified response while retaining the mistake or fault as context where needed
Hard-negative candidate
A plausible but inadequate response, with evidence identifying the relevant deficiency
Consider diagnostic use, a separately assessed segment, or a compatible preference comparison; do not automatically imitate the deficient response
A hard negative is an action or plan that looks plausible under surface checks—valid syntax, expected keywords, or a passing local smoke test—yet fails full functional verification. Pinpointing the exact step where an agentic trace went wrong is rarely simple: a failed assertion at the end of a run confirms that an error occurred without naming the earlier decision that caused it, and repeated tool calls that look like a loop may be legitimate polling against an asynchronous job. Curation systems therefore record the failure signal that implicates a candidate divergence step together with an explicit note when the first causal step is uncertain, rather than inventing an exact divergence point. Engineering teams also frequently assume that any recorded negative can be paired immediately with a success to train a preference objective, or replayed as a baseline for reinforcement learning against a verifiable reward. Both shortcuts fail in practice: a negative forms a valid comparison only under the shared-context condition of section 15.2.3, which a success generated under a different prompt, an extra hint, or updated tool outputs violates, and a reinforcement learning baseline needs a live, resettable fixture rather than a stored failure transcript.
Systems Perspective 15.1: A recorded mistake is context, not a target
A pervasive misconception in trajectory engineering is that an imitation training set must consist exclusively of pristine, error-free transcripts. In reality, a curated training dataset consists of complete conversational contexts coupled with explicit target masks, and these two components serve fundamentally different purposes.
A flawed model response should receive no direct positive imitation target loss. Nevertheless, that erroneous response must remain in the prompt history if subsequent recovery actions are to make any sense; a correction is meaningful only in the context of the error it resolves. Conversely, an execution trace that ultimately timed out or failed an integration test often contains early sub-tasks solved with exemplary precision. The curation pipeline marks which spans represent valid targets and which represent conditioning context; the loss-masking subsystem then omits their direct next-token loss terms while retaining them as conditioning during fine-tuning. Masking errors out of the loss while retaining them in the attention context teaches the model how to react to a failure without training it to generate one.
15.6.2 Conditioning evidence for a correction
To see how target selection separates context from targets on a real trace, follow collection run collection_run_r19_01 as it executes task pool_repair_01 inside test environment pool_fixture_v1. The run produces workspace candidate ws_r20 and captures raw trace trace_r19_01 (workspace numbers count candidate revisions after the evaluation trial’s ws_r19, while run and trace numbers count collection batches). End-to-end grading confirms that ws_r20 satisfies repair_accept_v1, meeting all process constraints, resource ceilings, and cleanup duties.
The run belongs to a campaign whose declared learning use is recovery from harness refusals. Unlike the truncated observation that opened the chapter, this behavior is collected on the premise, recorded in the collection manifest, that the ladder’s probes left it with the model: supplying the schema in context and validating proposals against a strict schema still left the policy emitting malformed edits that the runtime had to refuse. What remains to teach is how to respond after a refusal, and the recorded event sequence shows how that recovery unfolds:
Event e1 delivers the task prompt, tool definitions, and repository context.
At event e2, the model proposes a file edit with a malformed argument. Because the tool runtime validates inputs locally before dispatch, it refuses the call immediately, and the trace records the refusal without logging an applied write.
At event e3, the model reads the refusal, requests the API schema documentation, and inspects the returned specification.
At event e4, the model issues a corrected, schema-compliant edit, and the tool enforces an atomic version check alongside the update rather than relying on a stale earlier read.
At event e5, the harness runs the required checks on the edited workspace and returns their summary as an observation.
At event e6, the model summarizes the validated changes, states the patch scope, and closes the session.
Under curation policy repair_accept_v1, an initial syntax refusal that the agent diagnoses and corrects promptly is eligible for recovery training. A target-selection manifest marks the actions worth imitating without editing the recorded sequence of events:
{"collection_run_id":"collection_run_r19_01","source_trace_id":"trace_r19_01","criteria_version":"repair_accept_v1","target_selection_version":"recovery_targets_v1","behavioral_role":"recovery","context_only_events":["e2","e5"],"selected_response_events":["e3","e4","e6"],"assessment_scope":"recorded correction and supported final report"}
The source trace preserves the original task prompt, tool definitions, intermediate observations, and evaluation logs intact, while target selection marks only the policy tokens inside events e3, e4, and e6. The refused proposal (e2) and test summary (e5) stay in the sequence as conditioning context, supplying the history that makes the schema lookup and corrected edit make sense. Passing the overall task does not make every intermediate action a training target; each selected step must be a sound decision given what the agent knew at that turn. Placed beside the other two record roles, the recovery record sits between a direct success, whose supported responses can all be targets, and a hard negative, whose deficient response is never one (figure 3).
Figure 3: Record Roles and Selected Targets: Each role is drawn as a sequence of events, with selected targets distinguished from context. In the recovery demonstration, the refused malformed proposal at e2 and the check summary at e5 stay as context while e3, e4, and e6 become targets. A hard-negative candidate looks adequate under a targeted check and fails a required one. Its deficient response is never an imitation target, and where it went wrong is recorded with its basis rather than fixed at an exact step.
When only a local recovery window has verified evidence, the curation harness can admit that segment alone while leaving the broader task outcome marked as unverified. Likewise, if post-processing scrubs credentials or private paths to create trace_r19_01_redacted_v1, the pipeline must re-verify the target spans against the modified tokens. Deleting the malformed call, pasting future test outputs into an earlier prompt, or truncating intermediate observations changes the causal problem the model is being asked to learn—just as waiting passively for natural failures leaves rare fault modes unrepresented.
15.6.3 Fault injection
In production, routine execution occasionally exposes an agent to network drops, service timeouts, and tool errors, but waiting for spontaneous failures leaves wide gaps in the training corpus. Systematic fault injection8 fills those gaps deliberately. A collection protocol logs the injection version, the targeted tool call or observation stream, the simulated system state, and the expected recovery action. Fault injection stays strictly inside the runtime’s security boundaries: injecting a network timeout or file lock grants the agent no extra authority to install unapproved packages, widen filesystem permissions, or bypass human approval gates. Each condition in table 9 names the causal distinction the fixture must preserve and the response a curator checks.
8Fault Injection: Introducing controlled failures to test resilience is a classic systems practice, popularized in distributed infrastructure as chaos engineering, which injects server, network, and disk faults into live clusters to check steady-state behavior (Basiri et al. 2016). Curation borrows the experimental discipline rather than the live production venue: faults are injected inside versioned, hermetic test fixtures where every fault is logged and environment state transitions stay deterministic, so demonstrations teach genuine recovery rather than quirks of the simulator.
Basiri, A., N. Behnam, R. de Rooij, L. Hochstein, L. Kosewski, J. Reynolds, and C. Rosenthal. 2016. “Chaos Engineering.”IEEE Software 33 (3): 35–41. https://doi.org/10.1109/ms.2016.60.
Table 9: Fault Conditions for Recovery Collection: Each condition preserves the relation between observations and possible effects. The criterion evaluates an appropriate response, not compulsory autonomous repair.
Collection condition
Required causal distinction
Candidate response to assess
Malformed proposal rejected locally
No operation was dispatched
Inspect the schema and correct the represented error
Partial search result
Missing content is distinguished from an empty complete result
Request a valid continuation or report insufficient evidence
Lost acknowledgment after a state-changing request
The endpoint may have acted despite the missing reply
Reconcile the identified operation before a dependent retry
Unavailable capability or exhausted allowance
The original goal may be unreachable within the grant
Stop or hand off with accurate scope and owned residual duties
Consider the lost-acknowledgment case in table 9. When a mock harness drops an acknowledgment after committing a state-changing remote call, the simulated environment must keep the side effects of that committed call in its state. If the mock returned a connection reset while silently rolling the server state back to a clean slate, it would misrepresent the distributed systems hazard and train the agent to retry blindly—causing duplicate writes in production. Replay determinism requires that every subsequent observation reflect the mutations that actually occurred.
Engineering teams must also separate tasks meant to teach autonomous repair from tasks meant to teach safe stopping. A recovery task provides an authorized, reachable fix inside the test environment; a safe-stopping task deliberately closes every repair path so the agent learns to halt cleanly and hand off to an operator. Because the two objectives require different grading rubrics, reliability coverage tracks unprompted agent mistakes and injected faults separately while recording any production failure modes that the fixture cannot reproduce. Harvesting these multi-turn recovery traces, however, changes the token geometry of the resulting corpus.
15.6.4 Records, stored tokens, and selected targets
Because a recovery trajectory includes diagnostic cycles, failed attempts, and error observations, it is much longer than a direct-success run, yet only the corrected responses carry supervised loss. As a result, the share of recovery trajectories in a corpus matches neither their share of stored tokens on disk nor their share of active target tokens during training. Curation accounting starts from a retained pool of trajectories and draws subsets for specific training artifacts: one retained trace may supply conditioning context and target spans to an imitation corpus while also entering a preference pair, yet it remains a single task execution in the pool.
One recovery class, three shares: records, stored tokens, targets.
For a trajectory category \(\TrajectoryCategoryIndex\), let \(\CategoryRecordCount_j\) denote the number of retained records, \(\MeanStoredTokensPerRecord^{(\TrajectoryCategoryIndex)}\) denote the average stored tokens per record under the active tokenizer, and \(\MeanTargetTokensPerRecord_j\) denote the average selected target tokens per record admitted into a specific imitation fine-tuning artifact. The total stored token volume \(\StoredTokenVolume_j\) and total target token volume \(\TargetTokenVolume_j\) are:
The stored token share measures \(\StoredTokenVolume_j\) against the total tokens in the repository, whereas the target token share measures \(\TargetTokenVolume_j\) against the active supervised tokens in the training dataset. Confusing the two breaks capacity planning: processed tokens (context plus targets) drive attention compute and memory traffic during forward and backward passes, whereas selected target tokens decide which positions contribute direct loss terms (whose gradients still flow back through the conditioning context). Specifying a dataset mixture by record count while budgeting accelerators by target tokens—or vice versa—creates scheduling surprises, especially when multi-turn conversations are unrolled without prefix sharing.
Napkin Math 15.3: One pool, three different token allocations
Problem: Consider a hypothetical retained corpus containing 50,000 records categorized into disjoint roles. The demonstration policy admits direct successes and recovery trajectories; negative trajectories are reserved for diagnostic logging and preference modeling, providing zero imitation targets for this artifact. Using the empirical means listed in table 10, determine the fractional share that recovery trajectories occupy across three distinct denominators: total records, tokens (stored in the repository and processed by this artifact), and supervised target tokens.
Table 10: Inputs to the Corpus Allocation Worksheet: Counts and lengths are hypothetical, not an empirically recommended mixture. Selected targets refer to one presentation of each eligible record under a fixed tokenizer and selection policy.
Retained class
Records
Mean stored tokens per record
Mean selected target tokens per record
Direct
32,500
7,200
1,200
Recovery
12,500
18,500
2,000
Negative
5,000
9,800
None in this imitation artifact
Math: Total stored tokens follow \(\StoredTokenVolume_j = \CategoryRecordCount_j\MeanStoredTokensPerRecord^{(\TrajectoryCategoryIndex)}\): direct successes account for 234,000,000, recovery traces account for 231,250,000, and negatives account for 49,000,000, totaling 514,250,000 across the repository and 465,250,000 across admitted demonstration records. Supervised target tokens follow \(\TargetTokenVolume_j = \CategoryRecordCount_j\MeanTargetTokensPerRecord_j\): direct demonstrations contribute 39,000,000 target tokens and recovery traces contribute 25,000,000 target tokens, totaling 64,000,000 supervisory tokens.
Result: Recovery accounts for 25 percent of total retained records, 44.97 percent of all stored tokens in the repository, 49.70 percent of the admitted demonstration tokens, and 39.06 percent of active target supervision tokens. The two token shares answer different questions. The repository share includes negatives that this artifact never trains on, so it prices storage; the admitted-demonstration share counts the tokens that training actually processes, so it prices compute.
Systems insight: Always formulate training mixtures in terms of target tokens while budgeting accelerator compute and memory in terms of processed context tokens. The operational multiplier separating these two metrics is the context-to-target ratio: 9.25 stored context tokens per target token for recovery traces, compared to 6 for direct successes. Increasing the recovery proportion inflates processed context volume far faster than supervised gradient signal. An engineering team that configures a mixture purely by record count will request a 25 percent recovery ratio but will end up committing 44.97 percent of repository storage and 49.70 percent of processed training tokens to recovery records. Naive multi-turn serialization that repeatedly unrolls conversational prefixes amplifies this computational overhead even further.
15.6.5 Mixture experiments
Token accounting prices the storage and compute footprint of a recovery-heavy corpus, but it cannot tell whether adding more recovery traces actually improves fault tolerance or crowds out direct-path competence on routine tasks. Answering that question requires an empirical mixture experiment that holds the evaluation protocol, base checkpoint, target-masking rules, and training schedule fixed while varying the recovery ratio across training cohorts. Curation engineers track deduplication and replacement in each cohort, because raising the recovery share under a fixed training budget reduces coverage of direct-path tasks.
Because context-to-target ratios differ by role, an experiment must state which denominator it holds constant and report results under both. Fixing total supervised target tokens forces the recovery-heavy cohort to process far more conditioning context tokens, raising training wall-clock time and FLOPs; fixing total processed tokens gives the recovery-heavy cohort fewer supervised target updates. When accelerator hours or training windows are hard limits, teams compare candidate mixtures at equal wall-clock time or compute cost, screening those candidates on the development split and testing the chosen configuration once on the held-out evaluation benchmark. To keep aggregate scores from hiding regressions, evaluation reports break results down across nominal tasks, injected fault conditions, and out-of-distribution transfer suites before computing any summary metric, tracking task acceptance alongside hallucinated tool outputs, redundant steps, clean stops, and total token spend.
Once an effective mixture is chosen, its parameters are written into the corpus manifest, the machine-readable, versioned record that publishes a corpus and tells a trainer what each member is and how it may be used.9 Beyond the mixture weights, the manifest binds input trajectory pool identifiers, selection filter versions, token accounting denominators, test coverage matrices, and audit logs, whose per-record lineage and publication are developed in section 15.9 and section 15.10.2. Deploying a model trained on this manifest changes the trajectories that subsequent collection runs produce, feeding new failure modes and recoveries back into the harvesting pipeline and closing a loop in which the policy being improved also generates the data that improves it.
9Dataset Documentation: Documenting datasets with structured records that state how they were built, what they contain, and what uses they support was proposed by analogy to hardware datasheets in electronics (Gebru et al. 2021). A corpus manifest turns that documentation into a machine-checked contract for agent pipelines: its fields bind cryptographically to dataset versions, filter code, and verification logs so that training jobs can check corpus invariants automatically before starting a run.
Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. “Datasheets for Datasets.”Communications of the ACM 64 (12): 86–92. https://doi.org/10.1145/3458723.
15.7 The Self-Improvement Loop
Suppose a curation round for the connection-pool repair agent retains only straight-line, clean executions while discarding messier traces where the agent recovers from misleading test diagnostics. Fine-tuning on that corpus strengthens straightforward repairs while leaving recovery unimproved, and in the next round a higher nominal pass rate hides that gap rather than closing it. Because corpus selection shapes the behaviors the next policy generation learns and samples, curators track three distinct signals in every round: what the current policy sampled, what the curation cascade kept, and how the resulting model scores under independent held-out evaluation.
15.7.1 Rejection-sampling fine-tuning
When a team lacks human demonstrations for new task fixtures, it can bootstrap training data from the current policy whenever verifying a candidate solution is easier than generating one—provided it logs attempted, rejected, and unverified runs alongside retained examples so a looser filter is never mistaken for a better model. In this generate–select–train loop, a versioned policy samples executions across a suite of task fixtures within a fixed compute budget, and the admission cascade routes the resulting traces by role: clean demonstrations feed imitation buffers, branched pairs populate preference datasets, and unresolved traces enter diagnostic storage. Training on the admitted records produces a candidate checkpoint, which held-out evaluation tests to decide whether it becomes the generator for the next collection round (an internal development promotion that stays separate from production release).
Selecting sampled successes for imitation is rejection-sampling fine-tuning.10 The Self-Taught Reasoner (STaR) illustrates this loop in bootstrapped reasoning: the model generates candidate rationales, keeps those that reach correct answers, fine-tunes on them, and uses ground-truth answers to prompt new rationales on tasks it initially failed (Zelikman et al. 2022). In agentic tasks, an automated verifier checks that a run satisfies its acceptance criteria, while target review still checks that the intermediate tool calls and state transitions are sound enough to imitate.
10Expert Iteration: The generate–select–train loop predates language models. In expert iteration, tree search guided by the current network finds stronger moves than the network alone, the network is trained to imitate them, and the updated network guides the next round of search (Anthony et al. 2017). Rejection-sampling fine-tuning replaces tree search with repeated sampling and the expert with an automated check, which is why any blind spot in the check quickly becomes a habit of the loop.
Anthony, Thomas, Zheng Tian, and David Barber. 2017. “Thinking Fast and Slow with Deep Learning and Tree Search.”Advances in Neural Information Processing Systems (NeurIPS) 30.
Different learning objectives need different outcome mixes across attempts on a fixture. An all-success batch supplies valid imitation demonstrations showing clean paths through tools. An all-failure batch yields no end-to-end demonstrations under a success filter, yet still supplies localized recovery segments, negative candidates for preference pairs, and failure traces for new regression fixtures. Group-relative training objectives, by contrast, require outcome variation within the same fixture: because they score each rollout by its reward minus the mean reward of the rollouts sampled for that fixture, a batch in which every rollout gets the same reward produces zero advantage on every rollout and contributes no reward-surrogate gradient.
↪ Group relative policy optimization, which scores each rollout against the other rollouts sampled for the same task, is developed in Group Relative Policy Optimization.
The sampling budget required to harvest these different data types follows from the per-attempt pass rate. Suppose the collection policy succeeds on a fixture with probability \(\AttemptPassProbability\) per attempt, with each of the \(\IndependentAttemptCount\) attempts executed independently. A simple success filter extracts a demonstration from the fixture if at least one attempt succeeds, an event that occurs with probability \(1-(1-\AttemptPassProbability)^\IndependentAttemptCount\). This expected yield sets the baseline compute cost per retained trajectory when the collector keeps at most one success from each full batch. For preference comparisons based on binary task acceptance and group-relative objectives with binary rewards, useful within-group contrast requires at least one success and at least one failure. Subtracting both the all-failure probability \((1-\AttemptPassProbability)^\IndependentAttemptCount\) and the all-success probability \(\AttemptPassProbability^{\IndependentAttemptCount}\) yields the informative-task fraction:
With 8 attempts, a fixture the policy solves 5 percent of the time yields a success with probability only 0.34, requiring an average of about 24 attempts to harvest a single successful trajectory. That same fixture forms a mixed group of successes and failures with probability 0.34. At 50 percent, where the task is moderately difficult, the mixed fraction reaches 0.99, providing abundant pairwise contrast. When the pass rate reaches 95 percent, nearly every fixture yields a success, but the mixed fraction drops back to 0.34 because batches rarely include a failing attempt. Equation 3 measures contrast rather than general utility. All-success groups remain valid demonstrations for imitation learning, whereas Zero-contrast groups applies the \(\AttemptPassProbability^{\IndependentAttemptCount}\) and \((1-\AttemptPassProbability)^\IndependentAttemptCount\) terms to isolate groups whose uniform rewards offer zero reward-surrogate gradient in group-relative optimization.
Easy fixtures yield successes most often; mid-difficulty fixtures yield contrast most often.
Choosing fixture difficulty is therefore a resource allocation decision. Hard fixtures cost about 24 attempts per harvested success, but they force the agent into unfamiliar states and expose missing recovery skills. Easy fixtures cost almost nothing per success, serving as cheap regression anchors that preserve baseline competence. For rare or high-consequence task families, low harvest yields are a worthwhile price for keeping coverage intact. Recording the sampling and retention policies alongside the corpus makes these distribution shifts visible across retraining rounds, where repeated self-selection otherwise narrows coverage.
15.7.2 Staleness and coverage narrowing
A trajectory recorded minutes ago is already unusable if it called a deprecated tool schema or passed a flawed grader, whereas a trace recorded months ago by an older policy remains a valid demonstration as long as the tools and environment semantics have not changed. In an evolving agent system, staleness therefore measures architectural incompatibility rather than chronological age. Trajectory stores prevent silent corruption by binding four provenance fields to every record: the generating policy, the environment fixture, the tool interface schema, and the grading criteria. When grading criteria change, the pipeline re-runs verification; when tool schemas change, existing records undergo schema migration and revalidation; and as the policy improves, sampling shifts toward the harder edge cases the updated agent still fails.
Even when every schema and grader stays current, repeated self-training narrows behavioral coverage. Because strategies that cleared earlier filters appear more often in fine-tuning data, the updated policy reproduces those paths more often and samples them even more heavily in the next round. Left unchecked, training repeatedly on a model’s own filtered outputs shrinks distributional diversity, a form of model collapse (Shumailov et al. 2024) driven by the hidden feedback loop between a model’s outputs and its future training set (Sculley et al. 2015). Automated verifiers reject failures covered by their checks, but filtering alone cannot break that self-reinforcing loop; teams must measure and bound the drift directly.
Three telemetry signals warn when coverage is narrowing: the number of distinct normalized action sequences per fixture falls across rounds; pass@1 continues to rise while pass@\(\IndependentAttemptCount\) at large \(\IndependentAttemptCount\) flattens out, showing that the policy is concentrating probability mass on familiar routines rather than finding alternative solutions; and scores on held-out task families stall even as acceptance on training fixtures climbs. A drop in distinct action sequences can simply mean the agent is cutting out unneeded retries, so engineers read action diversity alongside held-out transfer scores: when shrinking diversity coincides with flat held-out performance, the policy is over-specializing.
Countering this narrowing requires bounding how heavily any single fixture or synthetic round can shape the dataset. At the fixture level, per-task retention caps keep easily solved problems from flooding the corpus, and deduplication filters drop identical normalized action sequences unless they occurred under genuinely different observations. Across retraining rounds, the corpus pairs synthetic runs with a permanent anchor of verified external demonstrations (such as curated human traces) that self-play cannot overwrite, while early-stopping rules halt the loop as soon as action diversity and held-out scores plateau, even if training-fixture acceptance keeps rising.
External demonstrations and held-out evaluation suites play distinct roles here. Curated human or expert demonstrations anchor the training mixture because they come from outside the current policy’s habits. Protected evaluation tasks, by contrast, stay outside both training and automated selection under the development and final-test split discipline of section 15.4. Because each promotion step in a self-improvement loop consults a holdout score, the benchmark that gates round-by-round promotion acts as a development set over multiple rounds, and a separate final-test suite must verify any final generalization claim. Periodic human audits across accepted, rejected, and borderline runs catch grader shortcuts and missed edge cases before promotion, while production telemetry supplies fresh task candidates from live traffic.
15.7.3 Production feedback signals
Production sessions expose edge cases that offline fixtures never anticipated, but treating raw user actions as ground-truth labels corrupts a corpus: a user who accepts a patch without editing it may never have run the tests, and a user who abandons a session may simply have been interrupted. Extracting training signals from live telemetry requires treating each observed event as evidence of the user’s immediate action rather than a verdict on task correctness (table 11).
Table 11: Production Feedback Signals: Preserve the observed event and its context before assigning a learning role. A signal may directly label its declared criterion without establishing unrelated properties.
Signal
Evidence Provided
Limitation
Possible Learning Use
Accepted without edits
Recorded acceptance of the delivered output
The user may not have checked correctness
Usability label under a declared criterion; candidate for task validation
Edited or reverted
A change to, or rejection of, the output
May reflect correction, style, or changed requirements
Proposed target or comparison after checking the applicable task and context
Mid-task correction
New user guidance at a particular turn
Does not establish the cause of the earlier behavior
Recovery context, intent clarification, or a candidate fixture
Retry or abandonment
A retry or session-ending event
Latency, cost, interruptions, and changed plans can produce the same event
Investigation or a label for the observed event, not assumed task failure
Explicit rating
The respondent’s recorded assessment
Missing responses and response selection limit population claims
Label for a defined satisfaction criterion, not automatic proof of task correctness
Misreading these signals distorts the training set. When a user edits an agent’s output to add a new requirement, the edited text answers the revised request rather than the original prompt; training on that edited output without updating the prompt teaches the model to guess unstated constraints. Likewise, pairing an agent’s draft with a user’s edit as a preference comparison is valid only when the context and task goal stay identical across both versions. Telemetry pipelines therefore treat missing feedback as unobserved rather than positive, check user consent and data-use permissions before ingestion, and turn recurring production failures into graded regression fixtures before feeding them into training. Together with retention caps and external anchors, these safeguards turn self-improvement into a concrete collection plan with explicit sampling budgets, automated verification stages, and human review gates. Because rollout generation scales out easily across accelerators while sandboxed verification and human review have tight throughput ceilings, the collection pipeline must pace sampling against downstream checking capacity so unverified traces never swamp staging storage.
15.8 Collection Pipeline Architecture
During a collection campaign, a fleet of rollout workers can finish hundreds of candidate traces while downstream test sandboxes are completely saturated, or a multi-predicate check can stall waiting on a single lagging reviewer after earlier checks have already passed. Because generation, verification, and dataset publication run at different speeds, structuring collection as an asynchronous pipeline decouples the stages within explicit capacity limits: when downstream verifiers or review queues fill up, backpressure throttles or pauses new rollouts before in-flight outputs exhaust staging storage.
15.8.1 Pipeline responsibilities
If a rollout worker tears down its execution sandbox before persisting the candidate workspace bytes, or if a network retry submits a duplicate trace without a stable run identity, the collector either loses the evidence for an accepted run or double-counts examples in the published corpus. To give every artifact and buffer reservation an explicit owner across handoffs, the pipeline divides into four responsibilities (table 12). These stages may share a single host or run as distributed services across a cluster; the division fixes ownership of work, state, and evidence rather than hardware layout.
Table 12: Collection responsibilities and handoffs: Each boundary transfers identified work under a capacity and evidence contract. Completing a rollout does not guarantee task acceptance or publication as a training example.
Reserve capacity for admitted work and its bounded output before starting the rollout
Rollout and capture
Execute through the trusted harness; record the available interaction history, candidate artifact, and capture gaps
Persist the required candidate bytes and manifest at a consistent boundary before releasing their only execution copy
Staging and assessment
Hold a bounded record or protected artifact references; run the selected cascade and any separately admitted review
Record criterion-level results and the proposed learning use, including unresolved assessments
Dataset publication
Apply the declared use, transformation, permission, and split policies to assessed records
Publish an identified derived version with its eligibility decision and provenance, or record another disposition
At each handoff, the pipeline checks an explicit contract. In a software repair collection run, for instance, the handoff from capture to assessment binds the persisted workspace bytes (ws_r20) to the acceptance policy (repair_accept_v1) and records which execution artifacts are available for the declared learning use within the trace capture limits. The verification cascade then runs the predicates of section 15.5, separating whether a rollout solved the task from whether it qualifies as a demonstration, a preference pair, or a rollout fixture. If a predicate cannot be completed immediately, the pipeline records its state as pending or unknown rather than treating it as a failure. Slow stages such as human review or formal verification pull cases from a dedicated, bounded queue; transferring a record to that queue requires an acknowledgment that buffer space is reserved, so stalled reviews cannot pile up and exhaust staging storage.
Because distributed workers and network links fail, message delivery is retried. Every submission is therefore bound to its collection run identifier, artifact version, and assessment policy. When a retry arrives with an identical payload under the same identifier, the stage returns the existing decision; if the payload differs from the earlier submission under that identifier, the pipeline routes the conflict to an operator queue. A retry never publishes duplicate training examples or frees a staging reservation twice, which sets up the quantitative sizing of those staging reservations.
↩︎ Semantic deduplication of retried writes, which acknowledges an identical retry and rejects a conflicting payload under the same identity, is established in The storage engine contract.
15.8.2 Collection credits and staging reserves
Staging needs hard admission bounds because how much it holds depends on verifier service time, and service time can collapse. Suppose completed rollouts arrive at verification at an average rate \(\ArrivalRate\), and \(\WorkerCount\) equivalent verifier slots each require a mean service time \(\MeanVerifierServiceTime\) per record. The pool provides an aggregate service capacity of \(\WorkerCount/\MeanVerifierServiceTime\) records per second, operating at nominal utilization \(\ServingUtilization=\ArrivalRate \MeanVerifierServiceTime/\WorkerCount\). In steady state, Little’s law (theorem 5.1) gives the mean number of records in service as \(\ArrivalRate \MeanVerifierServiceTime = \ServingUtilization \WorkerCount\), so the nominal utilization is also the fraction of slots occupied on average. That baseline holds only while the service time does, and the hazard that staging must be dimensioned for is the moment it stops holding.
↩︎ Little’s law, which ties mean occupancy to arrival rate and residence time, is established in Residence time and service rate.
The acute hazard is a sudden collapse in verifier service rate rather than a surge in rollout generation. Consider a defective test fixture or an unresponsive environment that forces every verifier execution to block until hitting its hard timeout \(\VerifierTimeout\). Aggregate checking capacity then falls from \(\WorkerCount/\MeanVerifierServiceTime\) to \(\WorkerCount/\VerifierTimeout\). If rollouts continue arriving at rate \(\ArrivalRate\), unassessed records accumulate in staging at a net rate of \(\ArrivalRate - \WorkerCount/\VerifierTimeout\) records per second. If each record reserves buffer space up to a maximum per-record ceiling \(\RecordSizeCeiling\), a staging storage pool of total capacity \(\StagingStoreCapacity\) that is empty when the burst begins is completely exhausted after
A store that already holds records at onset is exhausted sooner, in proportion to the space they occupy. Stopping dispatch does not stop growth at once, because rollouts already running still emit records. If those in-flight rollouts can reserve \(\InflightRolloutReservation\) and the controller needs \(\ControllerReactionDelay\) to act, Notebook 15.4 evaluates how fast staging fills on a representative verification fleet and where dispatch must stop, with the high watermark (the fraction of the store at which dispatch stops) bounded by
Napkin Math 15.4: How long a staging store survives a timeout burst
Problem: A hypothetical collector runs 1,024 concurrent rollouts that each take 45 s. 512 verifier slots take 16.875 s per record on average. Each record reserves a ceiling of 8 MB in a 32 GB staging store. Treat completions as a steady flow and all slots as interchangeable. A defective fixture then makes every test run to its 120 s timeout. How long until the store is fully reserved, and where must dispatch stop if the controller needs 30 s to act?
Math: Rollouts complete at 22.8 records per second against a nominal capacity of 30.3, giving a utilization of 0.75; by Little’s law, that is also the average fraction of the 512 slots in use. During the burst, capacity falls to 4.3 records per second, so staging grows by 18.5 records, or 147.9 MB/s of reservations, every second. Starting from an empty store, it is fully reserved after 216.3 s, about 3.6 min. When dispatch stops, the rollouts already running still hold up to 8.2 GB of reservations, and growth over the 30 s reaction delay adds 4.4 GB. Subtracting both from the 32 GB store places the high watermark at or below 60 percent of the store.
Systems insight: Size staging buffers for service-time collapses rather than nominal steady-state utilization. Even at a modest nominal utilization of 0.75, a verifier stall fills available storage in only 3.6 min. The exhaustion time \(\StagingExhaustionTime\) shrinks proportionally as the record ceiling \(\RecordSizeCeiling\) increases and scales linearly with total store size \(\StagingStoreCapacity\). Watermarks must account for work already in flight and controller reaction latency, backed by hard credit caps when feedback loops lag.
The in-flight term in that headroom shows why admission control must count unfinished work rather than queue depth. Queue length reflects past arrivals, while rollouts admitted before a verifier stall complete and emit records long after the dispatcher stops issuing new work. Admission controllers therefore limit both the count of active records and their aggregate reserved bytes, and they charge each reservation at the per-record ceiling \(\RecordSizeCeiling\) rather than the mean size, because mean sizes set steady-state storage while the ceiling bounds a worst-case burst.
A collection credit11 enforces both constraints through a single accounting token: every admitted rollout acquires one credit from a shared pool, reserving storage for its maximum possible payload throughout rollout execution, intermediate staging, and downstream assessment. The credit returns to the pool only when the record reaches a durable terminal disposition or completes an acknowledged handoff to another capacity-controlled queue. Consequently, every unit of pending work is explicitly accounted for across the entire cluster. When a trajectory exceeds its expected allocation, the capture harness enforces an explicit truncation policy rather than silently clipping required state, for example by recording the partial execution trace with marked omissions and flagging the artifact as ineligible for training regimes that require full fidelity. Because one credit spans all four responsibilities, the pool also bounds work that has left the collector but not yet reached a disposition (figure 4).
11Credit-Based Flow Control: The mechanism originates in network switches, where a transmitter may forward a cell only while holding a credit for a free downstream buffer slot, so buffers cannot overflow however bursty the traffic (Kung et al. 1994). The receiver returns the credit only after draining the slot, the analog of returning a collection credit upon durable disposition.
Kung, Hsiang-Tsung, Trevor Blackwell, and Alan Chapman. 1994. “Credit-Based Flow Control for ATM Networks: Credit Update Protocol, Adaptive Credit Allocation, and Statistical Multiplexing.”Proceedings of the Conference on Communications Architectures, Protocols and Applications (SIGCOMM), 101–14. https://doi.org/10.1145/190809.190324.
↪ Credit windows between cooperating agents, and the throughput a window of fixed size allows, are developed in Credit-based flow control.
Figure 4: Collection Credits Across Four Responsibilities: Each admitted run holds one credit through rollout, capture, staging, and assessment. The credit returns to the pool only when the record reaches a durable disposition or an acknowledged transfer to another bounded owner, such as the review queue. A watermark derived from in-flight work slows dispatch before the pool is exhausted, and a hard limit stops admission when that feedback arrives late.
The credit pool directly bounds the count and aggregate record volume of all outstanding runs. This mechanism governs staging buffers, not auxiliary execution resources. Accelerator memory for model inference, sandbox container filesystems, verifier scratch disks, and external object stores each require separate capacity pools with independent accounting; storing a lightweight URI reference in a message queue does not bound the multi-gigabyte container image or execution trace to which it points. Robust admission controllers verify reservations across all requisite resource pools up front. This pre-allocation invariant ensures that a pipeline stage never acquires one scarce resource, such as an accelerator slot, only to block indefinitely waiting for an unreserved downstream buffer. That credit bound holds even when a completion burst arrives while verification is paused, as notebook 15.5 works out for a hypothetical collector.
Napkin Math 15.5: Reserve capacity before the completion burst
Problem: A hypothetical collector provisions 12 end-to-end credits, each reserving a per-record ceiling of about 2.10 MB. Currently, 4 records sit staged in storage and 8 rollouts are actively executing, each holding an allocated credit while verification is temporarily paused. May the dispatcher admit an additional rollout? Furthermore, once all active rollouts finish and verification resumes, how long will this accumulated cohort take to drain?
Variables: 2 verifiers each assess one record at a time in exactly 10 s, including durable disposition; no new rollout is admitted during the drain, and transfer and publication add no delay.
Math: The 12 outstanding runs hold every credit: staged records reserve 8.39 MB and running rollouts 16.78 MB, for 25.17 MB (exactly 25165824 bytes) with 0 credits free, so the dispatcher admits nothing. Draining the queue takes ⌈12 / 2⌉ = 6 waves of 10 s, or 60 s.
Systems insight: Total byte consumption is strictly bounded by credits multiplied by the ceiling (\(\RecordSizeCeiling\)), independent of arrival timing, while drain latency scales as ⌈credits / verifiers⌉ times service time. Adding verifiers accelerates queue drainage without increasing the byte bound; expanding credits raises both. Engineers should dimension credit pools against physical staging limits and scale verifiers to meet drain deadlines, treating a slower assessment as a longer drain with the reservation still actively held.
The two models bound different quantities. Arrival and service rates govern how rapidly an outage consumes available headroom, while credit limits cap total buffer consumption regardless of timing. Production telemetry must continuously track queue depths, allocated vs. physically consumed bytes, the age of the oldest pending assessment, verification pass rates, and downstream sink throughput. Retries, escalated evaluations, and diagnostic reviews must all consume allocations from this same credit pool. When external dependencies fail, failure containment mechanisms prevent repeated re-verification attempts from amplifying an isolated service outage into cluster-wide failure.
↩︎ Circuit breakers, coordinated retries, and bulkheads that keep retry amplification from spreading a dependency outage are established in Tool Circuit Breakers.
15.8.3 The collection bottleneck
Under-provisioning cheap CPU verifier slots can throttle an entire accelerator fleet, leaving expensive generation hardware sitting idle while trajectories wait to be checked. While credit controls prevent staging buffers from overflowing during an outage, steady-state capacity planning compares generation, verification, and storage to find where throughput actually binds. With \(\AcceleratorCount\) accelerators each spending \(\AttemptGenerationTime\) accelerator-seconds per attempt, generation has a service ceiling of \(\AcceleratorCount / \AttemptGenerationTime\) attempts per second. With \(\WorkerCount\) verifier slots and expected slot service time \(\ExpectedCheckingTime\) per attempt from equation 1, assessment has a service ceiling of \(\WorkerCount / \ExpectedCheckingTime\). Charging every write at its per-record ceiling \(\RecordSizeCeiling\) gives a conservative provisioned storage rate of \(\DurableStoreWriteBandwidth / \RecordSizeCeiling\) for sustained write bandwidth \(\DurableStoreWriteBandwidth\). With enough credits, work, and scheduling capacity to saturate the stages, the idealized planning rate \(\SustainableAttemptRate\) is:
Equation 4 identifies the binding stage in this stipulated model. Its planned admitted-record rate is \(\SustainableAttemptRate \cdot \AdmissionRouteYield\), where \(\AdmissionRouteYield\) is the route yield. Actual throughput can be lower because of credits, scheduling, or another shared service. Smaller writes can exceed the conservative storage term, so that term alone does not diagnose an observed storage bottleneck.
The bound separates the stage that limits throughput from the stage that drives cost. In the illustrated low-priced deterministic route, verifiers occupy CPU slots that cost pennies per hour compared with accelerator instances, yet starving that pool stalls the whole collector. Under the notebook’s active-resource accounting, unused capacity is redeployed or unbilled, and generation compute and task yield dominate the charge per admitted record. A dedicated fleet billed during idle wall time has a different cost boundary. Deterministic verification is not inexpensive for every workload. A model judge or a human reviewer can change that split, and Exercise 5 in section 15.14 prices a route in which human review dominates. Notebook 15.6 evaluates this economic trade-off on a representative deployment.
Napkin Math 15.6: The cheapest stage can set the rate
Problem: A hypothetical collector uses 64 accelerators, each spending 60 s of accelerator time per attempt at $2.50 per hour. It checks attempts with the admission route of notebook 15.2, whose cheaper order occupies 6.24 s of slot time per attempt and admits 20.52 percent of attempts, on 4 verifier slots priced at $0.04 per slot-hour. Records of at most 8 MB go to a store that sustains 200 MB/s. Assume sufficient end-to-end credits and no further shared service bottleneck, charge each write at its record ceiling, and bill only active accelerator and occupied verifier time, with unused capacity redeployed or unbilled. Which stage binds this planning model, and what is the active-resource charge per admitted record?
Math: By equation 4, generation supports 64 / 60 s = 1.07 attempts per second, assessment 4 / 6.24 s = 0.64, and conservative provisioned storage supports 200 MB/s / 8 MB = 25. Assessment binds in this model, and matching generation needs 1.07 × 6.24 s, rounded up to 7 slots. Generation costs $0.042 per attempt; adding checking and dividing by the yield gives $0.20 per admitted record, of which checking is 0.2 percent.
Systems insight: Throughput bottlenecks and cost drivers are often decoupled. In this scenario, running only 4 inexpensive verifier slots leaves 39.9 percent of generation capacity unused. Despite this bottleneck, the financial cost of an admitted record is dominated by generation expense divided by yield, following the cost per accepted task formulation (equation 1.2). Adding verifier slots raises the idealized rate until generation saturates while leaving the active-resource charge per record unchanged. With a dedicated fleet billed during idle wall time, greater utilization can instead lower the bill per emitted record. Holding active work per attempt and resource rates fixed, halving the yield to 10.26 percent doubles the unrounded cost per admitted record, displayed to the nearest cent as $0.41. Fixture design and sampling policy control that yield; their measured cost and coverage consequences determine whether a change is worthwhile.
15.8.4 Artifact capture and resource release
Tearing down a rollout container before its logs and workspace diffs reach durable storage destroys the only copy of the attempt, while reassigning a pooled container before background writers exit lets a stale rollout pollute the next task. Collection pipelines therefore inherit the runtime’s sandbox lifecycle and reset contracts. Rollout workers must persist all candidate artifacts, interaction traces, and evaluation evidence to durable storage before tearing down the execution sandbox, and a pooled environment cannot be reassigned until background writer processes terminate, stale network credentials and access tokens are revoked, and filesystem reset postconditions pass. If a downstream verification stage needs interactive access to unpersisted sandbox state, it must hold that environment under an explicit time-bounded lease and operational budget; otherwise, the checker logs the assessment as unavailable and releases the host.
When a worker’s dispatch lease expires, expiration does not instantly terminate the executing process or roll back its external side effects. A replacement worker must therefore establish explicit ownership and verify fresh admission checks, while late-arriving outputs from the orphaned worker remain strictly quarantined to their original run. Similarly, if container cleanup fails or an external side effect leaves an indeterminate state, the owning process retains responsibility under the trajectory saga, even if the associated trajectory is discarded from training. Releasing a collection credit frees buffer staging, but it does not absolve the runtime of its teardown and cleanup obligations. Once physical movement from execution to durable disposition is bounded and accounted for, data governance determines artifact admissibility: which upstream source versions justify inclusion, what lossy or privacy-preserving transformations were applied, and which dataset split partitions and license restrictions govern the resulting weights.
↩︎ The trajectory saga, which keeps a compensation obligation open while an external effect remains unresolved, is established in The Trajectory Saga Pattern.
15.9 Trajectory Provenance Tracking
Months after an execution trace enters a training corpus, curators must answer concrete questions about its origin, validity, and rights. They need to know which collection run produced the sequence, which validation criteria were checked against which environment fixtures, whether the organization retains legal permission to train on the incorporated source code or API payloads, and whether a sanitization pass that stripped an API token or network address invalidated the runtime state that justified the agent’s next action. A preserved execution trace and its evaluation scores cannot answer these questions on their own. In a production curation pipeline, provenance provides the immutable, versioned record that answers them across every derivation of a trace. Consider a concrete recovery pipeline: a raw collection record, trace_r19_01, originates from collection run collection_run_r19_01 and candidate workspace ws_r20. If an automated scrubbing pass removes sensitive network addresses to create trace_r19_01_redacted_v1, the transformed artifact must receive an independent identity and its own admission decision instead of silently replacing the source record and inheriting its labels.
15.9.1 The lineage envelope
When a source repository revokes training permission or a grader bug invalidates a week of checks, an unindexed store of raw JSON traces cannot identify which corpus members must be pulled. A lineage envelope12 prevents that blind spot by recording the origin, dependencies, transformations, and admission decisions of each individual record version. Table 13 outlines its six dimensions. Rather than embedding inline copies of sensitive or high-volume execution state, envelope fields point to content-addressed, access-controlled artifacts.
12Why- and Where-Provenance: Database research distinguished two questions about a derived datum: which source records justify its existence (why-provenance) and which source locations its values were copied from (where-provenance) (Buneman et al. 2001). A trajectory record needs both. Where-provenance traces each retained field to the run, event, and transformation that produced it, and why-provenance traces its admission to the assessments that justified it, so a withdrawn source or a revised criterion can be followed to every affected corpus member.
Buneman, Peter, Sanjeev Khanna, and Wang-Chiew Tan. 2001. “Why and Where: A Characterization of Data Provenance.”Database Theory – ICDT 2001, Lecture notes in computer science, vol. 1973: 316–30. https://doi.org/10.1007/3-540-44503-X_20.
Table 13: Trajectory lineage envelope: Versioned references connect a record to its origin and subsequent decisions. Their presence does not certify correctness, complete capture, or permission for every use.
Record dimension
Information to bind
Question it answers
Collection identity
Run, task, fixture, candidate, and source-record versions; sampling or selection policy
Which attempt and inputs does this record represent?
Generation configuration
Available model and adapter identity, tokenizer, delivered-context references, tool interfaces, harness configuration, and sampling settings
What generated and mediated the represented proposals?
Environment and capture
Relevant initial-state and dependency versions, operation/result references, retained boundaries, omissions, and unresolved effects
Which interactions and evidence are available, and what is missing?
Assessment
Criterion and grader versions, per-criterion verdicts, evidence references, assessment time, and intended learning use
What was checked, on which artifact, for which admission decision?
Permission decision and scope, retention restrictions, split/group assignment, and corpus membership
Where may this version go, and which evaluation boundary follows it?
The envelope must record only what the runtime interfaces actually reveal. When an agent executes against locally controlled model weights, the envelope binds the loaded base model and adapter checkpoint digests. Hosted inference endpoints, however, typically expose only a reported version string and request configuration; in those cases, the envelope documents those exact parameters rather than fabricating an unverifiable weight hash. Similarly, a prompt-template identifier fails to capture dynamic context injected at runtime, and a container base-image digest omits mutable mounted volumes or external network state. Systems engineers must document omissions explicitly rather than relying on identifiers of ambiguous scope.
Evaluation assessments bind directly to the exact byte representation tested. If a trajectory earns a passing score on a targeted capability test, that verdict applies only to that evaluated representation; any pending criterion remains pending when the record enters cold storage. The downstream evaluation cascade alone determines whether the available evidence authorizes a specific learning use.
15.9.2 Integrity and authentication
A trainer decides what to train on by reading the corpus manifest, so it must be able to confirm two things before the first gradient: that the manifest is the one curation issued, and that no record was altered or swapped after admission, whether by a broken pipeline script or by anyone with write access to the store. Integrity mechanisms answer those two questions. They establish unmodified bytes, not correct content or permitted use. A cryptographic digest identifies a specified byte representation under standard collision-resistance assumptions. For that digest to remain meaningful across system components, the lineage envelope must fix both the canonical serialization scheme and the schema version over which the hash was calculated. Comparing an artifact against an expected digest detects bit corruption and unauthorized edits. Storing an unauthenticated digest directly beside mutable content in an object store, however, provides no protection, because an adversary or a broken pipeline script can overwrite the data and the digest together.
To guarantee authenticity, an authenticated manifest cryptographically binds expected artifact digests and associated metadata to a trusted issuing authority. Systems commonly use a message authentication code (MAC), such as HMAC as specified in RFC 2104 (Krawczyk et al. 1997), computed with a secret key held outside the dataset in a key management service. Because any party holding a shared symmetric key can forge a valid MAC, production architectures often prefer asymmetric digital signatures (such as Ed25519) when datasets are distributed across multiple worker nodes. The manifest must record the cryptographic algorithm, key version, and authorized issuing scope, and key rotation policies must match the operational lifetime of the corpus. Verification confirms only that the authenticated byte representation has not changed since issuance; it covers nothing outside that boundary.
Krawczyk, Hugo, Mihir Bellare, and Ran Canetti. 1997. HMAC: Keyed-Hashing for Message Authentication. RFC 2104. Internet Engineering Task Force. https://doi.org/10.17487/RFC2104.
13Merkle Tree: A tree in which each internal node holds the hash of its children, introduced to build digital signatures from a conventional one-way function (Merkle 1988). Its useful property is that proving one leaf belongs under a trusted root needs only the sibling hashes along one path, logarithmic in the number of leaves. A corpus manifest can therefore commit to millions of record digests under one root and still verify a single member cheaply, but the root commits only to the leaves that were hashed.
Merkle, Ralph C. 1988. “A Digital Signature Based on a Conventional Encryption Function.”Advances in Cryptology — CRYPTO ’87, Lecture notes in computer science, vol. 293: 369–78. https://doi.org/10.1007/3-540-48184-2_32.
Cryptographic integrity must not be confused with semantic correctness. A faithfully authenticated trajectory trace can still hold hallucinated model outputs, an erroneous grader verdict, or private data that downstream consumers are prohibited from seeing. Similarly, computing a Merkle root13 over captured execution events proves that a given event belongs to the recorded tree, but it proves nothing about execution events that the logging harness dropped or failed to observe. Claims about trajectory reproducibility must therefore stay within the reconstruction and re-execution boundaries established in Recorded Trajectory Replay.
15.9.3 Derived records
Scrubbing a private file path or API token from a recorded trace in place can sever the causal link that justified the agent’s next command while erasing the only copy of what actually ran. At the same time, the data minimization and disclosure boundaries established in Disclosure minimization must be enforced before trajectory content enters an ingestion queue, a model-assisted curation worker, or a training store. To resolve that tension, a derived record is always created as a new version rather than an in-place modification. The derived record binds a reference to its parent trace, the versioned redaction policy and implementation, the explicit set of altered or excised fields, and any intentionally withheld content. Lineage references do not grant read authorization to restricted raw data, and systems must store original raw bytes only where retention policies explicitly permit them. When source records reach their retention expiration limit and are purged, the lineage envelope documents that boundary directly; retaining a cryptographic hash is not the same as retaining the source data. The source also keeps a reverse reference to each derived version, so that a later correction or withdrawal at the source reaches every artifact derived from it (figure 5).
Figure 5: A Derived Record and Its Lineage: Redaction produces a new record version instead of editing the source. The derived record carries its parent reference, the fields its transformation changed or omitted, and the source’s split group, but its assessment is pending until revalidation decides what the new view still supports. The source keeps a reverse reference, so a later correction or withdrawal can reach every derived version.
Heuristic pattern matching and named-entity recognition models act as screening mechanisms rather than complete sanitization guarantees. Engineers must validate these detectors across the full output pipeline using realistic end-to-end test cases, including deeply nested payloads and generated metadata. Synthetic canaries provide an empirical measure of detector recall: the curation audit log must record how many canaries were injected, how many escaped detection, which semantic classes and text transformations were tested, and where the inspection boundary was applied. Observing zero canary escapes bounds the leak probability only for the evaluated canary distribution. False positives carry their own cost, because over-aggressive scrubbing strips necessary context and distorts the training signal.
↩︎ Canary values, planted synthetic secrets whose escape reveals a leaking path, are established in Detectors and canary values.
When scrubbing identifiers across multi-step interactions, curators use consistent pseudonyms to preserve relational dependencies across turns while preventing identity linkage beyond the access scope that the trace’s disclosure boundary permits. Tokenization effects require careful handling: replacing an identifier with a fixed-width placeholder often alters the sequence’s token count and attention boundaries. Similarly, reserializing a structured payload can preserve outer JSON or YAML formatting while silently breaking nested shell escaping, regular expression matches, or tool invocation syntax. Consequently, checking that an artifact parses successfully is distinct from verifying that it still supports the intended learning target.
↪ Re-verifying selected response spans against token offsets after templating or redaction shifts token boundaries is developed in Rendering with versioned templates.
Revalidation must systematically test every property that the transformation could have influenced. This includes validating record structure, ensuring action-observation consistency, verifying that the sanitized context still justifies the target action, and re-evaluating any criteria whose supporting evidence was altered. If a transformation modifies executable code or environment-dependent identifiers, the pipeline must execute the code against an isolated, authorized test fixture to obtain fresh runtime evidence, while archiving prior execution logs as historical artifacts.
Example 15.1: A transformed record needs a new decision
Scenario: During a collection run, an agent’s observation records a private system path that a subsequent diagnostic command references, and the source trace captures the successful output of that command. The curation pipeline replaces the path with a sanitized placeholder throughout trace_r19_01_redacted_v1.
Diagnosis: The recorded execution output remains valid evidence only for the original run bound to trace_r19_01; the agent never executed the sanitized placeholder path. The curator must determine whether the sanitized observation and the subsequent command still constitute a coherent learning example. A consistent placeholder can teach an agent how to parse and inspect arbitrary paths without exposing production directory structures. However, replaying the trajectory against an active execution harness requires an authorized, compatible mock fixture; otherwise, the substituted path fails to resolve. If the substitution obscures the causal distinction that prompted the agent’s action, the pipeline must reject the target for supervised training or withhold the record. The derived record inherits the source’s task grouping and train/test split assignment, while all pending evaluation verdicts remain pending until formal revalidation completes.
Systems lesson: Cryptographic integrity establishes which source artifact was transformed, whereas revalidation establishes what the transformed trajectory teaches. Neither operation can alter what physically occurred during the original execution run.
15.9.4 Use permission and withdrawal
Permission to execute an agent against a task fixture or inspect an internal repository does not convey permission to retain its artifacts, fine-tune models on its data, or publish the resulting corpus. The lineage envelope must record the explicit licensing constraints, the evidentiary basis for data retention, and the designated policy authority or human reviewer who approved data usage. This accounting must cover both the primary task assets and all third-party software dependencies pulled into the environment during execution. Automated license detectors and embedding-based similarity screens help identify suspect components, but they serve as screening tools rather than autonomous legal adjudicators.
When provenance evidence is ambiguous or incomplete, the curation pipeline must withhold the record from downstream consumers while logging the specific withholding rationale. This quarantine is use-specific: an asset held from public pretraining might remain eligible for internal safety evaluations. Furthermore, use-permission holds operate independently of task failure labels, confidentiality tiers, and fault-containment quarantine. Resolving a safety fault or task defect does not automatically grant data-usage clearance.
⇄ Contrast: Incident containment, which holds the work a suspect trajectory might touch rather than withholding a record from one use, is established in Incident Containment.
Approved data permissions operate under strict temporal and organizational scopes. To enforce policy changes, reverse references must connect each source record to its derived views, tokenized dataset partitions, and published corpus manifests. When an upstream data provider revokes permission, or when an organization updates its privacy policies, the system must propagate the restriction or withdrawal to every downstream artifact. The ingestion pipeline immediately halts new admissions under the revoked policy, while the catalog traces all distributed artifacts. Simply deleting a record from an active storage bucket leaves distributed offline copies and completed model parameter updates intact; managing those downstream exposures requires explicit operational ownership, retraining schedules, and audited unlearning procedures.
Checkpoint 15.2: Defend a derived record's use
Each bullet asks for a decision a curator makes before a derived record ships.
A well-architected trajectory record has a verifiable origin, an explicit transformation history, a legally scoped permission envelope, and preserved evaluation boundaries. Connecting an execution trace through this provenance graph into a corpus manifest gives membership a precise meaning: verified fitness for a declared learning consumer. Carrying one record through every stage shows how these decisions fit together.
15.10 Case Study: Curating the Repair Record
To see how the curation stages compose, return to the recovery run collection_run_r19_01 whose targets section 15.6.2 selected. Its refused edit, schema inspection, and corrected edit produced ws_r20, which satisfied repair_accept_v1. The raw trace trace_r19_01 records that sequence without altering any earlier record, neither the pending trial verdict on ws_r19 nor the unresolved earlier test operation test_r18. The curation question is which parts of this record may enter a learning artifact, for which consumer, and on what evidence.
The training objective is recovery from harness refusals, the residual behavior that the campaign’s probes left with the model, and the target policy recovery_targets_v1 already fixed which tokens contribute direct supervised loss: the schema query e3, the corrected edit e4, and the final summary e6, with the refused proposal e2, the test summary e5, the task prompt, and the intermediate tool responses kept as conditioning context. Target selection is only the first stage of admission. The record that actually enters a corpus is a transformed view of trace_r19_01, so the remaining stages must verify that view, bind its provenance, and publish it to a manifest.
15.10.1 Admission of the derived view
Replacing sensitive fixture paths in trace_r19_01 with synthetic placeholders produces trace_r19_01_redacted_v1, but if that redaction strips the file names the agent needed to deduce the schema, training on the redacted trace teaches the model to guess rather than inspect. While the underlying execution stems directly from the original run, admission filtering must verify whether this sanitized view preserves the causal chain of the recovery lesson rather than assuming the raw run’s pass carries over. The admission pipeline does not demand byte-for-byte fidelity with historical environment state; it requires that the transformed trace still support supervised imitation of the selected targets. Table 14 evaluates this derived view against the curation cascade’s admission predicates, where each passing verdict in a production manifest links directly to an underlying verification log.
Table 14: An Admission Decision for a Derived Recovery Record: The predicates apply to identified evidence and a specified consumer. A missing required assessment leaves the decision pending or unknown rather than inheriting an unconditional pass.
Admission predicate
Evidence and decision for this record
Remaining boundary
Record adequacy, \(\text{chk}_{\text{record}}\)
The retained source and transformation records preserve the relevant delivered context, event associations, refusal, correction, and candidate-specific results
Retaining the malformed proposal provides necessary conditioning context without selecting it as a supervised training target
Permitted use, \(\text{chk}_{\text{use}}\)
The approved-use decision covers this derived view and imitation use; its source task group was assigned to training before collection
Path redaction maintains the original task group and split assignment without granting broader permissions or creating an independent evaluation task
Process conditions, \(\text{chk}_{\text{proc,demo}}\)
The declared assessment permits correction of this malformed proposal and supports the required authority, resource, and residual-duty conditions
Because the harness rejected the malformed proposal prior to dispatch, no unauthorized disk modifications occurred; other prohibited operations require distinct safety evaluations
A use-specific review finds that the transformed context supports the selected schema inspection, corrected edit, and bounded report
Admitting these specific target tokens for supervised imitation does not validate other model turns or authorize arbitrary downstream tasks
Because each predicate evaluates to true, the pipeline admits the transformed example as a verified decision demonstration. Had path substitution stripped the file names essential for the agent to deduce the correct schema, target suitability (\(\text{chk}_{\text{target,demo}}\)) would have failed. Furthermore, while this redacted trace is valid for token-level imitation learning, it cannot serve a downstream consumer that requires executable replay, because executing the synthetic paths would require spinning up a fresh environment with matching mock fixtures. A single execution lineage can therefore support fine-tuning while being disqualified from offline replay benchmarks. Neither a valid cryptographic digest nor an overall task pass provides blanket approval across different training regimens.
15.10.2 Corpus manifest publication
Admitting trace_r19_01_redacted_v1 into a storage bucket without binding its target mask, split group, and grader versions leaves a downstream trainer free to supervise the refused edit e2 by mistake or leak the task into evaluation. Publishing the curated trajectory to the corpus release repair_demonstrations_v1 prevents that drift by recording every link in the artifact’s provenance chain inside the corpus manifest. For this member, the manifest binds the record to its originating run (collection_run_r19_01), task identifier, fixture snapshot, candidate code patch, and evaluation grader versions. It references both the transformed trace (trace_r19_01_redacted_v1) and the target selection policy (recovery_targets_v1), while logging the specific data governance approvals, task group assignments, and split partition versions. Any missing metadata or inactive model dependencies are explicitly tagged as unavailable rather than omitted.
At the corpus level, the manifest catalogs the source data pools, sampling weights, token accounting budgets, domain coverage metrics, and the formal evaluation protocol. Maintaining clear distinctions among raw interaction counts, stored trace tokens, and selected supervised target tokens prevents deceptive accounting. During training, serialization formats and loss masks enforce these target boundaries, ensuring that future trajectory steps never leak into earlier conditioning windows.
This curated trajectory yields a single demonstration artifact. It cannot be repurposed as a preference pair, because the rejected proposal e2 and the later edit e4 follow different prefixes (e4 comes after the refusal and the schema lookup), so they are not alternative continuations of one context. A preference artifact would need competing completions sampled from one saved branch point. Similarly, publishing an environment for reinforcement learning rollouts requires exporting the task fixture, harness configurations, and dynamic graders, rather than static token sequences.
Publishing an immutable corpus manifest establishes a versioned training baseline, which remains separate from deployment governance for any resulting model. Tracking lineage guarantees that if an upstream fixture defect or data privacy request emerges later, pipeline operators can cleanly trace and purge all derived artifacts across downstream model versions. At the infrastructure layer, ephemeral execution sandboxes are destroyed and billed compute charges are settled independently of whether a trace reaches the corpus manifest.
15.10.3 Cost per emitted artifact
In agentic data collection, raw trial volume is a deceptive metric. A pipeline may execute thousands of autonomous interactions yet emit only a small handful of usable training samples. Trajectories routinely fail safety filters, duplicate existing task solutions, crash during tool dispatch, or remain stalled in asynchronous evaluation queues. Consequently, systems engineers must evaluate collection economics using the cost per emitted artifact, amortizing all spent compute and infrastructure overhead across the accepted dataset.
For a given collection cohort and accounting cutoff, let \(\CollectionCohortCost\) represent the total monetary expenditure incurred across environment provisioning, inference generation, tool sandbox execution, automated verification, human review, redaction transformations, and storage. Let \(\EmittedArtifactCount\) denote the number of valid artifacts emitted for a designated consumer by that cutoff. The cost per emitted artifact is defined as
The numerator \(\CollectionCohortCost\) captures all unproductive runs, rejected proposals, and pipeline failures; if a cohort yields zero usable artifacts (\(\EmittedArtifactCount = 0\)), the pipeline registers the incurred financial loss without dividing by zero. Structurally, equation 5 mirrors the cost per accepted task formulation (equation 1.2): the unit cost equals the average cost per attempt divided by the end-to-end emission yield. This metric isolates data acquisition efficiency, distinguishing curation costs from both runtime inference serving and subsequent model fine-tuning.
Napkin Math 15.7: Cost follows the emitted artifact
Problem: A collection cohort initiates 500 collection attempts, incurring $300 in fully accounted costs across environment provisioning, generation, tool calls, automated verification, curation transforms, storage, and human review. The demonstration selector emits at most one artifact per task. What does each emitted artifact cost, and which operational lever most effectively reduces that price?
Table 15: Cohort Dispositions for One Consumer: These hypothetical counts belong to this acquisition worksheet, not the earlier token-pool or cascade-planning examples. The detailed registry retains each nonselection reason.
Disposition at the cutoff
Attempts
Meaning for this demonstration artifact
Selected and emitted
15
All required admission conditions and selection decisions are complete
Otherwise assessed but not selected
470
Record and use conditions pass; a recorded process, outcome, target, or redundancy decision excludes the attempt
Ineligible for this use
10
A specified split or transformation-use condition fails; no other row counts these attempts
Pending assessment
3
A required decision is still expected within its observation window
Unknown at the cutoff
2
Required evidence is unavailable or insufficient, with the reason retained
Total accounted
500
Every initiated attempt appears once
Math: The selected-and-emitted row of table 15 yields an emission rate of 15 / 500 = 3 percent. The average cost per attempt is $300 / 500 = $0.60. Applying equation 5 yields $0.60 / 3 percent = $20 per emitted artifact. This emission yield differs from a task-completion rate, because the curator caps repeated solutions for the same task and several nonselected runs successfully resolved their repair targets. It is also far below the route yield of notebook 15.2, which counts the records that pass one route’s predicates. Emission yield charges every initiated attempt with the per-task cap, redundancy decisions, ineligibility, and whatever is still pending at the cutoff.
Systems insight: Because unit cost equals attempt cost divided by emission yield, curation yield serves as the primary economic lever. For example, raising yield to 6 percent cuts the unit price in half to $10. With 97 percent of attempts failing to emit a usable artifact, engineering effort should focus on diagnosing nonselection causes before purchasing additional generation compute. High redundancy indicates that the system should launch fewer attempts per task; ineligibility points to split or permission defects caught too late; and backlogged pending evaluations point to verification bottlenecks.
A published manifest documents what was emitted, verifies eligibility across all admission gates, isolates supervision targets, and tallies total acquisition cost. Operational efficiency and curation hygiene, however, are prerequisites rather than guarantees of model improvement. The test of a curated corpus is offline evaluation against held-out task groups under controlled benchmark protocols, where neither a low cost per artifact nor a high emission yield can substitute for task performance.
15.11 Fallacies and Pitfalls
A data collection pipeline can preserve every execution record faithfully and still assemble a counterproductive training set. The mistakes below stem from confusing raw trace telemetry with the deliberate pedagogical choices required to train an agent, and each carries tangible computational or behavioral costs.
Fallacy: A successful task makes every action in its trace a suitable demonstration.
Task acceptance evaluates end-to-end task completion, whereas target selection decides which specific token sequences should receive gradient supervision. Consider an agent that successfully debugs a codebase after several stumbling attempts. Its transcript might include an initial malformed JSON tool call, a parser rejection from the runtime environment, and a subsequent correction. While that malformed call must remain in the transcript as conditioning context so the agent understands the environment’s feedback, it must never be included in the training loss as a target. In the recovery worksheet, a typical recovery trajectory contains 18,500 stored tokens, but only 2,000 of those tokens are designated as supervised targets (section 15.6). Supervising the entire accepted trace forces the model to memorize and reproduce its own intermediate errors. In a sound curation pipeline, record validity (whether the telemetry accurately reflects what happened), task acceptance (whether the agent fulfilled its objective), and target eligibility (whether an action should be reinforced) are three distinct filtering decisions.
Pitfall: Fine-tuning on every failure before probing its cause.
When an agent fails, the easiest operational reaction is often to bundle the failed trajectories into a fine-tuning dataset. In the intervention ladder worksheet, systematic counterfactual probes demonstrate that 90 percent of observed failures originate in the supporting infrastructure below the model weights, such as malformed tool schemas, missing environment variables, or overly aggressive sandbox timeouts (section 15.2.2). Retraining the model across the entire failure corpus requires only $58 of accelerator time, and that modest price creates a dangerous illusion of progress. Fine-tuning merely encourages the model to memorize ad hoc compensations for broken tooling. As soon as an external API changes or the brittle workaround misfires, the failure resurfaces. Systems engineers should run counterfactual probes first, fix defects in tools and runtime environments directly, and restrict fine-tuning to the remaining 300 genuine behavioral failures.
Fallacy: A random split of collected traces holds out unseen tasks.
Applying uniform random sampling across collected trajectories partitions individual attempts rather than underlying problem instances. When an evaluation harness collects 8 attempts per task and allocates a 80 percent fraction to training, 83.2 percent of the tasks end up present in both the training and test sets (section 15.4). Under this configuration, test metrics reflect the agent’s ability to recall previously seen problem setups rather than its capability to generalize to novel tasks. Data splitting must occur at the granularity of task groups, with environment-seed variants inheriting their group’s assignment. Furthermore, any downstream derivatives (redacted transcripts, paraphrased prompts, or deduplicated variants) must inherit the assignment of their source task so that syntactic transformations do not accidentally leak problem state across the evaluation boundary.
Pitfall: Cheapening the verifier cascade by dropping or loosening a predicate.
When verification overhead begins to strain cluster budgets, engineers often attempt to reduce costs by removing expensive check predicates or loosening test assertions. This strategy alters the semantic definition of success rather than optimizing the system. In the cascade worksheet, simply reordering evaluation stages based on their computational cost per rejected candidate saved 16.9 percent of total verification work while maintaining the exact same candidate yield of 20.52 percent (section 15.5.3). Dropping a verification check or silently discarding unresolved edge cases shifts the distribution of accepted trajectories, degrading the downstream training corpus. Performance optimizations should focus on scheduling predicates efficiently under fixed acceptance criteria. In addition, execution pipelines must isolate and report inconclusive runs separately rather than collapsing them into either rejections or unqualified admissions.
Fallacy: Setting a recovery percentage by record count fixes the training mixture.
Trajectory counts provide an unreliable unit of measure when balancing mixed training datasets. A curation system operates across three distinct metrics: raw trajectory records, total stored context tokens, and actively supervised target tokens. In the token budget worksheet, recovery trajectories make up 25 percent of total records, but they represent 44.97 percent of stored tokens and 39.06 percent of selected training targets (section 15.6). Because recovery sequences involve multi-step error correction and verbose environment feedback, their token footprint differs substantially from standard trajectories. Specifying mixture proportions solely by record count miscalculates both the gradient signal allocated to recovery behaviors and the computational budget required for training. Engineers must define training mixtures in terms of supervised target tokens, and validate the resulting policy resilience against held-out task benchmarks under a declared compute budget.
Pitfall: Sizing collection staging from average verifier utilization.
Dimensioning staging storage against average verification throughput invites buffer exhaustion during outages. A verifier pool running at an average utilization of 0.75 appears to have substantial headroom during normal operation. However, automated verification runtimes suffer from bimodal service times. When a faulty test fixture, infrastructure timeout, or broken container environment forces every evaluation to run until its hard execution deadline, service throughput collapses while incoming agent trajectories continue to arrive. In the burst exhaustion worksheet, this prolonged tail latency exhausts the staging store in approximately 3.6 min (section 15.8). Buffer capacity must be bounded by worst-case service latency rather than nominal arrival rates. Robust ingestion architectures use credit-based flow control to restrict outstanding verifier reservations, sizing storage watermarks to absorb both in-flight execution bursts and downstream controller response delays.
15.12 Summary
A trajectory record and a training example answer different questions. The record says what happened under stated conditions. The example asserts that some part of what happened is worth reproducing, for a named learner, and will be judged against evidence that learner never saw. Every mechanism in the chapter keeps those two apart: probes keep system defects out of the weights, task-group splits keep evaluation evidence out of selection, target selection keeps a faithfully recorded mistake in context and out of the loss, and lineage keeps a transformed record from inheriting a decision made about its source.
Collection credits, lineage envelopes, and the corpus manifest make those decisions operational. The price of the result follows the book’s cost-per-accepted-task form twice over: checking work per admitted record is cascade cost divided by yield, and cost per emitted artifact is attempt cost divided by emission yield, so yield is the lever in both. Self-improvement changes the source distribution with every round, which is why the manifest keeps demonstrations, comparisons, and rollout fixtures distinct and why external assessment stays outside the loop. The quantitative decisions of the chapter rest on the relations collected in table 16.
Table 16: Key Equations of Trajectory Curation: Each relation fixes a quantity that a split, admission, mixture, or collection-capacity decision depends on. The right column locates its derivation.
\(\JaccardSimilarity(1-\JaccardSimilarity)/\MinHashPermutationCount\) and \(\LshCandidateProbability=1-(1-\JaccardSimilarity^{\LshRowsPerBand})^\SignatureBandCount\)
Variance of the MinHash similarity estimate, and the banded index’s retrieval curve that sets review load
\(\StoredTokenVolume_j=\CategoryRecordCount_j\MeanStoredTokensPerRecord^{(\TrajectoryCategoryIndex)}\) and \(\TargetTokenVolume_j=\CategoryRecordCount_j\MeanTargetTokensPerRecord_j\)
Stored tokens of admitted records set compute, and selected targets set supervision; a mixture is specified in one, budgeted in the other
\(1-(1-\AttemptPassProbability)^\IndependentAttemptCount\) and \(\InformativeTaskFraction(\AttemptPassProbability,\IndependentAttemptCount)=1-(1-\AttemptPassProbability)^\IndependentAttemptCount-\AttemptPassProbability^{\IndependentAttemptCount}\)
Harvest yield per fixture under a success filter, and the fraction of groups that carry contrast
Key Takeaways: Give every learning artifact a declared use
Weights are the last remedy, not the first: Counterfactual probes place many failures in context, schemas, or the runtime, where a reversible repair removes them for every model version. Training earns only the residual that survives, and demonstrations, preference comparisons, and rollout fixtures each carry their own admission conditions.
A split protects a claim, not a file: With several attempts per task, a random split over traces puts most tasks on both sides, so the test score rewards recall of solved siblings. Assign whole task groups, carry the assignment into every derived record, and fix it before collection begins.
Cheaper checking must admit the same corpus: Ordering independent, fixed-cost predicates within permissible orders by cost per rejection, \(\CheckSlotTime_i/(1-\CheckPassProbability_i)\), cuts verification work at an unchanged accepted set. Dropping or loosening a predicate buys a different corpus, so its lower unit price is not a saving.
A recorded mistake is context, not a target: Recovery records keep the error and its refusal as conditioning history and select only the assessed correction as supervision. Mixtures are specified in selected targets and budgeted in stored tokens, because context per target differs by class.
Self-selected data narrows what it rewards: When the policy supplies its own records, rising acceptance can hide the recoveries the selector rarely keeps. Distinct action sequences per fixture, pass@1 against pass@\(\IndependentAttemptCount\), and an external anchor show whether the loop is still learning or only repeating itself.
Staging is sized for the burst, not the average: A verifier timeout burst, not average utilization, exhausts reserved staging, so each run holds a credit at the record ceiling until durable disposition. The cheapest stage can bind the illustrated planning rate while generation divided by yield sets its active-resource price.
The manifest is the contract the trainer reads: Records and derived artifacts bind to source versions, checks, transformations, permissions, split membership, and intended consumers. Integrity proves unmodified bytes, not truth or permission, so every derived record earns its own admission decision.
Verified trajectory post-training (principle 13) now has a concrete decision boundary: evidence supports a specified learning use, and the corpus manifest carries that decision through selection and transformation so the trainer can tell context from targets. The trained policy still faces evaluation and still runs under the runtime’s authority and resource controls.
15.13 Historical Perspective and Further Reading
The principles governing trajectory curation adapt lessons from data mining, information retrieval, and relational databases. Data mining first identified data leakage as a systemic failure mode where information about an evaluation target inadvertently reaches the learner through correlated records (Kaufman et al. 2012). In software modeling, duplicated source files were shown to inflate benchmark accuracy artificially until evaluation splits underwent thorough deduplication (Allamanis 2019). Modern screening pipelines borrow directly from web search, where minwise hashing estimated document resemblance from compact signatures (Broder 1997) and locality-sensitive hashing reduced high-dimensional similarity search to sublinear time (Indyk and Motwani 1998). Database query processing supplied the twin mechanisms of admission control and data provenance: optimizers established how to order expensive filter predicates by their cost-to-selectivity rank to minimize wasted computation (Hellerstein and Stonebraker 1993), while provenance frameworks formalized which source records produced a derived value and why (Buneman et al. 2001).
Kaufman, Shachar, Saharon Rosset, Claudia Perlich, and Ori Stitelman. 2012. “Leakage in Data Mining: Formulation, Detection, and Avoidance.”ACM Transactions on Knowledge Discovery from Data (TKDD) 6 (4): 1–21.
Zelikman, Eric, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. “STaR: Bootstrapping Reasoning with Reasoning.”Advances in Neural Information Processing Systems (NeurIPS) 35: 15476–88.
Uesato, Jonathan, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. “Solving Math Word Problems with Process- and Outcome-Based Feedback.”arXiv Preprint arXiv:2211.14275, ahead of print. https://doi.org/10.48550/arXiv.2211.14275.
Lightman, Hunter, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. “Let’s Verify Step by Step.”The Twelfth International Conference on Learning Representations (ICLR).
Learning from self-generated execution traces shares an equally deep lineage in reinforcement learning and symbolic reasoning. Classical imitation learning demonstrated that a policy trained exclusively on expert demonstrations compounds its errors when deployed, because small mistakes push the agent into unfamiliar states where it cannot recover (Ross et al. 2011). Ross, Gordon, and Bagnell showed that querying an expert on the learner’s own visited states restores linear error bounds; recovery demonstrations are the systems realization of that theoretical remedy. Expert iteration formalized this dynamic into an alternating loop of generation, search-based selection, and retraining, using tree search to discover improved decisions for a neural network to imitate (Anthony et al. 2017). Bootstrapped reasoning extended this loop to language models by sampling reasoning paths and filtering for traces that reached correct answers (Zelikman et al. 2022). Process supervision then tightened verification from coarse terminal outcomes down to individual reasoning steps (Uesato et al. 2022; Lightman et al. 2024).
Every self-training method eventually meets the same problem, because the filter that selects training data is itself an imperfect part of the system it is meant to improve. Production machine learning pipelines first exposed the systemic risks of hidden feedback loops, where an operational model’s outputs silently shape its own subsequent training distribution (Sculley et al. 2015). Reward modeling demonstrated Goodhart’s law directly: excessive optimization against a learned proxy drives reward scores up while true task quality deteriorates (Gao et al. 2023). Training repeatedly on a model’s own unchecked outputs speeds model collapse, in which diversity shrinks with each round and the rare cases in the tail disappear first (Shumailov et al. 2024). Agentic execution sharpens each of these failure modes because trajectories do not exist in isolation. They execute stateful tool calls, operate under specific environment permissions, and can inadvertently manipulate or game their own evaluation harnesses. The safeguards of this chapter, from verifier cascades and disjoint task-group splits to lineage envelopes and dataset manifests (Gebru et al. 2021), apply classical systems isolation and provenance guarantees directly to the trajectory as the atomic unit of machine learning data.
Shumailov, Ilia, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. “AI Models Collapse When Trained on Recursively Generated Data.”Nature 631 (8022): 755–59. https://doi.org/10.1038/s41586-024-07566-y.
For further reading, start with these:
End-to-end arguments(Saltzer et al. 1984): The classic statement of why a function belongs where it can be implemented completely and correctly. In trajectory curation, this principle argues for repairing tool interfaces and runtime environments directly rather than training model weights to work around platform flaws.
Predicate migration(Hellerstein and Stonebraker 1993): The foundational database paper on ordering expensive filter checks by cost per rejection, formalizing the exchange argument used to structure multi-stage verifier cascades.
Imitation learning as no-regret online learning(Ross et al. 2011): The analysis demonstrating why training exclusively on clean expert prefixes leaves an agent unprepared for its own mistakes, establishing the theoretical rationale for recovery demonstrations.
Hidden technical debt in machine learning systems(Sculley et al. 2015): A comprehensive systems catalog of data dependencies and feedback loops that any self-improving corpus pipeline inherits.
Datasheets for datasets(Gebru et al. 2021): The documentation discipline that a corpus manifest adapts into versioned, machine-auditable metadata fields.
Reward-model overoptimization(Gao et al. 2023): A controlled empirical study showing how aggressive optimization against a learned proxy reward drives proxy scores up while true task quality degrades.
Saltzer, Jerome H., David P. Reed, and David D. Clark. 1984. “End-to-End Arguments in System Design.”ACM Transactions on Computer Systems 2 (4): 277–88. https://doi.org/10.1145/357401.357402.
Hellerstein, Joseph M., and Michael Stonebraker. 1993. “Predicate Migration: Optimizing Queries with Expensive Predicates.”Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data, 267–76. https://doi.org/10.1145/170036.170078.
Sculley, D., Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. “Hidden Technical Debt in Machine Learning Systems.”Advances in Neural Information Processing Systems (NeurIPS) 28: 2503–11.
Gao, Leo, John Schulman, and Jacob Hilton. 2023. “Scaling Laws for Reward Model Overoptimization.”Proceedings of the 40th International Conference on Machine Learning 202: 10835–66.
15.14 Exercises
Every given below is a hypothetical problem input chosen for round arithmetic, not a measurement of any system. Each problem asks for a calculation and the curation or collection decision it supports.
Sizing a protected split (Basic). A collection job makes \(\IndependentAttemptCount=5\) attempts on each of 2,000 repair tasks. A colleague proposes assigning individual traces to training with probability \(\TrainingSplitFraction=0.9\) and to the final test otherwise.
Compute the fraction and number of tasks whose attempts straddle the partition, and the smallest \(\IndependentAttemptCount\) at which a majority of tasks straddle.
Compute the probability that a given test attempt has at least one sibling attempt of the same task in training, and state what the resulting test score measures.
The team switches to a group split (section 15.4). Treating tasks as independent trials with an acceptance rate near one half, compute how many task groups the final test must hold for the standard error of its task-level acceptance rate to be at most 0.025, and what fraction of the 2,000 tasks that reserves.
Ordering an admission route (Basic). A demonstration route has three independent predicates beyond record adequacy: a permitted-use lookup \(\text{chk}_{\text{use}}\) that takes 0.05 s and passes 70 percent of records, a sandboxed test run \(\text{chk}_{\text{out}}\) that takes 30 s and passes 50 percent, and a model-judged target review \(\text{chk}_{\text{target}}\) that takes 4 s and passes 80 percent. Target review does not need the test result, and the current order is \(\text{chk}_{\text{use}}\to \text{chk}_{\text{out}}\to \text{chk}_{\text{target}}\).
Rank the predicates by cost per rejection (equation 2) and choose the order.
Compute \(\ExpectedCheckingTime\) per attempted record and per admitted record for the current and chosen orders (equation 1), and the fraction of verifier-slot work the change saves.
A colleague proposes dropping \(\text{chk}_{\text{target}}\) instead, because it lowers checking time per admitted record further. Compute that figure and the new yield, and explain why the comparison with part (b) is not a cost comparison at all.
Three denominators for one mixture (Basic). A retained demonstration pool holds 20,000 direct-success records averaging 6,000 stored tokens and 1,000 selected target tokens, and 5,000 recovery records averaging 30,000 stored tokens and 2,500 selected target tokens.
Compute the recovery class’s share of records, stored tokens, and selected targets, and the context per target token for each class.
The team wants recovery to supply half of the selected targets. Holding total selected targets at the current value, compute the presentations each class needs, how many recovery records must repeat, and the change in processed tokens.
Holding processed tokens at the current value instead, compute the total selected targets at the same half-and-half mixture. State which budget this team should fix when training time is the scarce resource, and what the arithmetic cannot decide.
Spending a sampling budget on contrast (Intermediate). A collector has 8,000 attempts for 1,000 repair fixtures in three difficulty bands: 400 fixtures with per-attempt pass rate \(\AttemptPassProbability=0.1\), 400 with \(\AttemptPassProbability=0.5\), and 200 with \(\AttemptPassProbability=0.9\). Attempts are independent, and each fixture receives \(\IndependentAttemptCount=8\).
For each band and in total, compute the expected number of fixtures that yield at least one success and the expected number of mixed groups (equation 3). Compute the attempts spent per mixed group.
A colleague proposes dropping the easy band and spending its 1,600 attempts on the hard band at \(\IndependentAttemptCount=12\). Compute the new expected number of mixed groups and state what the imitation corpus loses.
Decide whether to adopt the proposal for a group-relative objective and for an imitation artifact, and explain why the easy band’s all-success groups are not wasted attempts.
Reviewing after selection (Intermediate). A cohort makes \(\IndependentAttemptCount=10\) independent attempts on each of 200 tasks. Each attempt succeeds with probability 0.2 and costs $0.50 for generation, verification, and storage. Every successful attempt goes to human target review at $6, which passes 75 percent of the successes it sees, independently. The selector emits at most one artifact per task, the first passing success in attempt order.
Compute the expected number of emitted artifacts, the cohort cost, and \(\CostPerEmittedArtifact\) (equation 5).
The team instead reviews a task’s successes in attempt order and stops at its first pass. Show that the set of emitting tasks is unchanged, and compute the expected reviews per task, the new cohort cost, and \(\CostPerEmittedArtifact\).
A second proposal halves \(\IndependentAttemptCount\) to 5 with the stopping review in place. Compute \(\CostPerEmittedArtifact\) and the emitted count, and decide between the two proposals, naming what the cheaper unit price costs the corpus.
The research agent’s collector (Intermediate). The research agent reads untrusted sources, so each collection record keeps the retrieved pages that its claims cite, and the permitted-use review of that retained content is expensive. A collector runs 32 accelerators at 40 accelerator-seconds per attempt, checks attempts on 16 verifier slots, and writes records of at most 1 GB to a store that sustains 500 MB/s (decimal units). The independent predicates take and pass as follows: record adequacy \(\text{chk}_{\text{record}}\), 0.02 s and 98 percent; permitted use \(\text{chk}_{\text{use}}\), 6 s and 60 percent; citation grounding \(\text{chk}_{\text{out}}\), 3 s and 70 percent; target review \(\text{chk}_{\text{target}}\), 2 s and 90 percent. No predicate depends on another’s result. Use the saturated planning model with sufficient end-to-end credits, no additional shared service bottleneck, and each write charged at its record ceiling; the calculated rate is not a measurement of actual collector throughput.
Order the predicates by equation 2 and compute \(\ExpectedCheckingTime\) and the yield \(\AdmissionRouteYield\).
Evaluate equation 4, name the binding stage in this model, and compute its planned admitted records per hour. Compute the write bandwidth that would let generation bind, and the admitted rate then.
The binding stage differs from the repair collector of notebook 15.6. Explain why, and evaluate a proposal to cut the record ceiling to 100 MB by truncating retrieved pages. Propose a design that lowers stored bytes without weakening \(\text{chk}_{\text{record}}\) or \(\text{chk}_{\text{use}}\).
Staging production feedback for the support agent (Advanced). The support agent serves customers who are waiting, so its collector captures production sessions it cannot delay or pause. Sessions finish at \(\ArrivalRate=50\) per second and last 120 s, with capture running throughout. A policy-grounding judge assesses each captured record on \(\WorkerCount=40\) slots in a mean of 0.6 s. Records have a ceiling of 250 KB, and staging holds \(\StagingStoreCapacity=4\) GB and at most 10,000 outstanding records (decimal units). During an incident the judge’s endpoint degrades and every assessment runs to its 10 s timeout. The controller needs 20 s to act, and staging starts nearly empty.
Compute the nominal utilization, the degraded capacity, and the staging growth rate. Compute the time until the byte limit and until the record-count limit is reached, and state which binds.
Size the high watermark on the binding limit, counting records from sessions already capturing and growth during the controller’s delay.
Backpressure on the source is unavailable here, unlike in the repair collector. Compute the capture fraction that holds staging steady during the incident, and critique a proposal to keep capturing only the shortest sessions. State what the corpus manifest must record about the uncaptured sessions.
Designing a leakage screen within a review budget (Advanced). A corpus of 100,000 records is screened with 128-component MinHash signatures before split assignment (section 15.4.3). Assume that 2,000 record pairs are true relatives with Jaccard similarity 0.8 and that every other pair has similarity 0.1. Reviewers can inspect 20,000 candidate pairs per screening pass.
For banding designs of 32 bands of 4 rows, 16 of 8, and 8 of 16, compute the expected false candidates among unrelated pairs and the recall of true relatives. Decide which designs fit the review budget.
Redesign the screen so that it keeps the recall of the 32-by-4 design and fits the budget, using the full signature. Justify the chosen threshold with the estimator’s standard error, and estimate the machine work it adds.
Name a relationship that no choice of bands will catch, and the mechanism that must catch it instead.
What’s Next: From corpus membership to learning targets
Translating an admitted recovery record into a gradient update requires reinforcing the correction rather than the refused proposal it conditions on. Trajectory Fine-Tuning opens with the record this chapter published to repair_demonstrations_v1, whose targets recovery_targets_v1 already fixed. It first asks which parts of the repair behavior belong in weights at all, then turns the curation contract into training examples. Serialization must reconstruct each prefix exactly as the agent received it, loss masking must assign direct supervised loss terms to the selected targets alone, and batching must keep independent examples from reading one another. Preference learning applies a different objective to justified comparisons. Curation decides which material is eligible and why, and the next chapter decides how that material shapes an update and whether the changed policy earns a release under the same runtime controls.
