Why does a configuration that clears its offline evaluation still need evidence gathered while it runs?
An offline evaluation picks its tasks, fixtures, and graders in advance, and live traffic honors none of those choices. Production brings inputs no fixture anticipated, dependencies that slow down under load, and tool results cut short without warning, while every admitted task acts with real authority. When the rate of accepted tasks drops, the count says that something failed but not where, because one trial spreads its model calls, tool executions, and checks across processes and hosts. Finding the cause takes a record of each trial, yet keeping every record costs too much and exposes private data, and sampling records at random discards exactly the rare failures engineers most need. A failure that recurs ten times under a one-percent sample leaves no detailed record about nine times in ten. Even a kept record shows only what happened. It can show that an observation was truncated and that the runtime refused an action, but whether a complete observation would have produced an accepted repair is a question only a controlled rerun from the saved state can answer. Releasing a change raises its own questions. A small canary does not make a regression cheaper to detect, because a monitor needs the same number of tasks at any traffic share; the canary limits only the harm from problems no monitor measures. Rolling back stops new work but does not recall requests already sent or restore files already changed. Operating an agent therefore means deciding in advance what to record, how to test an explanation, how much live work to expose, and what a release must show before it widens, and then writing down what is still unknown. Every exposure the agent carries reaches production at once: the state a failure leaves behind, the horizon over which harm builds before a monitor decides, and the authority whose effects outlast a rollback.
Learning Objectives
Explain why a diagnostic trace, the durable recovery log, and the accounting record carry different guarantees
Analyze a trace’s critical path to attribute elapsed time among overlapping spans
Design verdict-aware trace retention that keeps rare failures within storage and privacy budgets
Diagnose an agent failure through a controlled intervention, stating what the result does and does not establish
Design staged releases that size canary exposure with sequential stopping rules and independent hard tripwires
Evaluate an integrated evidence record against a declared conjunctive release gate
14.1 From Offline Verdict to Live Operation
The offline comparison of candidate_v2 against baseline_v1 ended without a decision in either direction. The candidate accepted more of the paired tasks, but its paired interval does not exclude a loss larger than the declared tolerance, and a powered follow-up sample is still to be planned. That verdict concerns a configuration on a sampled task population. It says nothing about why a particular trial failed, and nothing about what the configuration does once it acts on live work, where inputs are not fixtures, dependencies are shared, and every admitted task exercises real authority.
↩︎ The paired comparison that leaves candidate_v2 inconclusive against its noninferiority margin is established in Paired outcome comparison.
Operating a configuration carries every exposure at once: the state a failure leaves, the horizon before a monitor decides, and the authority a rollback cannot recall.
The repair agent (Four lighthouse workloads) supplies this chapter’s case. One trial of candidate_v2 on the connection-pool repair fails after its search tool returns a silently truncated result, and the next model response proposes rewriting a file outside the permitted repair scope, which the runtime refuses before dispatch. Aggregate counts record the failure; they cannot say whether the model, the observation adapter, a slow dependency, or the runtime is responsible, or whether repairing one of them would change the outcome. Answering those questions takes evidence the offline harness never collected, gathered while the system runs and kept long enough to inspect.
Four kinds of operational evidence fill that gap, and each has its own limits. A trace binds every model call, tool execution, and check to its trial, so that a delay or a failure can be attributed along the trajectory’s critical path (section 14.2). Retention policy decides which traces survive storage and privacy budgets, and the accounting record keeps the evaluation denominator complete when diagnostic detail is sampled (section 14.3). An incident postmortem turns an observed mismatch into a hypothesis and tests it by intervening on a saved prefix of the recorded trajectory, since no amount of rereading historical telemetry can answer a counterfactual (section 14.4). Staged release then exposes a revised configuration to live work only as fast as sequential evidence and hard tripwires allow, because a rollback stops new admission but settles nothing already done (section 14.5). The chapter closes by joining these records with the offline comparison into one decision record and a declared release rule (section 14.6), beginning with the trace that explains where the truncated-observation trial spent its time.
14.2 Distributed Trajectory Tracing
When an evaluation suite reveals that a candidate agent completes fewer tasks or suffers severe latency regressions, aggregate summary metrics rarely pinpoint the root cause. In the connection-pool repair, a candidate might stall during remote inference, wait on a blocked test container, or waste cycles parsing malformed tool outputs. Without fine-grained visibility, these delays blend into an undifferentiated wall-clock total. A trace transforms these scattered signals by binding every interaction back to the specific evaluation trial, candidate revision, and runtime operation, replacing disjointed service logs with an end-to-end causal record.
Distributed tracing tracks units of work across process and network boundaries, an operational model established at production scale by Dapper (Sigelman et al. 2010). At the core of this model, a span represents a single contiguous unit of execution defined by a unique identifier, explicit start and end timestamps, structured attributes, and optional timed events. Hierarchical parent-child relationships capture synchronous nesting, while trace links correlate decoupled asynchronous interactions that fall outside a single call tree, following the standard OpenTelemetry model (OpenTelemetry Authors 2026). Within an agentic trajectory, a large language model (LLM) generation step, a mediated tool execution in a sandbox, and subsequent result validation each form distinct spans. Omitting instrumentation at any of these boundaries leaves unmonitored voids, obscuring whether an unexpected delay originated in model inference, queueing overhead, or environment setup.
During the connection-pool repair, every model invocation and sandbox tool call emits both a durable log record and a diagnostic trace span, often carrying the same operation identifier. Treating those two streams as interchangeable is a dangerous operational shortcut: if an execution engine relies on its diagnostic tracing pipeline to reconstruct state after a worker crash, a single load-shedding event that drops spans or truncates large tool payloads will corrupt recovery. Separating the recovery path from the observability path ensures that telemetry backpressure never compromises execution durability.
Systems Perspective 14.1: Recovery history and diagnostic traces
The durable trajectory log preserves the authoritative, acknowledged state transitions and input artifacts required to reconstruct execution state after a failure. A diagnostic telemetry stream that happens to share identical span identifiers cannot replace this log. Diagnostic traces prioritize performance analysis, bottleneck detection, and interactive debugging; under high load or network partitions, tracing pipelines legitimately sample traces, truncate large payloads, or drop spans. The same runtime event can emit records to both pipelines, but their guarantees differ completely. An observability agent might discard an optional span to protect telemetry egress, while the durable recovery engine must guarantee that operation identifiers, resource reservations, and partial results persist to durable storage.
↩︎ The durable trajectory log, from which recovery reconstructs execution state, is established in The log as recorded evidence.
Systems engineers must maintain this distinction when tracking asynchronous operations such as the test run on ws_r18 that recovery left unresolved before evaluation began. When an HTTP client monitoring that sandbox times out and terminates its diagnostic span, the underlying test does not vanish; instead, the workload continues running inside the container until the sandbox emits an authenticated callback with the test results, which the operation table records against the original operation while the trajectory stays paused. To reflect that split execution without rerunning the test or resuming the model prematurely, the distributed trace correlates the late-arriving callback with the original dispatch attempt through a trace link, preserving both the client-side timeout and the true server-side execution lifetime.
Captured content demands an equally strict boundary. An evaluation harness must distinguish between what an agent proposed, what intermediate middleware modified, and what the model actually received. The delivered-input manifest records the exact byte sequence presented to the model context window, which frequently diverges from raw tool output due to prompt templating, token budget truncation, or context compression. Cryptographic hashes verify payload integrity against historical records, but a hash digest cannot reconstruct payload bytes that were omitted or pruned from storage. Exact deterministic replay and forensic debugging therefore require preserving delivered-input manifests and raw observations in content-addressed storage alongside the trace metadata.
14.2.2 Trace evidence fields
When services emit unstandardized telemetry tags—or mark a span successful simply because a test process exited with code 0 despite failing assertions—engineers cannot tell which candidate revision a log line belongs to or whether a green span actually passed its acceptance checks. An evaluation harness instead requires a disciplined, minimal schema that links experimental parameters directly to runtime and network observations. Table 1 organizes the essential fields into five functional categories. The attribute names may follow a standard such as the OpenTelemetry semantic conventions; the five groups are the minimal information required to diagnose agent behavior.
Table 1: Minimal trace evidence groups: Keep identities, observed facts, runtime decisions, and capture limits distinguishable. A completed span is not a completed task.
Record group
Information to preserve
Question it answers
Evaluation identity
Task identity, trial identity, fixture and criterion versions, system configuration, candidate artifact revision
Which experiment and candidate does this observation concern?
Observed work
Trace and span identities, operation kind, producer, start/end boundary, clock domain, parent or dependency links
Which work was observed, where, and in relation to what other work?
Admission and attempt
Model invocation or logical operation identity, transport attempt, applicable decision and policy references
Was this proposal withheld or admitted, and which attempt produced the evidence?
Outcome and usage
Response status and source, evidence scope, observed token/tool usage, unresolved liability, task-verdict reference
What is known about the operation, its costs, and the separate acceptance decision?
Capture limits
Delivered-input and observation references, omitted ranges or bytes, instrumentation version, known drops, retention/access classification
What can an investigator inspect, and what does this record leave unavailable?
This schema carries the task, trial, configuration, and artifact distinctions of the acceptance record into each observed interaction, while constraining provider provenance to what the remote interface exposes. A fundamental rule of evaluation tracing is that span status must remain decoupled from the final task verdict. An operating system process running a test suite may exit with code 0, prompting the tracing layer to mark the span successful, even though the test output reveals that the candidate failed critical assertions. Conversely, a client span might terminate with a timeout error while fetching auxiliary documentation, yet the candidate agent may still synthesize a valid patch from cached context. The authoritative task acceptance decision relies on independent verification suites and often arrives long after the agent’s execution spans have closed.
Propagating context across architectural boundaries such as an API gateway, an asynchronous queue, or an isolated worker sandbox demands standardized metadata carriers. The W3C Trace Context specification (W3C Distributed Tracing Working Group 2021) provides standard HTTP and messaging headers to propagate correlation identifiers across distributed components. Trace identifiers, however, establish correlation, not authorization. A worker sandbox receiving a trace identifier cannot treat it as an admission token, nor can an agent bypass security policies by injecting arbitrary parent headers. Admission control, credential validation, and payload sanitization remain the exclusive responsibility of the runtime host. Furthermore, trace headers must never carry sensitive prompt tokens, credentials, or proprietary task data across untrusted network boundaries. When black-box tools or legacy binaries cannot emit internal spans, the orchestrator must record host-observed boundaries, explicitly tagging the span as an external dispatch-to-response measurement rather than an internal execution trace.
W3C Distributed Tracing Working Group. 2021. Trace Context: W3C Recommendation 23 November 2021. World Wide Web Consortium.
14.2.3 Span overlap and latency attribution
Summing span durations to obtain total agent execution time is a common error in telemetry analysis. Because a parent span encloses the lifetimes of its children, adding parent and child durations double-counts work. Furthermore, when an agent orchestrator fires multiple tool calls concurrently, for example a static analysis linter alongside a targeted test suite, their child spans overlap on the timeline. Execution order stems strictly from completion dependencies, never from parent-child hierarchy alone. Comparing durations measured on a single monotonic clock is straightforward, but correlating timestamps across distributed worker hosts requires an explicit alignment model.1 Without synchronized time sources or clock-offset modeling, apparent span overlap between distinct machines can reflect nothing more than physical clock skew. Instrumentation boundaries introduce similar attribution challenges: client-observed model latency encompasses TCP connection handshakes, TLS negotiation, queueing delay at the inference gateway, and network transfer time, meaning that time to first token is never an isolated measurement of model prefill compute.
1Happened-Before Order in a Trace: The partial order that Harness regression replay used to match replayed requests applies to spans as well. Parent and link relations recorded at dispatch carry causal order between spans on different hosts; cross-host timestamps without an alignment model do not, so two spans that appear to overlap may be ordered, and two that appear ordered may be concurrent.
2Critical Path Method: Critical-path planning was formalized in 1959 for scheduling large engineering projects, computing for each activity the float by which it can slip without delaying completion (Kelley and Walker 1959). The notebook measures a related quantity, the duration gap between the critical branch and the next-longest parallel path.
Kelley, James E., and Morgan R. Walker. 1959. “Critical-Path Planning and Scheduling.”Papers Presented at the December 1-3, 1959, Eastern Joint IRE-AIEE-ACM Computer Conference, 160–73. https://doi.org/10.1145/1460299.1460318.
To quantify where latency optimizations yield true wall-clock savings, systems engineers measure the critical path through the trajectory’s dependency graph, the longest chain of dependent operations recorded in the trace.2 Sequential operations add their durations directly, whereas concurrent branches that join at a common synchronization barrier contribute only the duration of the slowest branch. Shortening a branch off the critical path leaves elapsed time unchanged, and shortening one on it helps only until the next-longest branch takes its place (notebook 14.1).
↪ An acceleration is priced by the time it removes from this critical path, a trade-off developed in Critical-Path Latency.
Napkin Math 14.1: Observed time and the critical path
Problem: How much elapsed time do the recorded spans explain, and how much would shortening one check reduce the segment if the other durations and dependencies stayed unchanged?
Assumptions: Table 2 gives observations from a separate hypothetical timing trial that inspects ws_r19. The model first proposes checks, the runtime admits two independent read-only checks, and synthesis waits for both. All intervals use one host clock, no hidden dependency delays either check’s start, and admission takes no separate time. The enclosing segment runs from retrieval dispatch through the final response.
Table 2: Hypothetical observed span intervals: The two checks overlap. The enclosing segment measures 11.7 s; its duration is not additional work to sum with these children.
Observed span
Interval after segment start (s)
Required predecessor
Retrieve supporting context
[0, 0.2]
Segment start
Model call proposing checks
[0.2, 3.2]
Retrieval response
Targeted tests
[3.2, 9.2]
Validated and admitted check proposal
Static lint check
[3.2, 5.2]
Same proposal; independent of targeted tests
Model call synthesizing the report
[9.7, 11.7]
Both check responses
Math: Adding all leaf durations yields 13.2 s, an overestimate that double-counts concurrent execution during the validation phase. Following the path through retrieval, planning, the longer check (targeted tests), and synthesis, which counts the overlapping checks once, explains 11.2 s. The remaining 0.5 s occurs between the completion of the slowest check and the dispatch of the final synthesis call. Because the trace lacks instrumentation for intermediate runtime bookkeeping, prompt serialization, or context collation, this duration remains unattributed; an unmeasured interval highlights a boundary where instrumentation must be refined rather than evidence of worker idle time.
Intervention: Suppose an optimization accelerates the targeted test suite from 6.0 s down to 1 s while all other components remain unchanged. Because the static lint check requires 2.0 s, lint replaces the test suite as the bottleneck on the parallel branch. Recomputing the critical path yields 7.7 s for the total segment, achieving an overall elapsed reduction of 4 s.
Systems insight: A performance optimization on a parallel execution branch yields wall-clock returns only up to the duration gap to the next-longest concurrent path. Accelerating the test suite by 5 s yields only 4 s of end-to-end savings because static linting immediately becomes the new critical-path limiter. Systems architects must recompute the critical path across the entire dependency graph before estimating the value of an optimization. Furthermore, theoretical speedups must always be verified empirically in controlled trials; real-world optimizations can shift resource contention, alter cache locality, change garbage collection pressure, or elicit different token generation lengths from the model.
Checkpoint 14.1: Interpret the trace's evidence
Distributed trajectory tracing provides the causal fabric needed to diagnose why an agent succeeded, failed, or stalled. However, capturing detailed telemetry incurs concrete engineering costs in CPU overhead, network bandwidth, and persistent storage. In high-throughput evaluation harnesses, preserving every token, tool payload, and internal span across thousands of benchmark trials is rarely feasible. Defining what to measure is only the first half of the observability problem; the second half is establishing capture and retention policies that preserve critical failure evidence without overwhelming the telemetry infrastructure.
14.3 Trace Retention and Privacy
If a delayed regression suite for the trial that produced ws_r19 fails hours after execution concludes, that trial suddenly becomes diagnostically indispensable. By that time, however, its runtime process has terminated and its model and tool spans have closed. The forensic value of the run depends entirely on what the telemetry system chose to keep. Trace retention therefore requires integration with final task verdicts and their arrival timelines, rather than relying purely on process exit codes or span completion markers. Because retaining exhaustive execution histories rapidly escalates storage expenditures and expands sensitive data exposure, systems designers must determine which evidence each evaluation goal demands, and which diagnostic guarantees remain possible when capture budgets are constrained.
14.3.1 Accounting records and head sampling
If a telemetry pipeline samples trial registrations at the same rate as diagnostic span bodies, any trial that times out, crashes a worker, or drops its spans disappears from the evaluation denominator, biasing pass rates toward fast, easy runs. Beyond the recovery history and diagnostic traces separated in perspective 14.1, evaluation requires a third artifact: the accounting record. This record catalogs every admitted trial, its exact environment configuration, evaluation criteria, current verdict status, measured resource expenditures, and missing telemetry markers. While these three records can reside on shared physical storage, sampling policies must apply exclusively to diagnostic detail. Telemetry systems must never sample recovery history or the evaluation denominator.
Diagnostic detail also expands substantially faster than the underlying trajectory when spans mirror raw message history. If each model-call span replicates the entire conversation context sent to the provider, diagnostic trace volume inherits the quadratic resend pattern formalized in equation 2.9. Telemetry systems avoid this runaway expansion by referencing immutable conversation prefixes via cryptographic content digests and storing each new tool observation exactly once. This content-addressable deduplication keeps storage growth linear in the number of turns, provided the underlying blob store retains the referenced content for the lifetime of the trace.
Head sampling decides whether to retain detailed diagnostic traces at the initial entry point of execution. Drawing a random sample with recorded inclusion probabilities allows engineers to compute unbiased population estimates for metrics such as mean latency or aggregate token consumption. However, uniform head sampling struggles with rare failure modes, as an intermittent defect can easily slip through without leaving a single retained trace.
Because head sampling decides retention before execution outcomes are known, each occurrence of a failure mode is retained independently with sampling probability \(\HeadSamplingProbability\). Consequently, a failure mode that occurs \(\FailureModeOccurrenceCount\) times leaves no detailed diagnostic trace with probability \((1-\HeadSamplingProbability)^{\FailureModeOccurrenceCount}\). At \(\HeadSamplingProbability =\) 10 percent, a failure mode that occurs 5 times carries roughly a 59 percent probability of leaving zero detailed examples in storage; only when that mode recurs 50 times does the probability of a complete miss fall to 0.5 percent. By its mathematical nature, uniform sampling is least likely to preserve the rare, anomalous executions that engineers most urgently need to inspect.
To capture these elusive events without retaining every routine execution, tail sampling defers the retention decision until operational evidence accumulates downstream, such as a failed acceptance assertion, an unhandled tool exception, an unexpected human intervention, or an anomalous latency spike. Tail selection deliberately concentrates diagnostic storage on high-value, problematic runs, producing a trace collection that is intentionally unrepresentative of normal production traffic. Engineers must therefore compute population failure rates from the complete accounting record rather than from the tail-sampled trace collection, and whenever teams derive statistical estimates from sampled records, the telemetry infrastructure must record the exact inclusion probabilities alongside the active selection policy version.3 Diagnostic workflows also require a baseline for comparison: the pipeline must retain a known random sample of ordinary, accepted trials as a control group, because an isolated failure trace cannot reveal which intermediate steps were anomalous and which were routine.
3Uniform Trace Sampling: The first production version of one large-scale tracing system sampled uniformly, averaging one traced request per 1,024 candidates, which sufficed for high-traffic services because events of interest recurred often enough to be captured; lower-traffic workloads could miss important events at that rate (Sigelman et al. 2010). Its later adaptive scheme targeted a rate of sampled traces per unit time and recorded the sampling probability with each trace, the inclusion-probability bookkeeping that population estimates require.
Sigelman, Benjamin H, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, and Chandan Shanbhag. 2010. Dapper, a Large-Scale Distributed Systems Tracing Infrastructure. Dapper-2010-1. Google.
14.3.2 Tail sampling and verdict windows
If a tail-sampling buffer evicts trace bodies before a slow regression suite finishes grading ws_r19, the accounting record logs the late failure while the spans needed to explain it are already gone. Because tail selection spans multiple operational stages, an execution trace is held provisionally in a staging buffer while acceptance checks run, receiving its final retention classification only when conclusive evaluation evidence arrives within a declared verdict window. When the staging buffer expires before an asynchronous failure is reported, the accounting record must explicitly flag the missing diagnostic detail so audit integrity holds. Table 3 outlines this tiered retention policy, establishing a strict precedence order so that overlapping classification rules do not count or persist a single trace multiple times.
Table 3: Verdict-aware trace selection: Selection uses declared task criteria and available evidence, including delayed evaluation outcomes. Successful process completion does not imply task acceptance.
Evidence at the decision boundary
Possible detailed-trace policy
Limit to record
Failed task criterion or enforced-policy violation
Give high retention priority, with the criterion and supporting evidence
A process error does not inherently prove whole-task failure, and buffer saturation may still force eviction
Pending or unknown task verdict
Retain provisionally or under an explicit unresolved-case sample
State the deadline, missing evidence, and what happens when the window expires
Accepted but unusually slow, costly, or intervention-heavy trial
Retain under declared diagnostic thresholds
Selection enriches those behaviors and does not establish their population frequency
Ordinary accepted trial
Retain a known random control sample
Preserve its inclusion probability and distinguish unsampled from lost
Eviction priority: When ring buffers experience memory pressure, eviction policies must shed routine successes and unclassified traces before dropping runs flagged for policy violations or failed assertions. While differential eviction priority substantially improves diagnostic survival rates, extreme burst traffic can still cause trace loss.
Physical buffer limits inevitably govern this architecture. Buffer managers must enforce four concurrent bounds: active trace count, aggregate byte volume, individual trace size limits, and maximum staging age. Sizing from averages, such as mean concurrent traces multiplied by mean trace size, is a frequent engineering trap, because cascading failures typically produce bursts of unusually long, tool-heavy trajectories that quickly exhaust buffer memory. Telemetry systems must expose real-time metrics for head-dropped records, buffer evictions, and span truncations broken down by cohort and selection class. Without this visibility, correlated loss during incidents will silently purge the exact failure traces engineers need to diagnose system outages. Furthermore, while deferred tail selection curtails long-term downstream storage, it does not reduce the network ingress and memory needed to buffer traces prior to selection, and strict privacy restrictions apply to these staging buffers as well. The accounting record therefore sits outside the buffer (figure 1). Every admitted trial enters it directly, together with every recorded drop, eviction, and truncation, so the denominator survives whatever the buffer loses.
Figure 1: Verdict-Aware Trace Selection: Trace bodies pass a fallible minimization boundary into a bounded buffer that holds them until task verdicts, which can arrive late, reach the selector. Selection applies a declared precedence among the classes of table 3, keeps inclusion probabilities with retained traces, and discards unselected bodies. Every admitted trial also enters the accounting record directly, together with recorded drops, evictions, and truncation, so the denominator does not depend on which bodies survived. Assigning high retention priority protects failure traces under moderate load, though severe buffer saturation can still force evictions before verdicts arrive.
Napkin Math 14.2: What verdict-aware selection saves
Problem: How much downstream diagnostic storage do uniform head sampling and verdict-aware tail selection use for the same hypothetical workload?
Assumptions: Each hour brings 10,000 trials with equal 1.4 MB permitted trace bodies. At the selection boundary, 400 have failed criteria, 100 remain pending or unknown, and 9,500 are ordinary accepted trials. These classes are disjoint, repeat for 24 hours, and all survive collection. Head selection samples each trial with probability 5 percent. Tail selection retains the failed and unresolved classes and independently samples 2 percent of ordinary accepted trials.
Math: Keeping every body requires 336 GB over the stated period. Head sampling retains an expected 500 traces per hour, or 16.8 GB over the stated period. It includes an expected 20 failed and 5 unresolved trials per hour, not guaranteed counts. Tail selection retains 400 failed, 100 unresolved, and an expected 190 ordinary accepted trials: 690 traces per hour and 23.184 GB over the stated period.
Result: Full retention uses about 14.5× the downstream bytes of the tail policy. Tail selection consumes more capacity than uniform head sampling in this scenario, but it guarantees diagnostic visibility for every failed and unresolved execution rather than retaining only an unpredictable handful of anomalous runs.
Systems insight: Retained volume is governed by two terms: an inelastic storage floor determined by the failure-plus-unresolved rate (16.8 GB per day in this workload) and an adjustable overhead determined by the control sampling fraction on routine successes, where each percentage point of control sampling adds 3.2 GB per day. Capacity planning must size the baseline floor to accommodate elevated failure rates during service disruptions while dialing the control rate to fit available long-term storage budgets. These calculations reflect decimal gigabytes for raw trace payloads; indexing metadata, replication factors, accounting records, and upstream ingestion buffers impose additional overhead.
14.3.3 Disclosure minimization
Dumping raw model prompts and unredacted tool responses directly into a shared tracing backend turns the observability pipeline into a secondary data leak, spreading customer records, database keys, and proprietary source snippets across search indexes and crash logs where retroactive scrubbing cannot reach them. Diagnostic investigations rarely require wholesale ingestion of raw prompts and complete tool outputs; structured metadata (typed return codes, schema versions, sub-span latencies, policy decisions, and hashed artifact pointers) often provides complete debugging context without exposing sensitive user data. Even when credentials and API tokens are cleanly scrubbed, tool arguments and responses frequently contain private records or personal identifiers, so authorization policies must govern specific data attributes and restrict access by investigator role rather than granting blanket read access to an entire telemetry tier. Data minimization belongs at the trusted capture boundary inside the execution runtime, before unscrubbed buffers enter network queues, ingestion daemons, or alerting systems.
Automated redaction mitigates disclosure risks, but systems designers must treat every scrubbing mechanism as inherently fallible. Regular expressions miss non-standard or obfuscated secret patterns; Shannon entropy detectors flag UUIDs and cryptographic nonces while missing predictable credentials; and named-entity recognition models err, skipping private names or stripping valid operational syntax required to diagnose a tool error. Because heuristic sanitizers cannot guarantee zero disclosure across arbitrary payloads, architectural schema design must prioritize structural field exclusion over text redaction, so that unnecessary fields are never captured in the first place. For retained fields that undergo transformation, pipelines must validate sanitization rules against curated benchmark suites containing real-world sensitive patterns. When downstream diagnosis requires correlating multiple actions by the same entity without revealing real-world identities, telemetry systems should replace identifiers with keyed cryptographic pseudonyms. Generating these pseudonyms via a hash-based message authentication code (HMAC) requires a securely managed secret key rather than a public salt (Krawczyk et al. 1997, sec. 2). However, engineers must recognize that persistent pseudonyms preserve relational linkability across traces. Regular key rotation limits longitudinal correlation across evaluation epochs, at the cost of diagnoses that must follow one entity across several weeks. Pseudonymization is not complete anonymization, nor does it justify indefinite trace retention.
Krawczyk, Hugo, Mihir Bellare, and Ran Canetti. 1997. HMAC: Keyed-Hashing for Message Authentication. RFC 2104. Internet Engineering Task Force. https://doi.org/10.17487/RFC2104.
14.3.4 Replay fidelity after redaction
Sanitization creates an inherent tension between privacy enforcement and replay fidelity. Scrubbing or masking input tokens alters the precise byte sequences that drove agent behavior. Stripping a database column from a tool response may conceal the exact formatting defect that triggered a model’s parsing error, while replacing an address string can shift sub-word token boundaries, steering the model down an entirely different decoding path during re-execution. Consequently, a redacted trace cannot serve as a bit-exact substitute for the environment interactions that actually occurred. When forensic investigations require exact replay and compliance policies permit it, systems must isolate raw execution payloads in a dedicated, access-restricted artifact repository, completely separate from general diagnostic telemetry. The diagnostic trace then stores only cryptographic hashes pointing to these secured blobs, alongside metadata recording the exact transformation pipeline used to generate the sanitized view. When raw payloads expire under data-retention schedules or cannot be kept due to privacy constraints, teams must formally narrow the scope of their diagnostic claims: replaying an agent against synthetic mock fixtures validates whether the model handles a class of problems in a fresh run, but it does not prove what occurred during the original production execution.
Checkpoint 14.2: Retained evidence and supported conclusions
Storage budgets determine which runtime observations survive, while privacy policies govern which preserved artifacts an investigator may legally inspect. Every postmortem analysis and regression benchmark operates within the boundary those two constraints set.
14.4 Incident Postmortems
Return to the trial of candidate_v2 from the chapter’s opening, which fails on the connection-pool repair after receiving a silently truncated search result. The retained response ends in the middle of a source excerpt with no truncation notice, although the bounded-observation contract requires one, and the next model response proposes replacing a file outside the permitted repair scope. The runtime rejects that proposal before dispatch. While the execution trace pinpoints where the observation was cut short and where the runtime blocked the illegal operation, it cannot reveal whether a complete observation would have produced an accepted repair. An incident postmortem4 connects such telemetry to a testable explanation and a scoped corrective change.
4Blameless Postmortem: Site reliability engineering conducts blameless postmortems to identify contributing causes without indicting any individual or team (Beyer et al. 2016, ch. 15). Agent incidents extend this stance from people to components: a model, an observation adapter, and an admission gate can each contribute, and naming one culprit early hides the conditions the corrective change must address.
Beyer, Betsy, Chris Jones, Jennifer Petoff, and Niall Richard Murphy. 2016. Site Reliability Engineering: How Google Runs Production Systems. O’Reilly Media.
↩︎ The bounded-observation contract, under which an adapter that cuts a result must say so in a typed truncation notice, is established in Truncation notices and inspection handles.
Rigorous postmortems keep these initial findings strictly separated: delivering an excerpt without truncation metadata demonstrates an interface defect, whereas emitting an out-of-scope file replacement reveals how the foundation model behaved when starved of context, and intercepting that illegal write confirms that the admission gate functioned as designed. Because failures in agentic systems typically stem from compounding interactions across subsystems, prematurely pinning the fault on a single component obscures the precise conditions that engineering fixes must resolve.
14.4.1 Reconstruction and replay
Rereading a static trace cannot show whether candidate_v2 would have repaired the connection pool given an untruncated search result, yet re-running a modified agent against recorded tool outputs without request matching can pair new model calls with stale environment replies. The three uses of a recorded trajectory from durable execution become the postmortem’s three instruments, each answering a distinct diagnostic question:
Time-travel replay folds recorded decisions from a compatible execution checkpoint to recover the exact system state visible to the agent at the moment of failure.
Harness regression replay executes deterministic component code against captured inputs while disabling live external side effects.
A counterfactual branch forks execution from a saved prefix, altering a specific variable or interface contract and collecting fresh observations along the divergent path.
↩︎ The three uses of a recorded trajectory, and the limits of what each one shows, are established in Recorded Trajectory Replay.
Time-travel replay only inspects the past, and no replay mode can synthesize an unrecorded historical outcome. Demonstrating that a revised routine now passes its tests does not change what the original trial delivered, observed, or did. Furthermore, the fidelity of time-travel replay remains strictly bounded by the capture contract defined in perspective 14.1. Examining the delivered context reveals the truncated excerpt presented to the model; reviewing separately preserved raw stdout reveals what the adapter dropped; and inspecting redacted logs reveals only execution metadata and high-level status.
Harness regression replay is especially effective for isolating data transformations without incurring expensive model invocations. Supplying captured tool responses to legacy and revised observation adapters, with live effect paths disabled, deterministically verifies that the legacy adapter omitted a required partial-result marker while the revised adapter correctly supplies it. However, this offline harness test cannot predict how a model will respond to the revised observation envelope. Answering that question requires a live experiment. Any alteration to prompt text, sampling temperature, model configuration, or tool observation formats represents an intervention rather than a passive replay. Recorded downstream responses remain reusable only under the matching-request rule of Hermetic Evaluation Gyms; matching solely on tool names risks fabricating artificial successes by pairing new model actions with obsolete environment responses.
14.4.2 Fault attribution
Blaming the model for proposing an out-of-scope file rewrite—or trusting the model’s own retrospective explanation of why it failed—confuses a downstream symptom with the upstream truncation defect that triggered it. In the connection-pool incident, comparing the search tool’s raw output against the delivered envelope confirms that the observation adapter silently truncated the text, yielding an engineering hypothesis: omitting the completeness signal misled the model into generating inappropriate repair proposals. Validating this hypothesis demands empirical outcomes measured after an intervention rather than repeated scrutiny of historical telemetry. Table 4 distinguishes useful evidence from claims that telemetry cannot support alone; its rows represent overlapping investigation paths rather than mutually exclusive diagnoses.
Table 4: Evidence for diagnostic hypotheses: Localization constrains an explanation without making the subsystem categories exclusive.
Suspected boundary
Evidence or controlled comparison
Limit of the finding
Context assembly
Compare retained source versions, delivered messages, and omission metadata
An omitted fact is not proof that restoring it will complete the task
Tool observation
Compare captured output with the adapter’s declared and delivered envelopes
A truncation defect does not make the model’s later inference correct
Model response
Repeat the specified invocation or continuation under controlled conditions
A different response does not identify temperature or sampling as the original cause
Runtime admission
Exercise captured proposals and control inputs against the stated guards
A rejected proposal differs from an unauthorized operation that was dispatched
Execution environment
Compare endpoint evidence, configuration, and relevant fault records
A timeout or process termination signal alone does not identify its underlying cause
System diagnostics must account for this ambiguity at every layer, because neither low-level system signals nor high-level model outputs are self-explanatory. An operating-system signal that terminates a container cannot distinguish an out-of-memory eviction from an administrative kill, just as a schema-valid model response can still rest on unverified assumptions and an agent’s retrospective self-explanation remains a generative hypothesis rather than empirical evidence.
14.4.3 Controlled diagnostic experiments
If an engineer ships an omission marker and pagination together while external service latency drifts between runs, a higher pass rate cannot distinguish whether the marker, the pagination, or the quieter infrastructure fixed the failure. To move from suspicion to actionable diagnosis, a controlled ablation protocol defines five elements before execution: the baseline checkpoint boundary, the specific architectural intervention, all frozen system configurations, allowable sources of stochastic variation, and uniform acceptance criteria. Resetting repository files while allowing external service behaviors to drift introduces confounding variables; interleaving experimental trials across branches helps decouple the intervention effect from infrastructure drift. Disentangling bundled changes such as a partial-result marker and pagination reads requires factorial conditions that vary each mechanism independently; the NIST discussion of one-factor experiments demonstrates why varying a single factor at a time fails to expose cross-component interactions (Guthrie 2020, sec. 5.2.1.2).
Guthrie, William F. 2020. NIST/SEMATECH e-Handbook of Statistical Methods (NIST Handbook 151). National Institute of Standards and Technology. https://doi.org/10.18434/M32189.
Example 14.1: A controlled test of observation completeness
Problem: A diagnostic fixture reproduces a silently truncated search result during the connection-pool repair. What do the following hypothetical outcomes support about a proposed interface change?
Scenario: The experiment restores a private copy of a compatible saved prefix of pool_repair_01 for every continuation, under three branches: the original interface, an added warning, and a revised paginated interface. Each run maintains its own execution lineage and identity, meaning these continuation trials offer evidence about counterfactual behavior without altering the recorded history of the trials that produced ws_r18 or ws_r19. All conditions share the model configuration, sampling settings, starting candidate, permitted operations, budget, and checks, and each receives 20 fresh stochastic continuations. Acceptance uses the repair’s criteria and rejects any run that proposes the out-of-scope rewrite, which the unchanged admission gate blocks before dispatch, and every run reaches a known assessment.
The original-interface condition deliberately retains the adapter defect, delivering the truncated excerpt without an omission notice, and yields 1 accepted repair alongside 19 rejected runs that propose the forbidden rewrite. By contrast, a warning-only condition adds an instruction that search results may be incomplete without providing a continuation mechanism, producing 8 accepted repairs and 12 budget-limited runs that never reach an accepted repair, though none proposes the forbidden rewrite. Finally, the revised-interface condition supplies typed partial-result metadata together with a supported way to retrieve remaining results, achieving 20 accepted repairs with zero nonaccepted runs and no forbidden rewrite proposals; across all three conditions, the admission gate prevents any unauthorized write from dispatching.
Diagnosis: Suppressing an illegal write proposal is not the same as successfully repairing the connection pool. While adding a static warning discourages the model from proposing out-of-scope file modifications, several continuations exhaust their execution budgets without reaching an acceptable fix. Conversely, the paginated interface achieves a complete success rate across these trials, justifying broader evaluation of that interface package. However, this diagnostic outcome does not prove that the metadata marker alone drove the turnaround, nor does it imply that model behavior was blameless in the original incident.
These stochastic continuations characterize performance strictly conditional on this specific saved prefix and fixture environment. Even achieving 20 accepted runs out of 20 trials still leaves an upper bound of 16 percent on the failure rate under the Wilson score interval of Confidence intervals for acceptance. Furthermore, these diagnostic ablation runs must remain segregated from benchmark task-set evaluations; pooling them into release metrics would artificially inflate sample size and distort generalization estimates.
Systems insight: The observation adapter has an explicit, reproducible defect, and the counterfactual trials demonstrate that repairing it substantially improves continuation outcomes. When designing diagnostic experiments, engineers should preserve admission controls across all conditions, distinguish between safety enforcement (blocking forbidden actions) and functional utility (completing the task), and validate patches against diverse held-out tasks before declaring release readiness.
Because the revised interface restores the bounded-observation contract that the original adapter violated, pagination mechanisms must maintain snapshot consistency against an immutable source revision. Exposing a continuation token for content that the server cannot subsequently serve would substitute one observation defect for another.
14.4.4 Scoped regression tests
Testing an observation-adapter fix only through full end-to-end agent runs wastes model tokens on a deterministic serialization rule, whereas running only the adapter unit test never verifies that the repaired agent actually completes the connection-pool fix. A corrective patch therefore requires verification at two distinct architectural levels: the localized boundary where the defect occurred and the end-to-end task pipeline whose reliability it aims to restore.
At the boundary level, unit tests must remain entirely deterministic and decoupled from stochastic model calls. Observation-adapter test suites should systematically inject complete, truncated, malformed, and empty payloads, verifying that serialized envelopes correctly populate status fields and omission metadata. In parallel, admission-gate suites must directly submit synthesized out-of-scope file modifications to confirm that the policy engine blocks illegal writes prior to dispatch. Neither verification step requires invoking an LLM.
At the system level, regression tests execute the modified agent within an isolated fixture and evaluate the resulting candidate against the full acceptance suite. Engineers should never treat a bare runtime status of COMPLETED as an indicator of correctness, nor should they mistake a single passing test run for a guarantee of operational stability. Table 5 organizes the essential artifacts of an investigation, separating confirmed defects from ongoing statistical uncertainty.
Table 5: A bounded postmortem record: A rigorous incident record documents the empirical evidence supporting a fix alongside unresolved engineering uncertainties, avoiding oversimplified single-cause attributions.
Report element
Evidence to preserve
Incident and coverage
Task, trial, configuration, candidate, relevant operation identities, retained interval, and missing or redacted material
Established findings
Delivered partial observation without required metadata; exact forbidden proposal and recorded pre-dispatch refusal
Tested change
Interface revision, compatible fixture and saved boundary, branch identities, unchanged guards, and allowed randomness
Outcomes
Common acceptance and proposal/effect criteria, every run’s verdict, budget stops, and unavailable assessments
Corrective verification
Adapter and admission tests, repeated task-level checks, and the actual revision tested
Remaining questions
Untested fixtures and interactions, retention limits, residual-operation owners, and evidence needed for release
Checkpoint 14.3: Diagnosis and regression scope
Use the repair experiment to distinguish a localized defect from a release claim.
An incident postmortem fulfills its engineering objective when it documents an established interface defect, provides a reproducible regression fixture, and verifies a targeted fix. However, resolving a single localized failure does not guarantee general reliability across diverse production workloads. Staged release evaluation determines which additional tasks, edge cases, and runtime conditions the candidate system must navigate while teams gather the next tier of empirical evidence.
14.5 Staged Releases
Promoting a revised agent configuration straight from a passing fixture test to full production traffic exposes live repositories and external services to unmeasured failure modes—and when an agent regression triggers a rollback, stopping new traffic does not undo tool calls or cloud allocations already in flight. Although the revised observation interface produced accepted repairs on the isolated fixture, that result supports only testing the change on a broader task set; production authority is a separate grant. A release decision connects the change to its evidence and to the consequences of being wrong, which for an agent include operations and charges that remain outstanding after new traffic stops.
Staging separates evidence available before live execution from evidence that requires it: whereas offline trials compare configurations on declared fixtures, shadow execution tests a candidate against authorized copies of current inputs without assigning it the live task, after which a canary5 assigns a bounded portion of eligible live tasks to the candidate under an explicit release policy, the ramped online controlled experiment of web services applied to agent configurations (Kohavi et al. 2009). Each stage answers its own question, and causal and statistical release evidence (principle 12) supports a scoped decision at each step. Failure at any stage routes to the same enforced admission hold (figure 2), which stops new candidate work while recovery settles the work already in flight; the criterion each stage applies is developed in the subsections that follow.
5Canary Release: The name borrows from coal miners, who carried canaries underground as an early warning of toxic gas. The analogy holds in one respect and misleads in another: a canary detects a hazard by meeting it first, so a small allocation warns early but does not protect the tasks it exposes.
Figure 2: Staged Release and Its Hold Path: Each stage applies its own criterion: the lower bound of the paired acceptance interval against the declared margin offline, a divergence measure of tool-call drift in shadow execution, and a sequential likelihood-ratio monitor with hard tripwires in the canary. The symbols in the stage boxes are defined in the subsections that follow. A wider release requires every criterion of the declared release gate (equation 5). An unmet or inconclusive offline result, a drift review that finds a problem, or a canary alarm leads to the same hold. The hold stops new candidate admission; outstanding candidate operations keep their identities and owners, and recovery, not rollback, settles their effects.
14.5.1 Offline comparison and shadow evidence
Replaying live user requests through a candidate in shadow mode before it clears its paired offline comparison—or mirroring inputs without isolating side effects and checking recipient permissions—can leak private data to new endpoints or mutate production records twice. The offline decision therefore comes first, using the paired comparison of Paired outcome comparison with the same criteria, sampling unit, budgets, and predeclared margin; comparing separate Wilson lower bounds does not answer it. In the worked case the lower difference bound misses the tolerance, so the candidate stays offline, and favorable traces leave the unmet gate unmet.
For a configuration that clears the offline gate, shadow execution copies each input into a separate execution with its own identity and allowance, while the baseline still serves the original task. Privacy and authorization checks must permit the copied input, its destinations, and the shadow tool operations, since discarding the output does not license sending private data to a new service. The shadow environment also bounds the answerable questions. An isolated stateful replica supports checks of the resulting artifact, a recorded-response fixture supports a narrower comparison under the matching-request rule, and a synthetic success reply to an unperformed mutation establishes nothing about production. Table 6 distinguishes the live task from the additional experiment.
Table 6: Shadow execution boundaries: Input mirroring adds an experiment. It neither divides the baseline’s original task assignment nor makes the copied data and tool traffic harmless.
Boundary
Baseline execution
Shadow execution
Task assignment
Serves the original admitted live task
Receives an authorized copy as a separate experiment
Effects
Uses only the live task’s current grant
Uses separately bounded isolated state or an explicitly supported observation path
Data access
Follows the task’s disclosure and access policy
Copying inputs and forwarding tool arguments require their own permitted recipients
Acceptance evidence
Applies the live task criteria
Applies only criteria supported by the shadow fixture’s fidelity
Delivery and cleanup
Returns the task result through its authorized channel
Withholds candidate output from the live task and settles its own resources and charges
Read access still has effects. Queries can acquire locks, consume quotas, trigger logging, or disclose arguments, so the shadow design names the services it contacts, relying on sandbox isolation and tool mediation.
Tool selection, token use, and latency differences can expose a changed operating pattern before live task assignment. For tool selection, the Jensen–Shannon divergence (Lin 1991) between the baseline and candidate distributions of tool calls, \(\BaselineToolCallDistribution\) and \(\CandidateToolCallDistribution\), summarizes the change in one number. It is built from the Kullback–Leibler divergence \(\KlDivergence(\BaselineToolCallDistribution\,\|\,\CandidateToolCallDistribution)=\sum_\ToolCallCategory \BaselineToolCallDistribution(\ToolCallCategory)\ln\bigl(\BaselineToolCallDistribution(\ToolCallCategory)/\CandidateToolCallDistribution(\ToolCallCategory)\bigr)\), the expected log-ratio between two distributions over tool calls \(\ToolCallCategory\), which is zero only when they match:
Lin, Jianhua. 1991. “Divergence Measures Based on the Shannon Entropy.”IEEE Transactions on Information Theory 37 (1): 145–51. https://doi.org/10.1109/18.61115.
Unlike \(\KlDivergence\) itself, \(\JensenShannonDivergence\) is symmetric and stays finite when one configuration uses a tool the other never calls; with natural logarithms, \(0\le \JensenShannonDivergence\le\ln 2\), which gives a fixed scale for comparing releases. For a sense of that scale, suppose the baseline sends 70 percent of its tool calls to tests and the rest to lint, and the candidate splits them evenly. That plainly visible shift gives \(\JensenShannonDivergence\) of about 0.021, roughly 3 percent of the 0.693 maximum that two configurations with no tool in common would reach. A useful alert threshold is therefore a small number, and it means something only against the divergence between two baseline samples of the same size, which shows how far the distribution moves by chance. The distributions must come from comparable inputs and one declared action-counting policy, and difference alone is not regression, since the intended repair may remove redundant calls. A shadow result can justify another test or a hold; it cannot supply evidence for production effects that the shadow deliberately prevented.
14.5.2 Canary task assignment
Routing canary traffic per model call rather than per trajectory mixes baseline_v1 and candidate_v2 inside a single repair task, making any resulting failure or side effect impossible to attribute to either configuration. Because a canary is live execution with real effects, the release policy declares the eligible population, allocation rule, candidate configuration (model request, prompt, tools, harness, and verifier versions), comparison policy, resource limits, and stop conditions before any task is assigned. An allocation fraction \(\CanaryAllocationFraction\) limits the share of eligible tasks assigned under that rule; it is neither a per-task authority grant nor a worst-case bound on total effects.
The policy assigns the configuration at the task or trajectory boundary and preserves it through continuation. Retries, resumed work, and delayed labels keep the original cohort identity, and dependent tasks from one customer or repository may need that larger unit of randomization. This is trajectory-level sticky routing, a stable configuration assignment rather than a promise to retain one physical model instance; workers can change through the resume protocol, and cohort assignment alone does not guarantee warm serving state.
↩︎ The resume protocol, under which a replacement worker takes ownership of a trajectory, is established in Resuming a Trajectory.
Outcome delay matters even when assignment is correct. Under the four-verdict rule of The four-verdict rule, every admitted task stays counted in its cohort, because counting only completed fast tasks can make a slow failing candidate look favorable. The analysis specifies when labels are mature and how unresolved cases affect promotion, and a failed label-delivery path is an evidence failure, not a reason to drop those tasks.
14.5.3 Sequential monitoring
Repeatedly applying a fixed-sample test after each new canary outcome inflates its false-positive rate well above the nominal level (Johari et al. 2017). A sequential design instead declares in advance how accumulating evidence will be used. The sequential probability ratio test (SPRT)6(Wald 1945) is the compact example for two specified distributions, and it is a different procedure from the paired offline noninferiority test.
6Sequential Probability Ratio Test: This is the sequential stopping rule of When to stop searching early applied to release rather than to search. Because sampling stops as soon as the evidence is decisive, the test typically reaches a decision with substantially fewer observations than a fixed-sample test with the same error targets, which for a canary means fewer candidate tasks exposed before the decision.
Let independent task labels \(\SequentialTestObservation_i\) be Bernoulli with a stationary acceptance probability, and let the simple hypotheses be \(\AcceptanceProbability=\HypothesizedAcceptanceProbability_0\) and \(\AcceptanceProbability=\HypothesizedAcceptanceProbability_1\), with \(0<\HypothesizedAcceptanceProbability_1<\HypothesizedAcceptanceProbability_0<1\) and the lower value a particular degradation. Both values are design inputs, not estimates. With \(\AcceptedLabelCount_n=\sum_{i=1}^{\ObservedLabelCount} \SequentialTestObservation_i\), the log-likelihood ratio favoring the degraded hypothesis is
Success moves the statistic in equation 2 downward; failure moves it upward. Let \(\StatisticalSignificanceLevel\) target the probability of choosing degradation under \(\HypothesizedAcceptanceProbability_0\), and \(\HypothesisTestFalseNegativeRate\) the probability of choosing \(\HypothesizedAcceptanceProbability_0\) under \(\HypothesizedAcceptanceProbability_1\). For \(0<\StatisticalSignificanceLevel,\HypothesisTestFalseNegativeRate<1\) and \(\StatisticalSignificanceLevel+\HypothesisTestFalseNegativeRate<1\), Wald’s commonly used approximate log boundaries are
Discrete observations can overshoot these approximate thresholds because a single task label jumps across the boundary in a finite step rather than stopping on it exactly. Calibrating the realized error probabilities therefore requires evaluating the discrete walk under the actual model (Fischer and Ramdas 2026).
Fischer, Lasse, and Aaditya Ramdas. 2026. “Improving Wald’s (Approximate) Sequential Probability Ratio Test by Avoiding Overshoot.”IEEE Transactions on Information Theory 72 (4): 2457–71. https://doi.org/10.1109/TIT.2026.3658855.
Evidence drifts toward hold under degradation and toward selecting \(\HypothesizedAcceptanceProbability_0\) otherwise; between the boundaries, keep sampling.
Systems Perspective 14.2: Reading a sequential decision
At \(\CumulativeLogLikelihoodRatio_n\ge \SprtUpperLogBoundary\), the test selects the specified degraded hypothesis and the release policy can stop new candidate admission. At \(\CumulativeLogLikelihoodRatio_n\le \SprtLowerLogBoundary\), it selects \(\HypothesizedAcceptanceProbability_0\) against that particular alternative. Between the boundaries, evidence remains inconclusive. Selecting \(\HypothesizedAcceptanceProbability_0\) rejects one specified degradation; general noninferiority, broader traffic, and safety each need their own evidence. A deadline or sample cap that ends collection between the boundaries leaves the decision inconclusive.
A production stream can violate this model through changing task mix, shared outages, correlated tasks, or delayed labels. Processing results in completion order creates selection bias when duration depends on outcome, which a declared admission-order analysis avoids at the cost of waiting behind unresolved tasks. Testing several metrics or restarting after an unfavorable path needs error control beyond these two boundaries.
The same simplified model gives a planning approximation for the expected stopping sample count. Under \(\HypothesizedAcceptanceProbability_1\), the expected increment is \(\SprtExpectedLogIncrement_1=\HypothesizedAcceptanceProbability_1\ln(\HypothesizedAcceptanceProbability_1/\HypothesizedAcceptanceProbability_0)+(1-\HypothesizedAcceptanceProbability_1)\ln((1-\HypothesizedAcceptanceProbability_1)/(1-\HypothesizedAcceptanceProbability_0))\). Neglecting boundary overshoot and substituting the target decision probabilities gives
Hory, Cyril, and Alfred O. Hero. 2004. Sequential Probability Ratio Test for the Detection of a Single Electron Spin in the OSCAR Setup. arXiv preprint quant-ph/0402181.
Napkin Math 14.3: Sequential evidence takes calendar time
Problem: Consider a separate, hypothetical later configuration that has met its offline and shadow prerequisites. Its monitor compares \(\HypothesizedAcceptanceProbability_0\) equal to 0.90 with \(\HypothesizedAcceptanceProbability_1\) equal to 0.75, using design targets \(\StatisticalSignificanceLevel\) equal to 0.01 and \(\HypothesisTestFalseNegativeRate\) equal to 0.05. At 60 eligible tasks/hour and a 5 percent candidate allocation, what stopping-sample and calendar-time scale does the no-overshoot approximation predict under \(\HypothesizedAcceptanceProbability_1\)?
Assumptions: Outcomes are independent with constant probability, admissions arrive at regular intervals, and each label arrives exactly 0.5 h after admission with none lost. The test is untruncated, with no operational tripwire.
Math: The approximate boundaries are 4.5539 and -2.9857. Under \(\HypothesizedAcceptanceProbability_1\), the expected increment is 0.09233. Neglecting overshoot and substituting the target decision probabilities gives expected terminal log-likelihood 4.1769. Dividing that quantity by the expected increment gives approximately 45.24 labeled tasks. The average includes both possible decisions, including mistaken selection of \(\HypothesizedAcceptanceProbability_0\); it is not the expected number until an alarm.
Candidate admissions occur at 3 tasks/hour. The corresponding admission-time scale is 15.08 h; adding the fixed label delay gives approximately 15.58 h until a decision. Tasks admitted while labels are pending can still be executing when the test stops.
Systems insight: The sample count is fixed by the hypotheses and error targets, so calendar time to a decision scales as \(1/\CanaryAllocationFraction\). Doubling the allocation to 10 percent cuts the time to about 8.0 h. Choose \(\CanaryAllocationFraction\) from the decision deadline and from the exposure the release can tolerate to harms the monitor does not measure, and keep independent admission and effect limits, because an expected sample count is not a maximum.
The sample count and the canary fraction control different harms. Under \(\HypothesizedAcceptanceProbability_1\), each candidate task fails about \(\HypothesizedAcceptanceProbability_0-\HypothesizedAcceptanceProbability_1\) more often than it would under the baseline. With allocation fraction \(\CanaryAllocationFraction\) of eligible arrivals at rate \(\ArrivalRate\) and label delay \(\CanaryLabelDelay\), the monitor decides after roughly \(\mathbb{E}_1[\SprtStoppingSampleCount]\) labeled tasks plus the \(\CanaryAllocationFraction\ArrivalRate \CanaryLabelDelay\) tasks admitted while their labels are pending, so the measured regression costs about \((\mathbb{E}_1[\SprtStoppingSampleCount]+\CanaryAllocationFraction\ArrivalRate \CanaryLabelDelay)(\HypothesizedAcceptanceProbability_0-\HypothesizedAcceptanceProbability_1)\) excess failures. In the notebook’s scenario that is about 7 at the canary fraction and about 11 at full allocation. The fraction enters only through the label delay; its main effect on a measured regression is the calendar time needed to collect the samples.
Harms the monitor does not measure behave differently. They accrue for every admitted candidate task until some other mechanism detects them, so their rate scales with \(\CanaryAllocationFraction\), and this canary exposes candidate tasks at 5 percent of the full-allocation rate. The test’s sample size fixes the harm from a measured regression; the canary’s size limits the harms the test does not measure. A production release of a foundation model shows what the second kind of harm looks like when it escapes.
War Story 14.1: The GPT-4o sycophancy rollback (2025)
Context: In late April 2025, OpenAI rolled out an update to the GPT-4o model behind ChatGPT, adjusting its default personality (OpenAI 2025b).
Mechanism: By OpenAI’s account, the update focused too much on short-term user feedback and did not account for how users’ interactions evolve over time. The model skewed toward responses that were overly supportive but disingenuous. OpenAI’s follow-up analysis reports that changes including an added reward signal from user thumbs-up and thumbs-down feedback weakened the influence of the primary reward that had held sycophancy in check, and that the offline evaluations and the A/B test of the update both looked good before launch (OpenAI 2025a).
Impact: The update reached ChatGPT users before the behavior was recognized, and OpenAI rolled it back, returning users to an earlier version with more balanced behavior.
Fix: OpenAI committed to refining training techniques and system prompts to steer the model away from sycophancy, adding guardrails for honesty and transparency, expanding predeployment testing with direct user feedback, and expanding its evaluations to catch behavioral issues beyond sycophancy.
Systems lesson: A release gate passes exactly the criteria it declares. A regression in a property that no criterion measures clears every term of the gate, and no sequential monitor can stop it because none is watching it. The durable responses are a new criterion for the next release and an admission hold on the regressed configuration.
OpenAI. 2025a. Expanding on What We Missed with Sycophancy. OpenAI blog.
The sycophancy regression is that unmeasured harm at full scale. Detection came from outside the declared criteria, and the effective response was an admission stop on the regressed configuration followed by settlement of what it had already done.
14.5.4 Operational tripwires
Waiting for a sequential statistical monitor to accumulate dozens of labeled tasks after a candidate dispatches a forbidden tool call allows a deterministic policy breach to keep repeating until the sample threshold is met. A hard operational tripwire therefore does not wait for statistical evidence: a verified forbidden dispatch, an exhausted release allowance, or too much unresolved candidate work immediately stops further admission under the declared policy. A task-count cap bounds effects only when each task’s effects are also bounded at enforcing endpoints; a routing percentage alone supplies neither bound.
↩︎ The per-trajectory budget vector that caps each task’s turns, tokens, elapsed time, and cost is established in Turn, token, time, and dollar budgets.
Rollback changes which configuration may receive new work. It does not retract transmitted requests, restore changed objects, or make a baseline worker eligible to continue a candidate trajectory without the ownership and compatibility checks of Resuming a Trajectory. Outstanding candidate operations keep their original identities, and an explicit recovery policy governs their effects, cleanup, and residual owners. Detection, confirmed admission stop, and effect settlement are distinct intervals, and effects that cannot be undone or remain unknown at the cutoff stay out of any average that implies every case was restored.
↩︎ Settling residual effects through admitted compensation under an accountable owner, once model-directed work has stopped, is established in Residual settlement.
14.5.5 Release measurements
Reporting only the pass rate of completed canary tasks—or comparing the throughput of an arrival-limited 5 percent canary directly against a fully loaded baseline—can make a slow, stalling candidate look both accurate and fast. Release reporting therefore keeps a fraction of tasks, a rate of completion, and the resources spent producing them distinct. The accepted task is the unit that trajectory goodput (principle 15) charges all initiated work against. That principle is developed with the economics of agents at scale; the release record needs only its measured rate, defined below.
↪ Trajectory goodput as the throughput metric for sizing an agent platform is developed in Residence, service, and quota.
For a declared admitted cohort of size \(\InitiatedTasks\), let \(\AcceptedTasks\) count tasks meeting the complete acceptance contract by the observation cutoff, so \(\AcceptanceFraction=\AcceptedTasks/\InitiatedTasks\). Pending and unknown cases sit in \(\InitiatedTasks\) as not accepted under the default of the four-verdict rule, and the report lists them separately from established failures.
For an observation interval \([\ObservationIntervalStart,\ObservationIntervalEnd)\), let \(\AcceptedTasks[\ObservationIntervalStart,\ObservationIntervalEnd)\) count acceptance events in that interval under the declared measurement policy. Over that interval, trajectory goodput is measured as
The denominator is elapsed observation time, not the sum of overlapping task durations, and the report states startup and drain treatment, concurrency, and arrival policy. A small canary can be arrival-limited, so its measured goodput is not comparable with a fully loaded baseline’s capacity. Table 7 keeps companion measurements tied to their denominators.
Table 7: Release measurements and their scope: Operational signals retain their units and populations. Zero observed violations does not establish zero future risk, and a denied proposal is distinct from an escaped effect.
Measurement
Denominator or boundary
Interpretation to preserve
Observed accepted fraction
All tasks in the declared admitted cohort
Report failed, pending, and unknown cases separately
Trajectory goodput
Elapsed observation interval
Match workload and capacity before comparing rates
Cost per accepted task
Accepted count, with all included cohort costs in the numerator
Include failed work and report unsettled liability; undefined when none are accepted
Intervention burden
Events per task, or tasks with an intervention divided by all tasks
Event counts can exceed one per task; routine required approval is not automatically a defect
Latency and settlement
Named start/end events and observation cutoff
Report unresolved cases; an empirical percentile is not a hard per-task deadline
Behavioral change
Comparable input mix and declared action-counting policy; \(\JensenShannonDivergence\) from equation 1
A different tool distribution may reflect improvement, regression, or changed demand
The paired comparison, diagnostic experiment, and sequential monitor answer different questions and cannot substitute for one another. A coherent release record brings them together with their fixtures, limits, and residual obligations so that the resulting decision can be inspected.
14.6 The Release Decision
In a paired evaluation cohort, candidate_v2 accepted more tasks than baseline_v1, while an isolated diagnostic probe pointed toward an effective observation-interface repair. Taken on their own, these indicators seem encouraging, but neither outcome satisfies a sound engineering release gate. The two results also concern different configurations. The 150-pair cohort compared candidate_v2 as submitted, and the worked trial of pool_repair_01 that produced ws_r19 sits outside that cohort. The interface repair changes candidate_v2’s observation adapter, so the configuration it would produce is a revision that needs its own paired evaluation; the probe is evidence about the defect, not about the revision’s cohort performance. Evaluating an agentic system requires an accountable evidence trail, in which engineers determine which claims have verified empirical support, which operational obligations remain unresolved, and what follow-up experiment the data actually justifies before routing production traffic to a new candidate.
14.6.1 The decision record
When an individual trial verdict, a paired cohort comparison, and a diagnostic probe sit in separate reports without their scope and provenance attached, a strong result on the local probe can easily be mistaken for proof that the candidate is ready to ship. A structured decision record prevents disparate experimental results from being conflated. Table 8 brings the worked individual trial, the paired cohort comparison, and the diagnostic interface experiment together in one decision record, preserving the provenance, scope, and operational boundary of each experiment so a positive result on a localized diagnostic probe cannot silently substitute for missing regression checks or fleet-wide superiority evidence. Rather than duplicating bulky workspace snapshots, the record links directly to immutable, content-addressed artifacts and explicitly logs when trace payloads, token charges, or downstream verdicts remain unavailable.
Table 8: An integrated evaluation record: Each field identifies what the evidence supports and which decision it constrains. Missing or pending evidence remains visible rather than becoming an implicit pass.
Evidence record field
Entry or required reference
Decision consequence
Task, fixture, and criteria
pool_repair_01, pool_fixture_v1, repair_accept_v1; record permitted effects, required checks, budgets, and residual-duty conditions
The verdict concerns this declared task contract, not unrestricted correctness.
Trial and artifact identity
The worked trial, configuration candidate_v2, artifact ws_r19; preserve grader and check versions
Do not confuse a trial, system configuration, source artifact, or historical operation.
Diagnostic completeness and population representativeness require evidence; redacted bytes cannot be claimed available for exact replay.
Diagnosis and regression
The controlled observation-interface experiment and its fixture-specific regression tests
Support a scoped intervention hypothesis without labeling the model or interface the sole possible cause. Its repeated fixture runs are not additional independent tasks in the paired cohort.
Release disposition
Retain the baseline; no live promotion of this candidate on the current evidence
Further work needs a stated question and planned evidence collection, not an automatic increase in traffic.
14.6.2 Evidence readiness
Promoting candidate_v2 on the current evidence would ship a configuration whose trial-level regression checks are unfinished, whose interface repair has been tested on only one saved prefix, and whose paired comparison interval still crosses the degradation threshold. The decision record leaves those three engineering obligations open, each at a different layer of the evaluation stack.
Trial-level verification: The targeted repair in ws_r19 passed its immediate assertion, but its full regression suite and final resource accounting remain pending. Engineers cannot mark an individual trial successful while side effects and outstanding token bills remain unquantified.
Component-level diagnosis: The observation-interface modification proved effective on continuations from one saved prefix of pool_repair_01, but success from a single prefix does not demonstrate generalized capability. Validating the interface repair requires testing across diverse environments and workloads.
Cohort-level release gate: The paired comparison across the task cohort yielded a confidence interval that spans the noninferiority threshold. The next comparison needs a preregistered paired sample designed for the anticipated gain, discordance, tolerance, and power.
Inspecting a single execution trace in greater detail resolves none of these three obligations: a granular trace can explain why an agent failed a specific step, but it cannot substitute for missing regression suites, multi-fixture validation, or cohort-wide statistical significance. Collecting that missing evidence under an established protocol is also distinct from altering the evaluation question itself. When a candidate fails to clear a deployment threshold, engineers may feel tempted to prune difficult tasks, drop incomplete trials, expand token budgets mid-flight, or tune task selection criteria. While these adjustments reliably inflate raw acceptance rates, they distort the operational meaning of the benchmark. Such post hoc filtering measures an artificially sanitized workload rather than genuine system capability. When task specifications, resource constraints, or operational boundaries legitimately change, the engineering team must record a revised evaluation protocol and test the configuration against a fresh, independent sample.
14.6.3 The declared decision rule
In the worked example, the lower limit of the paired acceptance interval drops below the permissible degradation threshold \(-\NoninferiorityTolerance\). The empirical evidence therefore establishes neither noninferiority nor superiority. The interval does not rule out the specified unacceptable loss, but it does not establish that candidate_v2 is inferior either. An inconclusive observed interval does not by itself determine the study’s planned power. The planning count in Paired outcome comparison is about 217 paired evaluations if the observed advantage persists, and even that count clears the gate with only roughly even odds, so a powered next sample uses the paired-variance planning approximation in Statistical Evaluation Rigor with its gain, discordance, and tolerance fixed in advance.
In production environments, engineering teams guard against wishful thinking by encoding deployment policies into explicit, declared decision rules. A release gate is formulated as a logical conjunction of independent criteria, each anchored to concrete empirical evidence:
Each predicate in equation 5 enforces a distinct operational requirement:
Paired noninferiority (\(\IndicatorFunction\!\left[\PairedDifferenceLowerLimit>-\NoninferiorityTolerance\right]\)): The lower bound of the confidence interval for the paired acceptance difference, \(\PairedDifferenceLowerLimit\), must exceed the allowed degradation margin \(-\NoninferiorityTolerance\). Clearing it supports, at the interval’s stated nominal confidence, the claim that the candidate does not degrade task completion by more than the tolerance.
Goodput preservation (\(\IndicatorFunction\!\left[\CandidateTrajectoryGoodput\ge(1-\GoodputRelativeTolerance)\,\BaselineTrajectoryGoodput\right]\)): Candidate trajectory goodput \(\CandidateTrajectoryGoodput\) must match baseline goodput \(\BaselineTrajectoryGoodput\) within a declared relative tolerance \(\GoodputRelativeTolerance\), evaluated across matched traffic profiles, workloads, and compute capacity.
Tail-latency constraint (\(\IndicatorFunction\!\left[\CandidateTailCompletionTime\le \CompletionDeadline\right]\)): The empirical 99th-percentile task completion time \(\CandidateTailCompletionTime\) across the evaluation cohort must not exceed the operational ceiling \(\CompletionDeadline\). This bound serves as a cohort-level service-level objective rather than a per-task watchdog timeout.
Zero-violation tripwire (\(\IndicatorFunction\!\left[\CandidateViolationCount=0\right]\)): The count of verified safety or contract violations \(\CandidateViolationCount\), such as unauthorized data modifications, privilege escalations, or uncontained tool executions, must equal zero. A single verified breach trips the gate immediately. Engineers must note, however, that observing zero violations within an evaluation cohort demonstrates only the absence of detected failures on that specific sample; it does not constitute a mathematical proof of absolute safety.
Certain operational metrics serve as diagnostic triggers rather than binary gates. For example, if a behavioral-drift metric such as the tool-distribution divergence in equation 1 crosses its declared alert threshold, the gate pipeline routes the candidate to manual engineering review rather than failing it outright. A significant change in tool usage frequency is not necessarily an anomaly; it often represents the exact behavioral modification intended by an interface or prompt redesign.
Because the release rule is a strict boolean conjunction, a failure in any single predicate yields \(\ShipDecision = 0\). Exceptional tail latency, lower token expenses, or an unblemished safety record cannot compensate for an inconclusive or degraded task acceptance rate. At the same time, the gate enforces only the predicates explicitly written into its formula. Any failure mode the conjunction does not monitor passes through it, as war story 14.1 showed.
For the worked candidate the first predicate already fails, so the gate returns zero without evaluating the goodput, latency, and violation terms, none of which the worked evidence measures for candidate_v2. The decision is to retain baseline_v1 in production and withhold live traffic from candidate_v2. The engineering team’s immediate next steps are to complete the pending regression runs for ws_r19, reconcile open token and tool expenditures, and pre-register an appropriately powered sample size before collecting further paired comparison data.
Automated gates evaluate these conditions deterministically, yet the underlying metrics reflect varying degrees of statistical uncertainty. A release evaluation report must therefore label each criterion with its precise operational status: satisfied, failed, or inconclusive under the declared evidence rule. Furthermore, individual 95 percent confidence intervals evaluated across separate operational metrics do not compound into an overall 95 percent system guarantee. When \(\TestFamilySize\) statistical predicates are each allowed probability \(\StatisticalSignificanceLevel\) of a wrong verdict, the chance that at least one of them is wrong is bounded only by \(\TestFamilySize\StatisticalSignificanceLevel\), since the probability of any error is at most the sum of the individual error probabilities, and it approaches that bound when the individual errors rarely coincide. That combined probability is the family-wise error rate, and the simplest control divides the allowed error among the tests, testing each predicate at \(\StatisticalSignificanceLevel/\TestFamilySize\). Production evaluation protocols must document their family-wise error accounting, specifying how they control for multiple hypothesis tests and for repeated peeking across sequential evaluation checkpoints.
Checkpoint 14.4: Defend the evaluation decision
An agent evaluation pipeline exists to produce an auditable decision, recorded immutably, that identifies the exact configuration under test, establishes what the empirical evidence proves, authorizes the next operational step, and transparently logs every duty that remains open. Skipping any link in that chain turns an operational measurement into a misleading release signal.
14.7 Fallacies and Pitfalls
Even an apparently thorough operational record can answer the wrong engineering question. The failure modes below break the link between operational evidence and deployment readiness, and each exacts a measurable cost in operational stability, exposure, or diagnostic visibility.
Pitfall: Relying on uniform head sampling to keep examples of rare failures.
Head sampling makes a retention decision at request ingestion, long before the agent’s final execution outcome is known (section 14.3). Because severe regressions and edge-case agent failures are inherently rare, uniform head sampling discards the vast majority of critical failure traces. At a 10 percent sampling rate, an intermittent failure mode that occurs 5 times across an evaluation run leaves no detailed trace 59 percent of the time, blinding engineers during root-cause debugging. Systems should instead employ tail sampling keyed to task outcomes and anomalous step traces. Engineers should retain a small, fixed uniform control sample to validate aggregate rates, while computing production failure rates from complete, lightweight accounting telemetry rather than sampled execution traces.
Fallacy: A small canary fraction limits the damage a regression can do.
For regressions that automated monitors track, the total blast radius depends on the sample count required to achieve statistical confidence rather than the traffic split percentage (section 14.5). In the reference monitor, a decision needs the same expected number of labeled tasks at any allocation (notebook 14.3), so the measured regression costs about 7 excess failures at a 5 percent allocation and about 11 at full allocation. A small fraction trims only the failures among tasks admitted while labels are pending, and it stretches the time to a decision to 15.58 h. A small traffic fraction limits damage only for unmonitored hazards or qualitative failures that human operators spot manually. Automated deployment systems must therefore enforce hard circuit-breaker limits and absolute failure thresholds that operate independently of the canary traffic fraction.
Pitfall: Treating a clean diagnostic run or a rollback as closing the incident.
A successful diagnostic run demonstrates stability only within the narrow confines of its specific test fixture (section 14.4.3). Statistically, observing 20 accepted runs out of 20 diagnostic executions still leaves the true failure rate as high as 16 percent at 95 percent confidence. Furthermore, rolling back candidate code or throttling traffic merely prevents new failures; it cannot retract state mutations, external tool invocations, or corrupted database records that the agent already generated. Teams must maintain regression monitoring, enforce bounded exposure limits, and retain designated recovery owners until all asynchronous downstream effects have fully settled.
Pitfall: Reading a successful span or a zero exit code as a passed task.
Span status records how one operation ended, not whether the task met its contract (section 14.2.2). A test process can exit cleanly while its output shows failed assertions, and a client span can end in a timeout while the candidate still produces a valid patch from cached context. A dashboard that counts green spans as accepted work therefore reports activity rather than acceptance, and the gap widens exactly when verdicts arrive late, after the execution spans have closed. Traces should carry a reference to the task verdict rather than a copy of it, and every acceptance figure should come from the accounting record, where pending and unknown verdicts stay in the denominator.
Fallacy: A configuration that clears every term of the release gate is safe to release.
A conjunctive gate enforces exactly the predicates written into it and nothing else (section 14.6.3). A regression in a property no criterion measures clears every term, and no sequential monitor stops it, because none is watching that property. The sycophancy rollback of war story 14.1 followed offline evaluations and an A/B test that both looked good. Zero observed violations in a cohort also bound the violation rate only as tightly as the cohort’s size allows. A passed gate supports the scoped claim its predicates state, and the canary’s allocation, the hard tripwires, and a recovery owner for every outstanding effect remain the protection against what the gate does not measure.
14.8 Summary
An agent configuration that acts on live work produces evidence that no offline comparison collects, and operating it well means deciding in advance which of that evidence to keep and what each piece can establish. A trace binds model calls, tool executions, and checks to their trial, but a span’s status is not a task verdict, and summing overlapping spans overstates elapsed time that only the critical path explains. Retention keeps rare failures only when selection waits for verdicts, while the accounting record, which is never sampled, supplies every population rate. A postmortem separates the findings each boundary supports and tests an explanation by intervening on a saved prefix, so its conclusions hold for that fixture and need broader evaluation before they support a release. Staged exposure then trades calendar time against harms the monitor does not measure, and a rollback stops new admission without settling what the candidate already did. The worked case ends where the evidence does. The interface repair is supported on its fixture, the offline comparison remains inconclusive, and the declared gate returns zero, so baseline_v1 stays in production while the team collects the evidence the decision record names as missing.
The quantitative decisions developed across this chapter rely on the mathematical relationships summarized in table 9. Each formulation either sizes the operational evidence required to support a claim or translates that evidence into a scoped promotion decision.
Table 9: Key equations of agent operations: Each relation holds under the sampling, independence, and labeling assumptions stated where it is derived; none replaces the acceptance contract, the four-verdict rule, or the accounting record.
The probability that a failure mode seen \(\FailureModeOccurrenceCount\) times leaves no detailed trace under head sampling at rate \(\HeadSamplingProbability\)
\(\CumulativeLogLikelihoodRatio_n\) against \(\SprtUpperLogBoundary=\ln\frac{1-\HypothesisTestFalseNegativeRate}{\StatisticalSignificanceLevel}\) and \(\SprtLowerLogBoundary=\ln\frac{\HypothesisTestFalseNegativeRate}{1-\StatisticalSignificanceLevel}\); \(\mathbb{E}_1[\SprtStoppingSampleCount]\approx\frac{(1-\HypothesisTestFalseNegativeRate)\SprtUpperLogBoundary+\HypothesisTestFalseNegativeRate \SprtLowerLogBoundary}{\SprtExpectedLogIncrement_1}\)
When a canary monitor selects the degraded or the baseline hypothesis, and the labeled tasks that decision takes
Key Takeaways: Keep operational evidence scoped to what it shows
A trace correlates; it does not authorize or accept: Spans bind model calls, tool executions, and checks to their trial and artifact, and the critical path through their dependencies, not the sum of their durations, explains elapsed time. A diagnostic trace may be sampled or truncated, so it never replaces the durable recovery log or the task verdict.
Retention decides what diagnosis can see: Uniform head sampling before a run completes systematically misses rare failure modes. Conversely, verdict-aware retention prioritizes intermittent bugs but skews raw telemetry distributions. Population failure rates come from complete accounting records; estimates from diagnostic samples require appropriate inclusion probabilities and reweighting.
A postmortem tests an explanation by intervening: Findings stay separated by boundary (interface defect, model response, runtime refusal), and a hypothesis is tested by a controlled branch from a saved prefix, not by rereading telemetry. Its conclusion holds for that fixture; broader claims need the evaluation of a revised configuration.
Canary allocation and monitoring control different harms: Sequential hypothesis testing, such as Wald’s SPRT, sets the expected sample count needed to reach a decision on a measured regression. Reducing the canary allocation primarily extends the time needed to collect that evidence while lowering the exposure rate to unmonitored harms. Unmonitored failure modes bypass statistical canary monitors entirely and require hard operational tripwires. Furthermore, rolling back routing terminates only incoming traffic; repairing external mutations and state corruption requires explicit recovery mechanisms.
A release is a decision record, not a scorecard: A production release gate mandates satisfying every declared acceptance criterion simultaneously. Superior performance on task accuracy cannot compensate for regressions in tail latency, budget breaches, or safety violations. Inconclusive trials must never be converted into default approvals, and rigorous experimental protocols require fixing hypotheses, sample sizes, and decision thresholds before collecting evaluation data.
Enforcing causal and statistical release evidence (principle 12) requires a release protocol to specify its task criteria, operational boundaries, exposure policy, and statistical uncertainty explicitly. By tying release gates directly to structured execution traces and paired counterfactual baselines, systems engineers ensure that promotion decisions rest on reproducible evidence with stated limits rather than on summary scorecards.
14.9 Historical Perspective and Further Reading
The statistics of staged release are older than electronic computing. During World War II, Abraham Wald developed the sequential probability ratio test (SPRT) to optimize industrial quality inspection, showing that deciding acceptance or rejection dynamically as evidence accumulates minimizes the expected sample size relative to fixed-horizon trials (Wald 1945).
Lamport, Leslie. 1978. “Time, Clocks, and the Ordering of Events in a Distributed System.”Communications of the ACM 21 (7): 558–65. https://doi.org/10.1145/359545.359563.
Fonseca, Rodrigo, George Porter, Randy H. Katz, Scott Shenker, and Ion Stoica. 2007. “X-Trace: A Pervasive Network Tracing Framework.”4th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 271–84.
Kohavi, Ron, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. 2009. “Controlled Experiments on the Web: Survey and Practical Guide.”Data Mining and Knowledge Discovery 18 (1): 140–81. https://doi.org/10.1007/s10618-008-0114-1.
Johari, Ramesh, Pete Koomen, Leonid Pekelis, and David Walsh. 2017. “Peeking at A/B Tests: Why It Matters, and What to Do about It.”Proceedings of the 23rd ACMSIGKDD International Conference on Knowledge Discovery and Data Mining, 1517–25. https://doi.org/10.1145/3097983.3097992.
OpenAI. 2025b. Sycophancy in GPT-4o: What Happened and What We’re Doing about It. OpenAI blog.
The telemetry required to diagnose multi-turn agent execution draws directly from distributed systems observability. Lamport established that asynchronous events across decoupled components admit only a partial order unless system metadata explicitly captures their causal links (Lamport 1978). Fonseca et al. (2007) extended this principle to modern network architectures with X-Trace, propagating deterministic task contexts across diverse protocol layers and administrative domains to reconstruct complete end-to-end execution paths. In the same years, production web services transformed deployment validation into continuous experimentation: Kohavi et al. (2009) codified online controlled experiments with staged canary deployments as standard release gates. Sigelman et al. (2010) then proved in Dapper that distributed tracing could operate continuously at production scale by decoupling span collection from the critical path and employing aggressive head-based sampling to bound runtime overhead to negligible levels. Google’s site reliability engineering practice established the blameless postmortem, institutionalizing structured incident postmortems as essential operational records for diagnosing systemic failures (Beyer et al. 2016). When engineering teams began monitoring live experimentation dashboards in real time, Johari et al. (2017) proved that repeated significance testing inflates false-positive rates far above the nominal level, establishing why sequential testing bounds, the descendants of Wald’s test, must govern automated canary rollouts. Agent releases add what Wald’s inspection lines did not face: labels that arrive hours after admission, tasks whose effects outlive the decision, and harms that no monitored metric records, the gap that the 2025 sycophancy rollback exposed (OpenAI 2025b).
Further reading.
Wald, “Sequential Tests of Statistical Hypotheses”(Wald 1945): The original derivation of the sequential probability ratio test, its approximate decision boundaries, and its average sample number, providing the mathematical foundation for staged canary deployments and early-stopping test monitors.
Sigelman et al., “Dapper”(Sigelman et al. 2010): The foundational architecture of production distributed tracing under strict overhead budgets, examining the sampling policies and trace contexts that enable fine-grained post-hoc diagnosis.
Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments(Kohavi et al. 2020): A practitioner’s guide to randomized experimentation in production systems, covering assignment units, sample-ratio mismatches, and the statistical pitfalls that invalidate live rollout decisions.
Beyer et al., Site Reliability Engineering(Beyer et al. 2016): The practitioner’s account of operating production services, including the blameless postmortem and the release engineering that agent operations extend from people to components.
Wald, Abraham. 1945. “Sequential Tests of Statistical Hypotheses.”The Annals of Mathematical Statistics 16 (2): 117–86. https://doi.org/10.1214/aoms/1177731118.
Kohavi, Ron, Diane Tang, and Ya Xu. 2020. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. https://doi.org/10.1017/9781108653985.
14.10 Exercises
Each problem states its constraints and ends in a decision about what the evidence supports or what to collect next. Givens are hypothetical unless stated otherwise, and every interval is a nominal 95 percent interval with \(\IntervalNormalQuantile=1.96\).
1. Where the time went (Basic). A repair-agent timing trial records these spans on one host clock, in seconds after segment start: retrieval 0.0 to 1.2; a model call proposing checks 1.2 to 4.0; targeted tests 4.0 to 13.0; a static lint check 4.0 to 7.5; a dependency audit 4.0 to 10.0; and a model call synthesizing the report 14.0 to 17.0. The three checks are independent of one another, synthesis waits for all three, and the enclosing segment ends at 17.5. (a) Compute the sum of the leaf span durations and the critical-path duration, and find how much of the 17.5 s segment the spans leave unattributed. (b) A cache cuts the targeted tests to 5.0 s with everything else unchanged. Compute the new critical path and the saving, and name the span that now limits the check phase. (c) A routing change instead halves both model calls. Compute its saving, compare it with part (b), and state which assumption about the unattributed intervals both estimates share. (d) Decide which change to fund first if each costs the same to build, and name the controlled measurement that must confirm the saving before it is claimed.
2. A retention budget for rare failures (Intermediate). A repair fleet runs 20,000 evaluation trials per day, each with a 2 MB (decimal) permitted trace body. At the selection boundary, 3 percent of trials have failed a criterion, 1 percent remain pending or unknown, and the rest are ordinary accepted trials. One failure mode occurs 8 times per week, and downstream diagnostic storage is capped at 1.5 GB per day. (a) Compute the daily volume of full retention and of uniform head sampling at \(\HeadSamplingProbability=2\) percent, and the probability that head sampling keeps no detailed trace of the rare mode in a week. (b) Compute the daily volume of tail selection that keeps every failed and unresolved trial plus a 1 percent random control sample of ordinary trials, and check it against the cap. (c) Design a selection policy within the cap that keeps every failed trial. State the sampling rates for unresolved and ordinary trials, the inclusion probabilities the record must keep, and one change to the trace body that would relax the cap. (d) An analyst reports the fraction of retained traces under the policy of part (b) that failed as the fleet’s failure rate. Compute the reported value, and name the record the failure rate must come from.
3. Drift in shadow execution (Intermediate). A shadow run of the revised repair configuration records its tool calls over four categories, read, test, lint, and edit. The baseline distributes its calls as 0.40, 0.30, 0.10, and 0.20, and the candidate as 0.30, 0.45, 0.05, and 0.20. Two independent baseline samples of the same size differ by a Jensen–Shannon divergence of 0.004 nats. (a) Compute \(\JensenShannonDivergence\) between baseline and candidate with equation 1, in nats, and express it as a fraction of its maximum, \(\ln 2\). (b) Release policy routes a candidate to engineering review when its divergence exceeds three times the baseline-versus-baseline value. Decide whether this candidate is routed, and whether the decision would change under a five-times rule. (c) The candidate is the paginated observation interface, which retrieves remaining results instead of guessing. Explain why more test calls and fewer reads may be the intended change, and state which measurement, not the divergence, decides whether the shift is an improvement. (d) The shadow copies also read from a production ticketing service. Name three effects such reads can have and the declarations the shadow design must make before it mirrors any input.
4. Postmortem of a wrong refund answer (Advanced). The support agent tells a customer that a purchase is refundable within 60 days, while the current policy allows 30. The retained trace holds three records. The raw retrieval result contains two policy passages, the current version and a superseded one, each with an effective date; the delivered context contains both passages with the effective dates removed by the observation adapter; and the answer quotes the superseded passage. The team restores a compatible saved prefix ending just before the retrieval call and runs 40 fresh continuations in each of three branches, holding the model configuration, sampling settings, prompt, and a policy-grounded reference check fixed; every run reaches a known verdict. Branch A keeps the original adapter and yields 18 wrong answers. Branch B delivers the effective dates and yields 3 wrong answers. Branch C removes the superseded passage from the index, so only the current version is retrieved, and yields none. Release policy requires the Wilson upper limit on the support agent’s per-session wrong-answer rate to be at most 2 percent. (a) For each claim, state whether the retained record alone, a regression replay of the captured result without a model call, or only the intervention can establish it: the adapter removes effective dates; the model received both passages; the missing dates contributed to the wrong answer. (b) Compute each branch’s wrong-answer rate with its Wilson interval. Decide whether the data support the claim that delivering effective dates reduces wrong answers from this prefix, and whether 40 runs per branch distinguish branch B from branch C. (c) Compute how many consecutive correct continuations a fix would need before its Wilson upper limit met the release policy, and the corresponding lower confidence bound on five-session consistency from branch C’s upper limit at 40 runs, assuming independent recurrences from this same saved prefix. (d) A teammate reports a fourth branch that delivers the dates and also lowers the sampling temperature to zero, with 40 of 40 correct, and proposes shipping it as the fix. Critique the proposal, choose a corrective change, and write the postmortem record’s lines for established findings, tested change, and remaining questions.
5. Size a canary against a deadline (Advanced). A repair-agent canary monitors acceptance with the SPRT of equation 2, comparing \(\HypothesizedAcceptanceProbability_0=0.80\) with \(\HypothesizedAcceptanceProbability_1=0.70\) at design targets \(\StatisticalSignificanceLevel=0.05\) and \(\HypothesisTestFalseNegativeRate=0.10\). Eligible tasks arrive regularly at 40 per hour, each label arrives 2 hours after admission when the regression suite finishes, and under \(\HypothesizedAcceptanceProbability_1\) the planning target is an approximate expected decision time of at most 24 hours from the first admission. Separately, a harm the monitor does not measure occurs in 0.5 percent of candidate tasks, and release policy requires the candidate to run for 7 days at its allocation before any wider release. (a) Compute \(\SprtUpperLogBoundary\), \(\SprtLowerLogBoundary\), \(\SprtExpectedLogIncrement_1\), and the no-overshoot \(\mathbb{E}_1[\SprtStoppingSampleCount]\). (b) Find the smallest allocation \(\CanaryAllocationFraction\) that meets that expected-time target. (c) Compute the expected excess failures from the measured regression at that \(\CanaryAllocationFraction\) and at \(\CanaryAllocationFraction=25\) percent. (d) If the allocation continues for all seven days, compute the expected count of unmeasured harms over the 7-day period at both allocations. Choose \(\CanaryAllocationFraction\) and justify it from parts (b) through (d). Explain why this exposure calculation is not an instruction to keep admitting the candidate after a degradation alarm. (e) A teammate proposes replacing the SPRT with a fixed-sample test rerun after every new label, stopping at the first value below 0.05. Critique the proposal.
6. Critique a release report (Advanced). A team asks to promote a repair-agent candidate with the following evidence. The candidate accepted 78 percent of tasks and the baseline 72 percent, reported as a 6-point gain. The baseline ran on 100 tasks sampled last month, and the candidate on 100 newly sampled tasks. The team dropped 12 candidate trials whose regression suites had not finished, so the candidate accepted 69 of the remaining 88. Report quality counted as accepted when an LLM judge scored it at least 4 of 5, justified by 90 percent raw agreement with a second judge. The failure rate in the release notes was computed from tail-sampled traces. (a) Under the four-verdict rule, bound the candidate’s accepted fraction of its admitted cohort, and state what the 6-point claim becomes. (b) Using equation 13.3 with \(\PooledAcceptanceProbability=0.75\), a 5 percent two-sided significance target, and 80 percent power, compute the tasks per independent arm needed to detect a 6-point difference. (c) Redesign the comparison as paired. Assuming a discordant fraction \(\DiscordantPairFraction=0.15\), an anticipated gain of 3 points, and \(\NoninferiorityTolerance=2\) points, find the number of task pairs at which the normal-approximation lower limit \(\EstimatedPairedAcceptanceDifference-\IntervalNormalQuantile\sqrt{(\DiscordantPairFraction-\EstimatedPairedAcceptanceDifference^2)/\EvaluationSampleCount}\) clears \(-\NoninferiorityTolerance\) if the gain holds. Contrast this expected-limit-centering count with the fixed-variance normal planning count at 80 percent power, using power quantile 0.84. Both calculations assume the stated gain and discordance; the powered count approximates a normal gate rather than calibrating the exact paired-score interval. (d) Identify the remaining defects in the report and write a defensible protocol: which criteria the automated gate enforces, which measurements the judge must carry, and which conditions the team must declare before collecting data.
What’s Next: From operational evidence to curated trajectories
Operation leaves a qualified record: verdicts held in the accounting record, traces retained with their inclusion probabilities, and a postmortem that separated an interface defect from a model behavior and a runtime refusal. Trajectory Curation uses that record to guide diagnosis, admission, and corpus design, and its first decision is triage. The truncated-observation failure that the postmortem traced largely to the observation adapter belongs in an interface fix, not in a corpus. A logged success is not automatically a useful demonstration, and a failure is not automatically a policy defect. Preserve criterion and configuration provenance, unresolved evidence, and the separation between development data and held-out evaluation. Curation turns that qualified record into a learning-data decision.
