20 Conclusion
Purpose
Agent stack focus for this chapter.
Why can individually tested components still leave an agent’s task unfinished or its effects unaccounted for?
An autonomous agent unites disparate systems disciplines (foundation models, memory hierarchies, isolated execution sandboxes, transactional event logs, and independent verifiers) into a single closed-loop control architecture. Each individual component can satisfy its local unit tests, yet the integrated system can still fail if the contracts connecting them are ungrounded or ambiguous. The core challenge of agentic systems engineering is designing the deterministic harness that bounds and verifies a stochastic proposer, a problem that larger models and more elaborate prompts do not remove. Across a trajectory horizon, carried working state, and delegated authority, reliable autonomy is achieved only when the runtime enforces invariant closure at every system boundary.
Learning Objectives
- Trace an incident-remediation trajectory, identifying the owner and record of each consequential transition
- Translate a task contract into bounds and acceptance requirements proportionate to its horizon, state, and authority
- Diagnose state and evidence mismatches at subsystem boundaries using the authority and lifetime of each record
- Resolve an uncertain external action without confusing a lost response with permission to repeat its effect
- Select a system intervention using attributed failures, outcome evidence, and its full deployment cost
- Construct a safety case that connects scoped claims to arguments, evidence, assumptions, and residual uncertainty
- Distinguish assistance with specifications from authority to adopt them and accept the resulting task
20.1 Answers to the Three Questions
Every mechanism in the book now carries a price in one unit, the cost per accepted task of each lighthouse (Cost per Accepted Task by Archetype). The three architectural questions posed in the introduction, each answered in part by every later chapter, therefore admit concise engineering answers, and each answer is a design claim that one live operational workload can test.
↩︎ The three foundational architectural questions governing agentic design are established in The book's three questions.
The first question asked when an adaptive, model-directed loop earns its cost over a deterministic workflow. The answer is settled not once for an entire application but at each decision point, by measuring the cost per accepted task under a fixed contract. A model should direct the next action only where interpreting dynamic, unstructured evidence improves the task acceptance rate by a larger factor than the added inference calls increase the cost of an attempt. Deterministic software should govern everywhere else. Evaluated by this metric, dynamic model loops can earn their cost at phases such as root-cause diagnosis, where the next diagnostic transition cannot be enumerated in advance, while deterministic control structures manage the predictable orchestration surrounding them. The incident below supplies a cost threshold for that comparison. Its price, worked out in section 20.3.2, shows the model calls to be a small fraction of one remediation attempt’s cost. Whether additional diagnostic calls earn that cost depends on the relative acceptance improvement measured against a matched baseline under the same contract and latency constraints.
The second question asked what operational machinery must surround the model. The answer scales with the task’s exposure to state, authority, and horizon. Carrying state across turns requires selectively retrieved context, explicitly budgeted serving state, and durable audit records that preserve source authority, whereas exercising authority over external services adds typed, mediated action interfaces, definitive effect settlement before any retry loop runs, and process containment enforced beneath the model. When those actions span a multi-turn horizon, the runtime must also wrap the trajectory in a supervisory harness, record a write-ahead log of intent before committing each external side effect, and register compensating workflows for actions that cannot be reversed cleanly. A read-only lookup needs little of this protective scaffolding, whereas an autonomous remediation agent operating on live production systems needs all of it.
The third question asked how operators verify that a task has actually succeeded. The answer requires objective evidence collected from an authoritative endpoint that directly monitors the target system, evaluated against an explicit contract adopted by system owners before execution began. This verification must satisfy a pre-declared closure evidence level, with any residual uncertainty explicitly quantified and assigned to an accountable owner. A model’s self-generated report never serves as evidence, however coherent or persuasive, because a component executing within the loop cannot certify its own output. Stated as an operational acceptance rule, this is the discipline of invariant closure (principle 1).
Each answer is straightforward to state for an isolated component; agentic architectures fail most often where the answers must hold across subsystem boundaries, because individually tested services can drop an in-flight task when state, authority, and verification pass between them. The answers are therefore tested on one complete incident handled by the remediation agent, the lighthouse that exercises horizon, state, and authority at once, followed from its contract across the memory and execution boundaries to the one boundary no runtime can close on the owner’s behalf.
20.2 One Incident Through the System
A database service triggers a query-latency alert shortly after a configuration change. An automated remediation agent inspects telemetry, correlates logs, and proposes reverting to the previous known-good settings. The human operator approves the rollback, but the configuration endpoint drops the network connection before returning an acknowledgment. Whether the mutation took effect is now unknown, the classic dilemma of a lost acknowledgment in a distributed system, and no fluent explanation from a language model can resolve it. If the runtime treats the timeout as a failure and retries under a fresh operation identifier, it risks executing the mutation twice; if it optimistically assumes success, it risks closing an incident while degraded queries continue to impact production; and if it simply halts, it leaves a live production change in an unverified state with no entity accountable for reconciling it. Each failure mode arises by default when every component satisfies its local contract and no component owns the boundary between them. The end-to-end argument that grounded the closure evidence levels (The invariant closure principle) applies to this single incident. An intermediate success report from a tool or a model call cannot establish acceptance, and only the principal holding the task contract can verify completion against direct evidence from the service.
Horizon, state, and authority return together, and the runtime closes each on one trajectory.
This incident, incident_42 on database service db_primary_01, is a workload of the remediation agent (Four lighthouse workloads), the book’s irreversible actor, which operates over a bounded horizon, mutates shared production state, and exercises real authority over a live service. Its single trajectory, from diagnostic telemetry through operator approval, dispatch, and acceptance, anchors the rest of the chapter. At every transition the criterion is the same. The system must durably and verifiably record what it proposed, what permissions were granted, what effects occurred, what evidence validates the outcome, and which principal bears ongoing responsibility. In this architecture, the managed database service owns its active configuration, the runtime orchestrator owns execution and control decisions, and the accepting authority verifies whether post-remediation evidence satisfies the delegated operational contract. One named human principal, operations_owner_42, is accountable both for any operation the runtime cannot settle and for the service once the incident is accepted. The runtime’s settlement process, a runtime component rather than a person, reconciles uncertain operations on that owner’s behalf.
20.2.1 Trajectory ownership
The incident lifecycle begins under incident_contract_v1, which grants bounded diagnostic read access and sets gates for proposing, approving, and validating configuration mutations. Because initial privileges are observational only, the runtime provisions read_scope_42 to query authorized service endpoints so that telemetry gathering cannot inadvertently trigger state changes or be coerced into production writes. Throughout this trace, identifiers such as read_scope_42 denote durable local record keys rather than opaque cryptographic hashes.
Next, the context manager gathers the initial latency alert, recent time-series metrics, active configuration parameters, and relevant runbook sections into an auditable manifest, context_manifest_42. Every observation in this manifest records its originating source, a version tag when available, and a precise ingestion timestamp. The retrieved runbook suggests a memory-pool misconfiguration, but it might describe a previous software deployment. By diffing the runbook recommendations against the active production configuration (config_v7) and current telemetry, the agent formulates a diagnostic hypothesis. Additional diagnostic queries, constrained by the task’s remaining call budget, must corroborate this hypothesis against alternative explanations such as downstream network congestion or disk saturation.
When the diagnostic evidence converges on a known regression, the model emits change_request_42, which specifies the target configuration keys, the desired values, and the expected baseline version config_v7. In accordance with incident change policy, an authorized human operator reviews the proposed diff and issues approval_42. This record binds the operator’s identity to the request identifier, the target host, the exact payload parameters, the prerequisite baseline config_v7, and an expiration window. The approval records permission for that exact payload and nothing broader; it does not perform the mutation. The model drafts the operational rationale, but it cannot grant itself execution authority. An approval granted minutes before execution is an earlier authorization check, and complete mediation (definition 7.1) requires a fresh one at the moment of use. Prior to network dispatch, the runtime action gate therefore rechecks operator authorization, compares the payload against the approval’s digest, inspects the expiration deadline, confirms resource quotas, and evaluates runtime preconditions. If an operator revokes the ticket or the model generates an altered proposal, the original approval no longer matches and cannot silently apply to a mutated repair payload.
↩︎ Rechecking an approved request’s digest, expiry, budget, and authorization at the dispatch boundary is established in Dispatch-time rechecks.
Even a thorough pre-dispatch check leaves a time-of-check to time-of-use (TOCTOU) race if a peer operator changes the settings between the gate check and the write. The endpoint closes this window with the conditional update of Conditional updates, an atomic expected-version check that applies the change only if the live version still matches config_v7 and otherwise rejects it, returning the task for re-diagnosis. The endpoint also makes the write idempotent in the sense of definition 7.5: it deduplicates requests by an operation identity bound to the caller, target, and payload, and retains that identity across a window longer than the maximum client retry horizon, so a replayed request inside the window is recognized rather than reapplied. A read-only operation-status lookup completes the endpoint contract. These guarantees must live in the endpoint itself, since a client tool schema can declare an invariant but only the server can enforce it. Before transmitting the RPC, the runtime appends an intent record for op_42 to a write-ahead journal at the durability boundary chosen for the deployment, together with its recovery descriptor: the obligation registration of the trajectory saga, here the corrective-action template, the preconditions it must satisfy, and the accountable owner, operations_owner_42. Once both are persistent on disk, the runtime dispatches the request with operation identity op_42 and expected version config_v7.
↩︎ Deduplication by an operation identity bound to caller, target, and payload is established in Operation identity and endpoint contracts.
The endpoint’s response is then lost in transit, and the runtime sees a socket timeout. This is the lost-acknowledgment problem that opened Recovering After Partial Effects, and the timeout yields no information about execution state. The network may have dropped the outbound packet, the endpoint may still be executing the transaction, or the endpoint may have committed the write and lost the acknowledgment on the way back. Instead of retrying blindly or declaring failure, the runtime logs the status of op_42 as unknown and hands it to the settlement process, which issues an authorized status query against the endpoint’s operation registry rather than a second mutating RPC. In the successful branch of this scenario, the lookup confirms that op_42 committed the requested payload, transitioning the database configuration to config_v8. The immutable operation identity binds this remote confirmation to the local dispatch record, resolving the ambiguity even if the original RPC was delayed in transit. If the lookup instead returns an empty record or meets a transient outage, the settlement process cannot conclude that the mutation failed, because the original request might still be queued in the network. In that branch the status stays unknown, preserved through the recovery descriptor as an open obligation of operations_owner_42 rather than lost when the process terminates.
↩︎ Registering the recovery obligation alongside the forward intent, before dispatch, is established in Obligation registration.
Confirming that config_v8 committed settles the mutation but not the incident. The context manager retires the stale config_v7 snapshot and assembles fresh telemetry under live load. Assume that latency percentiles and error budgets over an observation window, health_window_42, meet the thresholds in incident_contract_v1 and that the policy checks pass. An authorized principal or automated gatekeeper then signs acceptance_42, which binds together the original incident contract, the operator approval, the settled operation identifier, the new active configuration version (config_v8), and the validated metric window. Only then is the incident formally closed under contract for the evaluated duration. Monitoring the service after the window passes to operations_owner_42, the same owner who would have held op_42 had its status stayed unknown.
Seen from above, this one incident has crossed nearly every part of the book (figure 1). The adopted contract came first; the loop that diagnosed the regression was assembled from the model call, context, tools, and sandbox of Parts I to III; the harness of Part IV wrapped every step; and acceptance rested on external evidence rather than the agent’s report. Learning and delegation never entered, because neither was needed to close this task.
The audit chain summarized in table 1 enables an independent auditor or site reliability engineer to reconstruct the causal lineage of the incident without relying on a language model’s retrospective self-report. Structured records also expose gaps at once. A successful completion token from a model call cannot settle the network status of op_42, and an unanchored telemetry snapshot lacking a verified observation window cannot validate an acceptance claim.
| Transition | Owner | Record it leaves |
|---|---|---|
| Task admitted | Runtime under the delegated contract | incident_42, incident_contract_v1, limits, and read_scope_42 |
| Evidence selected | Context manager and diagnostic tools | context_manifest_42 with source, version, time, and omissions |
| Change proposed and approved | Model proposes; authorized operator decides | change_request_42 and its bound approval_42 |
| Change admitted and dispatched | Runtime action gate and operation journal | Durable op_42 intent, recovery descriptor, and dispatch record |
| Lost response reconciled | Settlement process for operations_owner_42 |
Source outcome for op_42, confirming config_v8 or preserving uncertainty |
| Incident accepted | Designated accepting principal with runtime checks | acceptance_42, health_window_42, and residual duty assigned to operations_owner_42 |
20.2.2 Adversarial evidence within the same task
The diagnostic reads assembled in context_manifest_42 also expose the trajectory to untrusted text, because production telemetry is rarely sterile. A log entry or issue tracker comment fetched under read_scope_42 can carry an indirect prompt injection, instructions written to exfiltrate database credentials to an external server. The payload enters the agent’s context through an authorized diagnostic read, yet the untrusted text inherits none of the operator’s administrative authority; an agent that acted on it would be the confused deputy of definition 7.3, spending the operator’s grant on the attacker’s intent. While structured provenance tags and XML delimiters help the model separate ingested data from operational instructions, model comprehension is not a security boundary, so isolation outside the model runtime must govern what a schema-valid tool call can execute using legitimate system privileges.
↩︎ Indirect prompt injection, delivered through an observation the agent reads rather than by the interacting user, is established in Prompt injection entry points.
In this incident scenario, an attempted exfiltration collides with layered architectural barriers. If the manipulated model emits a tool call directed to an unauthorized external IP address, the runtime action gate blocks dispatch, because read_scope_42 grants queries only against the named diagnostic endpoints and an arbitrary address is not among them. Independently of that grant, the sandbox’s egress policy drops unsanctioned outbound sockets at the network layer regardless of what code or parameters the agent generates. The read grant says which services the task may ask; the egress policy says which hosts any process may reach. Allowing unrestricted outbound connections would create an exfiltration channel even with local filesystems mounted read-only.
↩︎ Network egress enforcement placed outside the guest’s control boundary is established in Enforcement outside the guest.
The harder case uses only permitted operations. A diagnostic query to an endpoint inside read_scope_42 can carry a database connection string in one of its arguments, and both the tool and the destination are allowed. The dispatch release check of equation 9.3 catches it. The credential carries a confidentiality label whose recipient set excludes the telemetry service, the label propagates into the query string built from it, and the reference monitor rejects dispatch because the recipient is not in that set. Layered constraints thus bound the blast radius of a successful injection, at the gate, at the network layer, and on the data itself.
When the action gate intercepts an unauthorized or malformed action, it appends the rejection to the audit journal as an undispatched intent. An operation that has already crossed into production is a different matter, because terminating the sandbox does not undo an external state change; operations_owner_42 must determine from the audit log which side effects materialized and select the compensating action. If an adversarial prompt leads the model to assert that latency has normalized, the runtime ignores the assertion, since model output never substitutes for telemetry verified across health_window_42. The same ownership boundaries that handle component outages handle adversarial attacks.
The remediation trajectory terminates with a single model and a single runtime controller. Mining the trace for post-training or delegating sub-problems to sub-agents are later, optional decisions (section 20.4.2, section 20.6). Neither optimization alters the authorization invariants required to dispatch op_42 or the empirical evidence demanded to seal acceptance_42.
20.2.3 The principle table
Managing incident_42 draws on responsibilities established across separate chapters. The context window must represent the live configuration, operator approval must bind the exact mutation, and acceptance must verify the operation that actually committed on the remote service. No single mechanism guarantees all three. Table 2 collects the fifteen principles of the book into one reference, each with its governing equation or structural contract and the design consequence it predicts.
The three H·S·A exposures organize the table. Part I prices an isolated model call and states the statistical limits of verification. Parts II, III, and IV build closure for the three exposures a single call cannot manage on its own: state carried across turns, authority over external systems, and a horizon over which errors compound. Part V changes the policy weights without expanding authority, since fine-tuning alters which trajectories the model proposes but grants no new access. Part VI multiplies all three exposures across sub-agents. Invariant closure anchors the table, because it marks the one thing computation cannot certify for itself, the task contract that defines acceptance. The exposures also interact; persisting state over a long horizon requires durable write-ahead logging, and delegation carries state and authority across agent boundaries.
| # | Principle | Part | Primary exposure | Core equation or statement | What it predicts |
|---|---|---|---|---|---|
| 1 | Invariant closure | Introduction | All three (closure) | \(\Pr(\text{success})=(1-\PerStepErrorRate)^{\TurnHorizon}\) for unchecked steps; bounds enforced below the model, acceptance on evidence above it, under a contract only authorized principals adopt | Unchecked reliability decays exponentially in \(\TurnHorizon\); a check turns a detected violating step into a rejected attempt; a truncated payload becomes a diagnosable rejection; no model output relaxes its own acceptance criteria |
| 2 | Output-bound latency | I | None (prices the call) | \(\PerTokenDecodeTime \ge \Parameters\,\ElementStorageSize/\Bandwidth\) per output token when weight reads dominate | Generation time grows with output length; a shorter proposal cuts latency when decode dominates the call |
| 3 | The verification asymmetry | I | Closure evidence | \(\VerifierPrecision=\dfrac{\ValidCandidateRate(1-\VerifierFalseRejectionRate)}{\ValidCandidateRate(1-\VerifierFalseRejectionRate)+(1-\ValidCandidateRate)\,\VerifierFalseAcceptanceRate}\) | More candidates raise coverage, not selection accuracy; at a low valid rate \(\ValidCandidateRate\), a small \(\VerifierFalseAcceptanceRate\) fills the passing set, or the rewarded set under RL, with invalid candidates |
| 4 | Context is a selected working set | II | State | \(\SystemInstructionTokenCount+\ToolDeclarationTokenCount+\TaskStateTokenCount+\ObservationTokenCount\le \ContextWindow-\OutputLimit\) | Every schema or log token displaces evidence; fitting the window does not establish decision quality |
| 5 | Attention state is derived from the token prefix | II | State | \(\KvCacheBytesPerToken=2\LayerCount \KvHeadCount \AttentionHeadDimension \ElementStorageSize\) bytes held per token; reuse ends at the first changed token | One volatile field early in the prompt forfeits reuse of everything after it; placing stable material first where the task and request format permit increases potential prefix reuse. |
| 6 | Authority stays with the source | II | State | Use entry \(\DerivedMemoryEntry\) at generation \(\QueryWorkspaceGeneration\) only if \(\MemoryEntryBirthGeneration(\DerivedMemoryEntry)\le \QueryWorkspaceGeneration<\MemoryEntryDeathGeneration(\DerivedMemoryEntry)\) and access permits | Stale or unpermitted records pass on retrieval score alone; source checks, refresh, or exclusion can lower \(\StaleSupportRate\), while a cache hit alone supplies no freshness evidence |
| 7 | The typed action contract | III | Authority | Dispatch only a schema-valid, authorized call whose precondition the endpoint enforces with the mutation | A read-then-write race stays open until the endpoint checks the expected version at application |
| 8 | Settlement before retry | III | Authority, with horizon | \(\DuplicateEffectProbability=(1-\RequestLossProbability)\,\ReplyLossProbability\,(1-\RequestLossProbability)\) under blind retry | The duplicate rate tracks the reply-loss rate almost one for one; rare per call becomes routine per day across a fleet |
| 9 | Containment beneath the model | III | Authority | \(\text{admit}(\ProposedAccess,\CapabilityGrant,\AccessAttemptTime)\iff\text{authentic}(\CapabilityGrant)\land \ProposedAccess.\text{op}\in \GrantRights\land \ProposedAccess.\text{target}\sqsubseteq \mathcal{O}_\CapabilityGrant\land \GrantCondition(\ProposedAccess)\land \AccessAttemptTime<\GrantExpiryTime\); flows dispatch only if labels permit | Reach is bounded by the envelope, whatever the model complies with; injected text cannot widen a flow label; teardown leaves external effects in place |
| 10 | Trajectory-level encapsulation | IV | Horizon | Admit only if actual + reserved + proposed + finalization reserve \(\le\) allowance; derived grants only narrow | A loop that passes every per-call limit still halts at the dispatch ceiling or deadline; per-child ceilings alone would let \(\WorkerCount\) children spend \(\WorkerCount\) times the budget |
| 11 | Intent before effect | IV | Horizon, with authority | \(\ValidatedProposalEvent\prec \AdmissionDecisionEvent\prec \IntentAcknowledgmentEvent\prec \ExternalDispatchAction\prec \EvidenceRecordEvent\); \(\ExpectedIrreversibleLoss=\sum_\PlanAction \Pr(\text{bad}\ \PlanAction)\,\text{consequence}(\PlanAction)\,(1-\ActionRecoverableFraction)\) | Under complete mediation, a crash before the flush leaves no effect, and after it the log names every operation that needs settlement; past the pivot only forward recovery remains |
| 12 | Causal and statistical release evidence | IV | Closure evidence | \(\TasksPerComparisonArm\approx 2(\SignificanceCriticalQuantile+\StatisticalPowerQuantile)^2\,\PooledAcceptanceProbability(1-\PooledAcceptanceProbability)/\TargetAcceptanceDifference^2\) | Halving the detectable difference quadruples the tasks per arm, so a small suite cannot rank close systems |
| 13 | Verified trajectory post-training | V | None moved | \(\MaskedFinetuningLoss=-\frac{1}{\ActiveTargetTokenCount}\sum_t \PositionLossMask_{t}\log \Policy(\SequenceToken_t\mid \TokenPrefix)\) over admitted targets | A passing task check does not make every action a target; the active-token fraction sets the supervision each trajectory supplies |
| 14 | The coordination tax | VI | All three, multiplied | \(\text{Speedup}(\WorkerCount)=\dfrac{1}{(1-\ParallelFraction)+\ParallelFraction/\WorkerCount+\CoordinationOverhead(\WorkerCount)}\), judged on acceptance, elapsed time, and \(\CostPerAcceptedTask\) | The serial fraction caps speedup at \(1/(1-\ParallelFraction)\); growing overhead makes added agents slower; delegation pays only when its acceptance or deadline gain justifies its cost multiple |
| 15 | Trajectory goodput and the accepted task | VI | All three, priced | \(\CostPerAcceptedTask=\mathbb{E}[\AttemptCost]/\AcceptanceProbability\); \(\ConcurrentTrajectories=\TrajectoryAdmissionRate\,\mathbb{E}[\TrajectoryDuration]\) per resource | Raising acceptance can cut cost more than cheaper tokens; long tool waits raise held sandboxes in proportion, and the mean alone does not give deadline capacity |
These invariants bound every performance optimization. KV cache prefix reuse saves the same compute whether or not the cached text still describes the live service, so a reused prefix holding a stale diagnostic snapshot is cheap and wrong at once; freshness has to come from the source’s read or version contract, never from the cache. A smaller, faster model has economic value only if its acceptance rate and tail latency still meet the service level objective. Post-training and delegation remain elective interventions, justified only by matched comparison and cost-per-accepted-task accounting.
↩︎ Cost per accepted task, measured under a declared contract, checker, and environment, as the unit for comparing configurations is established in Cost per Accepted Task.
Checkpoint 20.1: Reading the principle table as a design tool
The equations in the principle table provide concrete quantitative estimates for sizing and evaluating real architectures rather than theoretical abstractions.
The principle table states what each mechanism guarantees in general. The incident still needs those guarantees written down as the specific terms it runs under: which outcome counts as recovery, which actions are permitted and when, and which evidence closes the task. That is the job of the incident’s task contract.
20.3 Workload Contract Specification
When the latency alert fires for incident_42, the runtime starts a diagnostic workflow, not an open-ended authorization to modify the production database. An unconstrained loop could restore query throughput by disabling secondary index updates or relaxing integrity constraints, trading durability for speed. The deployment therefore grounds the workflow in incident_contract_v1, an adopted task contract that separates the desired service outcome from the mechanisms an agent may use to reach it.
20.3.1 The incident’s five-part contract
Before dispatching any diagnostic query or rollback on db_primary_01, the runtime must know which settings may change, which endpoints may be read, and which metric window closes the ticket. The five contract components bind incident_42 to an identified service, explicit permission boundaries, and named decision authorities (table 3).
↩︎ The five-part task contract governing an agentic workload is established in The five-part task contract.
| Contract field | Incident-specific commitment | Owner or supporting record |
|---|---|---|
| Goal, \(\TaskContractGoal\) | Restore acceptable query latency for db_primary_01 without violating the required error-rate and change-policy conditions; leave unrelated settings unchanged |
incident_contract_v1, adopted by the authorized task owner |
| Environment, \(\TaskEnvironment\) | Operate on the identified live service; treat config_v7 as the expected baseline, not as a promise that the service remains unchanged |
Configuration source and its versioned operation interface |
| Permitted actions, \(\PermittedActions\) | Initially perform diagnostics under read_scope_42; apply the exact proposed settings only after the required current approval and dispatch checks |
change_request_42, approval_42, and the runtime’s action gate |
| Available observations, \(\AvailableObservations\) | Return authorized, bounded diagnostic results with source, version, capture time, and omission information | Tool result records and context_manifest_42 |
| Completion criteria, \(\CompletionCriteria\) | Confirm the authorized operation’s outcome, the required service metrics over the stated window, and the applicable change-policy checks | op_42, health_window_42, and the adopted decision in acceptance_42 |
The goal encodes process constraints alongside target metrics because the validity of an outcome depends on how the system reaches it. Here that dependency takes the form of approval bindings and version preconditions, and the contract places the preconditions at the infrastructure layer. A configuration endpoint without an atomic expected-version check cannot safely host this workflow, however often the runtime re-reads config_v7, because a concurrent write can slip between the final read and the update. Observation bounds constrain diagnostics just as strictly. Under read_scope_42 the agent may query telemetry from designated endpoints but not customer rows, payload logs, or external destinations, and tool output arrives truncated and flagged, with the full output in a separate log sink. Heuristic filters that strip known keys cannot prove an output free of sensitive data or injected instructions, so protection rests on structural boundaries over what the agent reads, where context flows, and which endpoints receive its output.
↩︎ Head-and-tail byte budgets and explicit truncation notices for tool output are established in Bounded output capture.
20.3.2 Exposures and envelope bounds
The introduction placed the remediation agent at a horizon of tens of turns, mutable shared production state (\(\StateExposureLevel_3\)), and changes that cannot always be compensated (\(\AuthorityExposureLevel_3\)), and incident_42 sits exactly there. Each exposure sets an obligation. Diagnosis needs finite allowances for model and tool calls, live service state needs source-aware tracking, and the step from diagnostic reads to a production change needs an explicit escalation of authority. The envelope attached to incident_contract_v1 applies the harness’s trajectory budgets: caps on turns, wall-clock time, observation size, disk, and spend, with the liability of each admitted call reserved in advance and a reserve kept for the health check and recovery.
↩︎ Multi-dimensional budget vectors and the advance reservation of each admitted call’s liability are established in Budgets and Ceilings.
Priced on the remediation workload profile used for this lighthouse in Cost per Accepted Task by Archetype, one incident_42 attempt runs about 10 turns and costs $20.45: $0.15 of model calls, $0.10 of telemetry queries and change tools, $0.20 for the health-window check, and $20 of on-call review for the approval. At 70 percent acceptance, each accepted incident costs $29.21 (equation 1.2). The model accounts for 0.7 percent of the attempt, while operator approval takes 97.8 percent of the money and 50 percent of the 30-minute time to mitigate. This profile sets a threshold for additional diagnosis rather than establishing superiority over a fixed workflow. If all other expected attempt costs remain unchanged, doubling the model-call charges raises attempt cost by its existing model share, 0.7 percent. The added calls lower cost per accepted task only if acceptance rises by a greater relative fraction under the same contract and latency limits; that gain must be measured.
The same profile shows where the danger lies. A wrong change, proposed in 4 percent of attempts and costing $12,000 when it fully commits, carries an expected irreversible loss of $480 per attempt if no check stands ahead of the write (equation 12.5), about 23× the cost of the attempt itself. For this workload, then, the envelope’s caps on turns and spend matter less than its gates: the approval binding, the expected-version check, and the health window that bounds how long a wrong change runs before anyone reverses it. The contract also makes idempotency, request deduplication, and an operation-status lookup prerequisites for admitting any remediation workload, so that a timeout leads the runtime to ask the endpoint for the operation’s state rather than issue a duplicate mutation (section 20.5).
20.3.3 Operating-envelope decisions
Suppose the run reaches its deadline while the status of op_42 is still being reconciled. Extending the deadline because the model predicts imminent success violates the envelope, and marking the operation failed fabricates an unverified outcome. The runtime instead halts further admission, logs the expiration, and preserves the pending operation identifier with its reconciliation quota, while operations_owner_42 remains responsible for the orphaned operation. Staying within the turn and token limits proves nothing about recovery either, because resource accounting and acceptance are separate boundaries: mutating the service, verifying its health, and closing the ticket are distinct milestones that each require their own evidence. Sizing those limits is therefore a trade-off between blast radius and completion rate, since tighter ceilings bound exposure at the risk of halting a viable repair, whereas looser ones grant latitude that requires broader human oversight.
↪ A worksheet mapping each contract condition to its design obligation and the evidence to record is developed in Selecting Responsibilities from the Task Contract.
20.4 State and Evidence Across Boundaries
Before the mitigation, context_manifest_42 selects diagnostic evidence describing config_v7. Once the endpoint confirms that op_42 applied config_v8, any prompt that still presents config_v7 as active state gives the model an incorrect basis for its next decision. The earlier observation remains true, but its role has changed; it no longer describes the live system and serves instead as historical evidence of the trajectory.
20.4.1 Evidence refresh and cached computation
That role change is easy to miss when derived stores and serving caches sit between the database and the next prompt. A cached summary or vector index that still reports config_v7 after op_42 commits misleads the next turn even though the KV cache for the old prompt tokens remains mathematically valid. To keep those guarantees separate, the incident’s records occupy distinct roles in an evidence chain. The configuration service is the source record, the runtime operation log is the event log of decisions, approvals, and settlement status, model-written incident summaries are agent-written memory, and diagnostic vector indices are derived retrieval indexes. The context manifest is none of these; it is the working-context view assembled for the next model call. No derived role can override the source, and the operation log attests only to the events it durably wrote, not to an external effect it has not verified (The Trajectory Log).
↩︎ Source records, event logs, agent-written memory, and derived indexes as roles in an evidence chain are established in Record roles.
When the source confirms config_v8, context assembly updates the active configuration entry, keeps the config_v7 record only where it justifies the remediation step, and invalidates derived artifacts that depend on the superseded version, such as diagnostic summaries and precomputed runbook excerpts. If the retrieval index lags, the runtime issues an authorized direct read, and if no read is possible the manifest records the omission rather than presenting stale data as current.
↩︎ Withdrawing a staged observation’s authority over current state when its source changes is established in Context Invalidation.
The serving system judges validity by a different rule. KV cache tensors computed for tokens describing config_v7 remain valid representations of those tokens whatever the service now holds, so substituting config_v8 evidence ends prefix reuse at the first changed token and leaves the unchanged prefix reusable. Prefix coherence (principle 5) governs computational reuse and source authority (principle 6) governs semantic freshness, and a cache hit establishes only the first. Freshness checks at prompt time also cannot freeze the service while the model deliberates, which is why the endpoint’s expected-version check at application remains the final guard.
↩︎ Reuse of KV state only over an exact token prefix computed under the same weights and execution parameters is established in Prefix compatibility.
Provenance follows the same separation. The runtime records which endpoint supplied each observation under read_scope_42, its timestamp and version, any filtering applied, and the model request that consumed it, and these host-verified events stay distinct from the model’s hypotheses about the cause of the alert. Text read from a trusted store, including an instruction injected into a log stream, gains no authority by being stored there. Permissions are evaluated against capability grants, and observed data remains passive payload.
20.4.2 Cross-subsystem seams and ownership
In the source-to-context example, storage and serving each operate within their local specifications while the agent as a whole acts on stale evidence. That failure pattern recurs at every subsystem boundary the incident crosses. Table 4 names five such seams, the last two of which arise only if the deployment adds learning or delegation.
| Seam | Incident-specific failure | Principles that meet here | Owner and required record |
|---|---|---|---|
| World and its representations | config_v7 is presented as current after config_v8 is confirmed |
Source authority (principle 6) against prefix reuse (principle 5) | Context and retrieval owners preserve source/version bindings in context_manifest_42; the action endpoint enforces the expected version when applying a change |
| Proposal to effect | A lost reply is treated as failure, leading to an unsafe repeat of op_42 |
Intent before effect (principle 11), then settlement before retry (principle 8) | The settlement process and operations_owner_42 retain the admitted intent, operation identity, and unresolved status until the endpoint supplies outcome evidence |
| Recovery and authority | A recorded recovery plan is treated as permission to apply a corrective change to a now-different service | Intent before effect (principle 11) against the typed action contract (principle 7) | operations_owner_42 retains the descriptor and residual duty; any corrective action needs current authorization and applicable preconditions |
| Recorded incident to learning, if chosen | Acceptance of the incident makes every step a training target, or the release grader becomes a selection reward | Release evidence (principle 12) against the verification asymmetry (principle 3) | Curation and evaluation owners separately admit any training record derived from the incident, keep its permitted use and evidence limits, and protect release evaluation |
| Across workers, if delegated | Diagnostic workers duplicate a consumable allowance or their agreement is treated as independent confirmation | Coordination tax (principle 14) against trajectory-level encapsulation (principle 10) | Delegating runtime records scoped assignments and reservations; the integration owner checks returned evidence and retains unresolved child obligations |
Delegation makes the final row concrete without changing the acceptance contract. A worker may receive a subset of read_scope_42, a reusable capability, but a consumable allowance cannot be copied the same way; concurrent workers reserve against one shared budget rather than each receiving an independently spendable quota (principle 10). Agreement among diagnostic workers also cannot substitute for verification, because workers that read the same noisy inputs or each other’s intermediate results fail together. Whether delegation pays at all is weighed against the coordination tax (principle 14) in section 20.6.
With these records in place, an auditor can reconstruct what the runtime observed, what it proposed, what authority it held, and what state the external world committed. Turning that evidence into an admissible next action, whether settling an ambiguous request or dispatching a compensating repair, falls to the execution and recovery boundary.
20.5 Execution Harness Synthesis
The capstone trace resolved its lost reply with a single status query, and it could do so only because every piece of state that query needed was persisted before the network dropped the reply. Three properties made each step admissible: durability that covers the declared failure domain, evidence states the runtime keeps apart, and obligations that outlive the process that created them.
20.5.1 Durable intent before dispatch
The trace wrote the intent for op_42 and its recovery descriptor before the request left (principle 11), binding the operation to its request, approval, endpoint contract, reservation, and accountable owner. Registering the descriptor only after a response would have left exactly the gap the lost reply exposed, an active remote mutation with no durable record of who must reconcile it. The ordering is a database write-ahead log applied to an external effect, with one difference that shapes everything after it. A database’s recovery manager can redo or undo a logged update itself, whereas the runtime can neither redo nor undo the endpoint’s mutation; the log tells it which operation to ask about, and only the endpoint can answer. The acknowledgment must also cover the declared failure domain, since a local fsync does not survive the loss of the storage device, and the execution gate withholds dispatch when that acknowledgment is missing or ambiguous. Recording intent commits the system to track the operation, not to execute it, so a pause or a revoked authority can still withhold the unsent request.
↩︎ Durable intent, the admitted operation acknowledged by storage that survives the declared failure domain before dispatch, is established in Durable Intent Before Dispatch.
20.5.2 Lost responses and unresolved operations
Neither the expired timer nor the surviving intent log can say which of the trace’s three outcomes occurred. Because the agent’s local log and the external configuration service cannot share one atomic transaction, recovery from a lost reply depends on guarantees at the endpoint, and issuing an idempotent status query rather than a fresh mutation is settlement before retry (principle 8). Once correlated evidence confirms that the endpoint applied config_v8, dependent tasks may consume the result, and any later replay of the same operation must conform to the endpoint’s repeat contract and pass admission checks again. Table 5 separates boundary states that naive agent architectures treat as interchangeable tool errors.
| Evidence at the boundary | What the runtime may conclude | Continuation and owner |
|---|---|---|
| Current admission fails before this attempt is released | This attempt was withheld through the gate; earlier attempts are unaffected | Record rejection and its reason; reconsider only under current authority |
op_42 intent survives, but its response is lost |
The external outcome remains unknown | Withhold dependent mutation; operations_owner_42 retains reconciliation and outstanding liability |
Correlated status evidence establishes application as config_v8 |
This operation’s effect is established under the endpoint contract | Record settlement evidence; obtain the service-health evidence required for acceptance |
| Status remains unavailable within the follow-up allowance | Automated reconciliation has not established the outcome | End this continuation as unresolved; retain the operation, evidence, reservation, and acknowledged handoff owner |
Settling the operation does not complete acceptance, which still needs service-health evidence. The operation’s budget reservation also stays open until settlement, because a charge for remote work can arrive after its reply, as the canceled child with a late bill showed in Spending Governance.
20.5.3 Recovery and stopping obligations
In incident_42, the configuration on db_primary_01 is restorable selected state, since the endpoint keeps config_v7 and a further versioned write can return to it, whereas the queries db_primary_01 served while a wrong configuration was live are a commitment that cannot be undone, which is why the health window rather than a later rollback limits the damage of a wrong change. To separate those cases before any mutation leaves the harness, a recovery contract asks three things of every effect before dispatch: its target state, the recovery action available under the task’s authority, and the evidence that would show recovery worked. Even a write that restores config_v7 is, in the vocabulary of the trajectory saga (definition 12.1), a compensating step, a new forward action that observers may see half-applied rather than an erasure of the first. It needs current credentials, approval, and preconditions checked against the live service, the same admission the typed action contract (principle 7) applies to any forward action. Because op_42 was itself a rollback, restoring config_v7 would reinstate the configuration the incident was opened against, so the useful correction after a wrong op_42 is usually a newly diagnosed change. Compensations can themselves stall or fail partially. Rolling back a local workspace cannot reverse an external configuration change, just as a network cancellation request cannot guarantee that the remote endpoint stopped executing. Until an authorized successor acknowledges the handoff, every residual obligation stays with operations_owner_42.
↩︎ Restoring selected state, compensating a known effect, and managing a commitment that cannot be undone are established as recovery contracts in Recovery contracts.
Stopping a run must preserve admitted work in the same way. A semantic watchdog that halts a stalled loop can mistake legitimate polling of a pending status for a livelock, so the harness backs it with a hard dispatch ceiling. A hold then suspends new speculative dispatches, preserves every admitted operation, and lets only separately authorized settlement and recovery work proceed.
↩︎ Semantic watchdogs that detect activity without progress are established in Semantic Watchdogs.
Checkpoint 20.2: Auditing the incident's action boundary
Each prompt names a failure at the action boundary; decide which record or check must catch it.
Because every failure in this trace is anchored to a defined boundary and an accountable owner, the record also shows where the next safeguard belongs. Choosing that safeguard is itself a decision that needs evidence.
20.6 Evidence-Based Systems Intervention
An execution trace supports a more precise question than whether the agent needs a larger model. If context_manifest_42 still presents config_v7 as current after settlement confirms config_v8, the defect lies in state selection. If the runtime blindly retries op_42 after a timeout, the defect lies in outcome handling and the endpoint’s recovery contract. Further training might change how often the model reaches these paths, but new weights cannot supply a missing state transition or make a non-idempotent retry safe.
Intervention selection pairs an attributed failure with a targeted remedy, and the intervention ladder orders the candidates by cost and reversibility. Context and retrieval, tool interfaces, and runtime control come before changes to model weights, which are reserved for residual failures that evidence traces to the model’s behavior rather than to a missing record or check. The ordering is a default justified by cost, not a sequence every failure must climb; an authorization bypass points directly to a runtime enforcement repair.
↩︎ The intervention ladder ordering candidate remedies by operational cost and reversibility is established in The intervention ladder.
20.6.1 Attribution and remedy selection
Placing a failure on that ladder requires more than reading the execution log, because a trace records only the path the run took. To isolate which layer caused a failed run of incident_42, the engineer reconstructs the saved input, tool contract, runtime state, and outcome, then varies one suspected factor at a time in a counterfactual replay: supplying config_v8 in the staged context tests whether stale state misled the model, whereas replacing a generic timeout with an explicit pending status tests whether the tool interface hid an in-flight operation. Once a single change repairs the failure on replay, the candidate fix is evaluated across the affected task suite alongside regression checks on unrelated workloads. Table 6 pairs each kind of intervention with the evidence that justifies it: three reversible structural fixes, two training changes, and delegation, which is a separate architectural decision.
| Intervention | Evidence motivating the change | Obligation the change leaves in place |
|---|---|---|
| Context and retrieval | Needed facts are absent, stale, or obscured in the actual model input | Validate source versions and preserve current task constraints |
| Tool interface | Arguments or outcome states are ambiguous to the caller | Enforce authorization and distinguish a rejected request from an unresolved effect |
| Runtime control | Budgets, action preconditions, or settlement rules are not enforced at the responsible boundary | Test enforcement and recovery under the declared failure model |
| Supervised fine-tuning | Admitted demonstrations represent a recurring behavior the current policy handles poorly | Supply current contracts and evaluate generalization and regressions |
| Reward-driven learning | New attempts can be assessed under a useful scoped reward without a demonstrated response at every decision | Protect assessment, inspect reward gaps, and evaluate held-out behavior |
| Delegated trajectories | Separate ownership and focused contexts may improve a declared task objective | Bound authority and spending, reconcile results, and compare with a capable single controller |
Adapting model weights reshapes which actions the model proposes; it never relieves the runtime of enforcing contracts, budgets, or security boundaries. Supervised fine-tuning imitates demonstrated recovery paths, and reinforcement learning from verifiable rewards optimizes behavior through exploration. Both must show their gains on the deployment distribution, and a sandboxed reward evaluator can still be gamed or miss edge cases of the task.
↩︎ Reward hacking, which exploits the gap between a verifiable reward and the intended task, is established in The score and the intended outcome.
Delegation lies outside the ladder. Partitioning a task across agents changes topology, ownership boundaries, and communication overhead rather than repairing a failed component, and a workload with cleanly decoupled subtasks may warrant it without any training. The baseline is a capable single controller with selective retrieval and parallel tool calls, and the comparison must charge the redundant contexts, serialization across dependencies, and reconciliation that make up the coordination tax (principle 14). Diagnostic sub-agents reading logs in parallel could shorten triage for incident_42, with no worker permitted to apply production changes, but only if the serial share and integration overhead pass the screen of equation 18.1. Diagnosis takes only 20 percent of this incident’s time to mitigate and approval takes 50 percent, so the screen is hard to pass, which is why incident_42 closed without delegation.
↩︎ Benchmarking a multi-agent design against a single controller with matched tools, budgets, and concurrency limits is established in Matched comparison conditions.
An intervention that improves reliability must also pay for itself. A one-time investment in data, training, and evaluation is recovered only after enough tasks have run for the per-task saving to repay it (equation 19.8), and the comparison holds only while both configurations accept the same share of tasks. Choosing an intervention therefore combines attribution, matched comparison, and end-to-end cost accounting, while authority and acceptance stay outside the configuration being changed. Once a remedy is chosen, the remaining question is what evidence supports releasing the changed configuration within its declared envelope.
↩︎ The break-even task volume of an adaptation investment is established in Payback of an adaptation investment.
20.7 Empirical Safety Cases
An incident record links an approved configuration change to an observed service outcome, but the link does not imply general correctness. The engineer must still establish which claims the record supports, which enforcement mechanisms make those claims trustworthy, and which failure modes remain unaddressed. The vehicle for that work is an empirical safety case, an inspectable, structured argument that specific claims hold within a defined operating envelope, as opposed to a certificate that the agent behaves correctly under all reachable conditions.1
1 Goal Structuring Notation (GSN): Kelly and Weaver (2004) presented a graphical notation for safety arguments that represents the claims being asserted, the strategies that decompose them, the operational context and assumptions under which they hold, and the empirical evidence that supports them. Structuring an argument explicitly uncovers claims lacking supporting evidence or unverified operational assumptions, which are the gaps that table 7 is designed to expose.
Each case connects three elements: a claim states a specific operational property such as authorization validity or idempotency, an argument explains how the architecture and runtime mechanisms enforce that property, and evidence backs the argument with audit logs, telemetry windows, regression suites, and operator review traces. As assurance-case methodology emphasizes (Lipson 2013), citing a governance tool or testing practice is not enough; the case must show enforcement, coverage, and relevance to the artifact actually deployed, and it records assumptions and counterexamples so that reviewers can see where the argument stops.
20.7.1 The incident safety case
Showing that op_42 restored latency on db_primary_01 during health_window_42 does not prove that the remediation agent is safe to run on arbitrary database incidents. Table 7 scopes the safety case for incident_42 to the mediated database change and its observed outcome, combining runtime records with the validation suites a production case requires so that each row states what the case must verify rather than asserting that every offline test ran during the live incident.
| Scoped claim | Argument and owner | Supporting evidence | Assumptions and residual uncertainty |
|---|---|---|---|
The mediated change to db_primary_01 requires approval for the exact request and current preconditions |
The action gate checks approval_42 against change_request_42, its target, desired settings, expected config_v7, and validity conditions before dispatch |
The approval and admission records, together with tests of missing, expired, mismatched, and stale approvals | The argument depends on controlled access paths and endpoint enforcement; an unrelated privileged operator or bypass is outside this particular gate’s claim |
| The lost response does not justify issuing a fresh duplicate change | The settlement process preserves op_42 for operations_owner_42 and reconciles through the endpoint’s operation-status and deduplication contract |
Durable intent and recovery descriptor recorded before dispatch; the lost-response status; the later source confirmation that op_42 applied config_v8 |
The endpoint must honor the stated identity, atomic version check, and retention contract, including delayed requests; unavailable status leaves the operation unresolved rather than proving failure |
The incident meets incident_contract_v1 under the recorded acceptance scope |
The acceptance owner checks the applied configuration, required change-policy conditions, and the declared service measurement window | acceptance_42 references op_42, config_v8, health_window_42, required checks, and the authorized adopter |
Measurements cover the named service and window, not every future condition; they do not prove the model’s diagnosis, and post-window monitoring remains with operations_owner_42 |
An agent’s diagnostic report can point engineers to anomalous telemetry or inconsistent logs, but it cannot prove its own successful execution. A recorded operator sign-off counts as evidence only where the contract requires it, and human review has its own failure modes and blind spots, so it needs the same provenance and scope checks as an automated assertion.
20.7.2 Release evidence and staged exposure
The three claims of table 7 concern one execution. Releasing the configuration that handled it is a population question, answered by the staged offline, shadow, and canary evidence of Staged Releases under causal and statistical release evidence (principle 12). In that comparison incident_42 counts as a single trial, and halving the detectable difference between two configurations quadruples the tasks per arm (equation 13.3), so one incident cannot settle a release. Had op_42 stayed unknown at the observation cutoff, that trial would count as unresolved even if latency later recovered, because healthy telemetry on one service does not settle an outstanding effect.
Staged exposure carries the limit the incident showed. Stopping a canary, revoking its tokens, or draining its traffic halts only future dispatches; a request the endpoint has already applied, as it applied op_42 before the reply was lost, persists with its settlement owner. A release safety case therefore documents how partially executed work is contained and settled rather than assuming that rolling back a container image restores world state.
↩︎ Tripwires that stop admission, and rollback that leaves transmitted requests and changed objects in place, are established in Operational tripwires.
20.7.3 Evidence preservation under policy change
An incident trace does not automatically belong in a learning pipeline. If an engineer proposes extracting a segment as training_record_42, the training admission predicates decide whether its context, targets, permissions, and provenance qualify, and an unresolved or aborted trace is not a negative example; labeling it as one biases the learned policy. Validation suites must also stay out of the training loop, because repeatedly selecting policies against the same evaluation set erodes its statistical validity (Dwork et al. 2015). Here the verification asymmetry (principle 3) meets release evidence (principle 12). Once a verification metric becomes a training loss or reward, it loses its standing as an independent assessment. This is Goodhart’s law in the form How search exploits imperfect checks met it under search and The score and the intended outcome met it under training. Optimization concentrates on the verifier’s blind spots, so release evidence must come from checks the optimization never observed.
↩︎ The admission predicates for a training record, covering record fidelity, permitted use, process and outcome conditions, and target suitability, are established in Admission predicates.
Updating model weights or prompts produces a new configuration that needs fresh release evidence, not an endorsement inherited from past incidents. A weight update also invalidates serving state, since KV tensors computed under the old weights are not valid for the new model even when the prefix text is identical. The safety case therefore binds to the exact configuration scheduled for deployment, and the resolution of incident_42 remains an audit of one execution, not a clearance for later versions.
Checkpoint 20.3: Defend the incident's acceptance
Each prompt tests whether a piece of evidence supports the claim it is attached to.
An inspectable safety case makes the acceptance decision transparent and lists its residual risks, including states outside the measurement window, shifting workloads, and unverified edge cases, as explicit design constraints. Naming these limits is not an admission of failure, because a case that names its limits is the only kind a reviewer can check.
20.8 Open Questions Within the Digital Boundary
The safety case for incident_42 establishes a bounded claim, that an approved configuration change produced acceptable service behavior under a declared contract. Even with every policy boundary enforced, every tool call sandboxed, and every transition logged, three systems questions remain open: how completely acceptance criteria capture service health, how to stay consistent when production state changes during deliberation, and how far the audit trail supports causal attribution.
Acceptance criteria are incomplete. The telemetry from health_window_42 satisfies the latency and error thresholds of incident_contract_v1, yet a service can meet those two metrics while exhausting background worker pools, corrupting downstream records, or leaking memory. Rubric-guided reviews can catch some of these regressions, but human and model judges have blind spots of their own. Tamper-proofing a check stops an agent from modifying its own evaluation without making the specification complete. Coverage grows only as unmodeled failure modes are found and written into the contract, and searching many candidate actions against an incomplete check overfits the search to the proxy.
↩︎ Calibrating rubric-based human and model judges, whose shared mistakes survive high agreement, is established in Calibration of qualitative judgment.
Concurrent state drift imposes a second limit, because production services do not pause while an agent deliberates. While the agent fetches telemetry, runs model calls, or waits for approval, traffic shifts, autoscalers adjust capacity, and peer services deploy schema updates. An expected-version check at the write keeps an approved change off a mutated baseline, but a rejected mutation only restarts the loop, since the agent must re-read state, revise its diagnosis, and seek a fresh approval. When churn is fast or deliberation is slow, the agent can livelock, spending its step and token budgets on retries that never land. Designing protocols that balance evidence freshness, lock granularity, and deliberation latency within bounded incident budgets remains an open systems challenge.
Causal attribution is narrower than the execution record, because an audit log establishes temporal order, not cause. That op_42 applied config_v8 just before health_window_42 recorded healthy telemetry does not prove the change resolved the incident; a transient network bottleneck, an upstream dependency, or a client retry storm may have cleared on its own. Controlled comparisons, fault-injection tests, and dependency analysis can support a causal claim where operations allow them.
A second attribution problem arises inside long trajectories, which incident_42 is too short to show. When a trajectory fails only at its final acceptance check, the postmortem faces two distinct questions. The first is locating the earliest observable mismatch, which is a replay problem: where retained inputs and boundary records support reconstruction, comparing the run step by step against a reference execution finds that divergence under a declared equivalence relation, even though divergence alone does not prove which action caused the failure. Showing which step was responsible, meaning that a different choice there would have passed, is a counterfactual problem whose search space is every untaken branch. That counterfactual search space grows as the branching factor raised to the task horizon \(\TurnHorizon\), the same exponent that drives the reliability decay of equation 1.1. It is the credit-assignment problem of Baselines and credit assignment seen from the debugging side, with one terminal verdict and every turn a candidate for blame. With an illustrative four candidate actions per step over a fifty-step task, that space spans \(4^{50} \approx 10^{30}\) paths, far beyond any practical replay budget. Passing checks every five steps can change this arithmetic. If check coverage establishes that the defect occurred after the last passing check, and the window’s state is isolated and safely restorable, the search covers a single five-step window of \(4^{5} = 1,024\) alternatives, about 20 minutes of replay at 1.2 s per check. Without those two conditions, the investigation must keep earlier steps and external changes as possible causes.
↩︎ Replaying a recorded run to its first divergence point is established in Locating a Divergence.
Digital systems, the scope of this book, routinely produce effects that cannot be restored: spent compute and API budgets, third-party financial transactions, messages sent to people, and credentials leaked to external logs. Physical actuation adds sensing, kinematics, continuous dynamics, and hard real-time deadlines beyond this book’s scope, but the line between software agents and robots is not a line between reversible simulation and irreversible reality. Both act across the causal boundary into a world whose state cannot be rolled back by terminating a process, and in both the engineer must catalog what state can be observed, what actions can be controlled, what effects can be restored, and which residual risks escalate to an accountable human principal.
These boundaries define the scope of an automated acceptance decision and the residual duties it leaves behind, and setting that scope is an engineering responsibility. Models and runtimes can propose plans, run diagnostic checks, and optimize candidate actions, but they cannot assume organizational accountability. Deciding which failure modes are acceptable, which invariants must hold, and which operational risks require human oversight remains the non-delegable duty of authorized system architects.
20.9 Essential Complexity and the Specification Boundary
Suppose an incident model proposes shortening the observation window so that database service db_primary_01 can be declared recovered sooner. The model can state the trade-offs, draft the revised acceptance predicate, and identify supporting metrics, yet none of that alters incident_contract_v1. Until the authorized contract owner adopts a revision through the change-management policy, the runtime evaluates the incident against the existing criteria. Helping to formulate a decision is different from holding the authority to make it binding.
Brooks famously separated the essential difficulty of conceptualizing complex software entities from the accidental difficulty of expressing them in a particular programming language or machine environment (Brooks 1987). In incident management, producing a configuration change is the accidental part; deciding which service behaviors count as recovery, and which risks the change introduces, is the essential work. An agent can generate a valid configuration payload while the recovery criteria remain poorly specified, and faster payload generation does nothing to resolve that ambiguity.
The boundary does not confine language models to syntax. An agent can propose requirements, weigh design trade-offs, expose inconsistent assumptions across services, or draft fault-injection tests, all of which is conceptual work, and its value depends on the verifiable evidence and arguments it produces. If a fraction of specification and review remains unaided by models, that work acts as the serial fraction in the Amdahl bound of equation 18.1, here with one human and one model rather than many workers. A task whose specification and review require half of the original effort cannot finish more than twice as fast, however quickly the model produces everything else, so end-to-end turnaround stays dominated by deliberate human specification and review.
The specification boundary applies the task contract of The five-part task contract to revisions of that contract, separating candidate proposal from authoritative adoption. The organization responsible for a service decides who may approve changes to its goals, permissions, and acceptance conditions, and the runtime enforces those policies. An agent gains no governance authority by authoring a new contract, and it cannot enlarge its own permissions or relax its evaluation when a task becomes difficult. This is invariant closure (principle 1) applied to the contract itself; the model can no more alter its own acceptance criteria than certify its own execution trace.
Approval bindings and acceptance records enforce the boundary in practice, because the agent cannot create either by declaring it. An agent that finds a real flaw in the criteria can trigger an authorized review, and the review records which contract governed the attempted execution and which evidence supports any revised claim, so that an unmet condition is never silently replaced by a more lenient one. Even a valuable contract revision proposed by an agent governs production only after it passes that formal adoption process.
The same boundary governs tests. An agent-written test suite has real value even when the model also drafted the implementation, but both can encode the same mistaken requirement. Candidates are therefore judged against the adopted contract, protected regression suites, live observations, and qualified human review, and agreement between a generated candidate and its co-generated tests never counts as independent verification. Across code, tests, and contracts alike, agents can support accountable decisions about bounded tasks, while deciding what those tasks mean remains the responsibility of the system’s owners.
20.10 Fallacies and Pitfalls
Accepting an automated remediation safely depends on keeping four concerns apart: capability, enforcement, evidence, and accountability. Even when the model proposes an accurate diagnosis and a valid fix, each answers a different question, and conflating them lets a locally successful component close the task before the environment has verified the result.
Fallacy: A sufficiently capable model removes the need for explicit runtime boundaries.
Deploying a more capable model improves diagnostic accuracy and produces fewer malformed actions, but training suppresses the rate of violating proposals without driving it to zero. Over multi-step operational horizons, compound probability amplifies whatever residual risk remains, as captured by equation 1.1. Even with a residual violation rate of only 0.1 percent per step, 4.9 percent of 50-turn trajectories generate at least one violating proposal. Across 10,000 daily trajectories in production, that tiny fraction yields roughly 488 hazardous proposals each day. Explicit runtime boundaries determine whether each proposal is safely intercepted or permitted to produce side effects on live infrastructure. In the capstone incident, even an accurate restoration proposal still requires strict approval bindings and version checks; model confidence can never authorize op_42.
Pitfall: Treating passing checks as proof of the entire completion claim.
An accurate check shrinks the invalid share from 70 percent to about one in nine, not to zero.
Observing acceptable service latency does not prove that an approved operation occurred, because external traffic might have simply subsided. Conversely, confirming that a configuration change took effect does not guarantee that the service has returned to healthy operation. A valid completion certificate such as acceptance_42 must assemble evidence of both correct execution and restored service behavior, alongside all mandatory policy checks. Passing a check is also weaker assurance than the check’s accuracy suggests, because under verifier precision (equation 3.3) the base rate dominates. If only 30 percent of proposed candidate actions are valid, a verifier that misses 10 percent of valid proposals while mistakenly passing 5 percent of invalid ones still leaves 11.5 percent of its passing set invalid. This is the verification asymmetry (principle 3) in numbers; high nominal accuracy does not ensure correctness when candidate quality is low. A defensible safety case therefore documents the scope and blind spots of each check and assigns an owner to any residual monitoring.
Fallacy: An intervention is justified because it is the next rung of the intervention ladder.
The intervention ladder organizes mitigation strategies by ascending operational cost and decreasing reversibility, but stepping to the next rung does not ensure that the action addresses the underlying defect. A stale context entry, an unconstrained retry loop, and an inaccurate model diagnosis demand fundamentally distinct remedies, and escalating to a heavier mechanism incurs substantial overhead whether or not it fixes the issue. For example, adapting a smaller model (priced in Payback of an adaptation investment) demands a one-time investment in data, training, and evaluation that is recovered only over a large volume of reviews, and the payback period lengthens in proportion when review volume falls short of plan. That investment yields zero return if the operational failure actually resided in prompt context or the retry policy. Interventions should follow directly from attributed cause, comparative outcome evidence, and total lifecycle costs; model fine-tuning and operational delegation remain specialized tools rather than automatic defaults.
Pitfall: Allowing the component producing a result to redefine the conditions for accepting it.
A model can identify an unstated requirement, draft a new integration test, or propose revising an impractical service-level objective, but it never holds the authority to shorten health_window_42, expand its own permissions, or certify its own completion claim. Redefining acceptance to declare success early also carries a statistical penalty, because halving the observation window halves the metric samples. For approximately independent measurements, this reduction widens the confidence interval on the service latency estimate by a factor of 1.41. The revised claim then rests on weaker evidence at the moment the incident is hardest to verify. Modifying acceptance criteria requires explicit sign-off from the designated system authority, backed by an audit record detailing which contract version governs the work. Delegating automated reasoning or execution to an agent never transfers the ultimate responsibility for accepting the result.
20.11 Summary
The incident connects the book’s components through one concrete task rather than an abstract reference architecture. Because context assembly supplies telemetry and logs with verifiable provenance and freshness, the foundation model can propose a grounded diagnosis and configuration rollback; the action boundary then admits that proposal only after validating its approval and version precondition, while durable logs record the intent before dispatch and track the unresolved outcome across the lost reply until service telemetry and policy checks verify recovery against the contract. Each component satisfies a local specification, and the end-to-end integrity of the system comes from the records, checks, and ownership chains that bind them together. Table 8 summarizes the quantitative relations that govern an agentic execution trajectory, linking the compounding error of unchecked steps to the economics of verification, retry safety, and parallel delegation.
| Equation | What it bounds or decides | Where derived |
|---|---|---|
| \(\Pr(\text{success})=(1-\PerStepErrorRate)^{\TurnHorizon}\) | Success of unchecked turns under the stated error model; sufficient check coverage and isolated, restorable state can confine a diagnostic search to one checked window | equation 1.1 |
| \(\CostPerAcceptedTask=\mathbb{E}[\AttemptCost]/\AcceptanceProbability\) | Charges every rejected attempt to the accepted ones, so raising acceptance can cut cost more than cheaper tokens | equation 1.2 |
| \(\BreakEvenTaskCount=\AdaptationInvestmentCost/(\BaselineTaskCost-\AdaptedTaskCost)\), \(\BaselineTaskCost>\AdaptedTaskCost\) | Task volume at which an intervention recovers its one-time cost; payback time is \(\BreakEvenTaskCount\) divided by task volume per unit time | equation 19.8 |
| \(\VerifierPrecision=\dfrac{\ValidCandidateRate(1-\VerifierFalseRejectionRate)}{\ValidCandidateRate(1-\VerifierFalseRejectionRate)+(1-\ValidCandidateRate)\,\VerifierFalseAcceptanceRate}\) | Valid share of a passing set; at a low valid rate \(\ValidCandidateRate\), a small \(\VerifierFalseAcceptanceRate\) fills the passing set with invalid candidates | equation 3.3 |
| \(\DuplicateEffectProbability=(1-\RequestLossProbability)\,\ReplyLossProbability\,(1-\RequestLossProbability)\) | Duplicate rate of a blind retry after a lost reply; settlement through the endpoint’s status record removes it | Retrying Uncertain Actions |
| \(\ExpectedIrreversibleLoss=\sum_\PlanAction \Pr(\text{bad}\ \PlanAction)\,\text{consequence}(\PlanAction)\,(1-\ActionRecoverableFraction)\) | Expected loss from actions without adequate reversal, which sets how much approval, checking, and compensation an action warrants | equation 12.5 |
| \(\TasksPerComparisonArm\approx 2(\SignificanceCriticalQuantile+\StatisticalPowerQuantile)^2\,\PooledAcceptanceProbability(1-\PooledAcceptanceProbability)/\TargetAcceptanceDifference^2\) | Release tasks per arm needed to detect a difference \(\TargetAcceptanceDifference\); halving \(\TargetAcceptanceDifference\) quadruples the evaluation bill | equation 13.3 |
| \(\text{Speedup}(\WorkerCount)=\dfrac{1}{(1-\ParallelFraction)+\ParallelFraction/\WorkerCount+\CoordinationOverhead(\WorkerCount)}\) | Delegation speedup, capped at \(1/(1-\ParallelFraction)\) and reversed once coordination overhead grows faster than the parallel share shrinks | equation 18.1 |
| \(\ConcurrentTrajectories=\TrajectoryAdmissionRate\,\mathbb{E}[\TrajectoryDuration]\) | Concurrent trajectories, and therefore the sessions, sandboxes, and cached state a workload holds at once | equation 19.4 |
Key Takeaways: Own every transition, bound every claim
- Every consequential transition requires an explicit owner and a durable record: The request, decision, side effect, and resulting evidence must remain structurally linked throughout the trajectory. An unknown outcome is never a license to retry blindly; an ambiguous operation retains its human or system owner rather than fabricating a premature success or failure.
- The contract, not a mechanism catalog, defines system boundaries: Task horizon, mutable state, and delegated authority dictate where a workload is vulnerable, and the formal contract determines which guardrails to deploy. Every telemetry record and intermediate artifact retains its provenance, authority level, and expiration window, ensuring that stale cached data never masquerades as fresh environmental evidence.
- Interventions must follow attributed evidence, prioritizing the lowest-cost remedy: Prefer lower-cost, reversible remedies when they address the attributed failure; initiate fine-tuning or model changes only when evidence supports them. Compare candidate mitigations under identical task constraints while accounting for development, inference, verification, and recovery costs. Model fine-tuning and multi-agent delegation must earn their keep through measurable end-to-end gains rather than architectural ambition.
- Claims must remain bounded by verifiable evidence, while ultimate adoption remains with human principals: A passing verification check supports acceptance strictly within its specified domain, with residual uncertainty and environmental assumptions made explicit. Models can synthesize specifications, generate test cases, and draft operational contracts, but designated human owners must ratify the governing contracts and acceptance procedures, and they retain accountability for whatever residual risks remain.
Unchecked success decays exponentially as the horizon grows.
This book opened with two foundational pieces of arithmetic, and every subsystem analyzed across these chapters addresses one of them. The first is horizon compounding (equation 1.1). With a per-step correctness rate of 99 percent, an unchecked trajectory succeeds 90.4 percent of the time across 10 turns, but falls to 36.6 percent across 100 turns. Post-training can reduce how often a policy proposes an invalid action, but it cannot drive violations to zero, so invariant closure (principle 1) stays anchored in mechanisms outside the model at any scale. Admission and verification turn detected violations into rejected attempts, typed action contracts intercept unsafe proposals before they take effect, and watchdogs limit how far an error propagates across turns. Durable intent logging and scoped compensation (principle 11) bound the cost of recovery, rolling back sandbox changes directly and compensating external effects through explicit forward operations.
Raising acceptance cuts cost per accepted task more than halving token price.
The second governing relation is the cost per accepted task (equation 1.2), which charges the compute and operational overhead of every rejected attempt to the few that are accepted. At $2.10 per attempt and a baseline 40 percent acceptance rate, each accepted task costs $5.25. Cutting the raw token price in half reduces this effective cost to $3.63, whereas improving task acceptance to 60 percent drives the cost down further to $3.50. The four lighthouses stress different terms of that balance, as Cost per Accepted Task by Archetype priced them. Sandbox holding and tools dominate the repair agent’s cost, research spends most on tokens through its sub-agent fan-out, and support spends most on escalation handoffs while its latency target needs separate capacity checks. Remediation spends most on human approval, and incident_42 showed why, since approval is most of each attempt’s cost and a wrong change costs far more than any attempt. In each case the intervention is chosen by pricing the work needed to improve the outcome, not by finding the largest cost term. Across all workloads, verification asymmetry (principle 3) keeps the denominator honest, because an attempt counts as accepted only within the explicit bounds validated by an authorized check.
This book began by distinguishing a proposed action from its realized side effect, and a plausible generation from an accepted task. In incident_42 that distinction ends in an auditable decision. acceptance_42 binds the governing contract to the settled change, the passing checks, and scoped telemetry, while operations_owner_42 remains responsible for the service after the window, and evidence that cannot be verified leaves an unresolved disposition rather than a fabricated completion. Each common failure of agent deployments meets a specific guard in this trace. A duplicated change from a blind retry meets the operation identity and settlement query, an incident closed before its change applied meets the acceptance record, and an untracked production change meets the intent log and its accountable owner. An audit trail of what the model proposed, what the action boundary permitted, what effects occurred, what evidence validated the result, and who remains responsible is the real deliverable of an agentic machine learning system.
Every mechanism in this book answers the same limitation, that the model samples stochastic outputs and cannot certify its own correctness. No advance in deep learning removes that uncertainty. What remains to engineer is the surrounding system, the harness and runtime that make a model’s proposals safe to execute, verifiable to accept, and economical to operate at scale.
20.12 Historical Perspective and Further Reading
The engineering challenges examined throughout this book did not begin with modern foundation models, because computer systems researchers have wrestled with authority, state coordination, and verifiable delegation for decades. Operating systems encountered the problem of authority first, when Dennis and Van Horn introduced capabilities to bind access rights directly to unforgeable tokens (Dennis and Van Horn 1966), establishing that components should operate only with explicit, unforgeable references. Saltzer and Schroeder formalized these protection mechanisms into durable design principles (Saltzer and Schroeder 1975), notably least privilege and complete mediation, which govern how an action gate inspects, validates, and filters tool invocations before side effects reach an external environment. Database systems confronted the complementary problem of mutable state and consistent outcomes. Gray established the classical transaction model by pairing write-ahead logging with atomic commit protocols and crash recovery (Gray 1978). Gray also recognized the limits of strict ACID guarantees, however, when coordinating long-running work across autonomous organizations (Gray 1981). That insight led directly to sagas (Garcia-Molina and Salem 1987), which replace distributed locks with sequences of local transactions coupled with compensating steps when an effect cannot be undone cleanly. Saltzer, Reed, and Clark formulated the end-to-end argument (Saltzer et al. 1984), establishing that correctness and final verification must ultimately reside at the application endpoints that understand the semantics of the task. Each domain met the same vulnerability, in which individual components succeed locally while the composite system loses track of global state or fails to verify completed work.
Artificial intelligence and multi-agent systems research approached the same boundaries from the side of autonomous decision making and structured delegation. Minsky identified the credit-assignment problem (Minsky 1961), highlighting how difficult it is to determine which intermediate action in an extended horizon caused an ultimate success or failure, a challenge that continues to complicate trajectory optimization and policy evaluation. Smith developed the contract net protocol (Smith 1980), organizing distributed problem solving through an explicit cycle of task announcements, bids, and awards among cooperating nodes. Wooldridge and Jennings characterized intelligent agents through autonomy, reactivity, goal-directed proactiveness, and social ability (Wooldridge and Jennings 1995), defining the behavioral boundaries that separate goal-seeking entities from static programs. In parallel, human-factors research established how to partition control between humans and automated systems. Parasuraman, Sheridan, and Wickens demonstrated that system designers must distinguish which cognitive functions are automated from the level of autonomy granted across each stage (Parasuraman et al. 2000). That operational distinction underpins modern approval gates, where a runtime permits a model to propose a sensitive action while withholding the authority to execute it without operator confirmation.
Modern machine learning systems joined this systems lineage when Sculley et al. (2015) demonstrated that the trained model represents only a small nucleus within a vast, complex harness. Most of the ongoing maintenance debt in production ML stems not from model weights, but from fragile boundary code, configuration debt, and hidden feedback loops. When large language models demonstrated the ability to follow instructions in context (Brown et al. 2020), they provided systems designers with a flexible, general-purpose proposer. Interleaving reasoning traces with concrete actions (Yao et al. 2023) and training models to invoke external tools (Schick et al. 2023) closed the execution loop, transforming passive text predictors into runtime engines that effect change across real environments. With that transition, classic security and reliability hazards reappeared immediately. Indirect prompt injection (Greshake et al. 2023) is the classic confused deputy problem (Hardy 1988) arising in a semantic medium, where an untrusted input misleads a privileged agent into misusing its authority on behalf of an attacker. Evaluating autonomous workflows, assigning credit across multi-step execution traces, and developing defensible safety cases remain formidable engineering challenges. Lasting progress will come from grounding agent designs in classical systems principles supported by empirical evidence.
Five foundational works trace this lineage from classical systems theory to dependable agent design:
- Saltzer et al. (1984) articulates why reliable verification must ultimately reside at the endpoints where application semantics are fully understood. In agentic architectures, it provides the core justification for why task acceptance and completion checks belong to the task owner rather than intermediate execution layers.
- Gray (1981) examines the transaction abstraction alongside its operational limits in decentralized, long-running workflows. It clarifies which atomicity and consistency guarantees an agent runtime can inherit directly, and where it must substitute compensation, idempotent retries, and asynchronous settlement.
- Brooks (1987) distinguishes between the essential complexity of specifying conceptual structures and the accidental complexity of implementing them. This distinction clarifies what model generation can streamline, such as boilerplate synthesis and API translation, and what remains hard systems engineering: defining crisp specifications, verifying invariants, and validating task contracts.
- Sculley et al. (2015) analyzes deployed machine learning through the lens of systems engineering, demonstrating how glue code, configuration drift, and hidden feedback loops dominate operational debt. Its cautionary analysis applies with even greater force when models shift from passive prediction to active tool invocation across production infrastructure.
- Kelly and Weaver (2004) introduces the Goal Structuring Notation (GSN) for constructing rigorous, auditable safety arguments. It illustrates how to link an explicit engineering claim directly to empirical evidence, contextual boundaries, and underlying assumptions, the discipline required when validating agent deployments and establishing operational acceptance.
20.13 Exercises
Every given below is a hypothetical problem input, not a measurement of a particular model, service, or deployment. The problems combine mechanisms from several parts of the book, and each asks for numbers and then for the design decision those numbers support.
Spacing checks and checkpoints on a long repair (Basic). A repair agent runs a 60-turn trajectory of 15 s turns, and each turn independently proposes a violating step with probability \(\PerStepErrorRate = 0.005\). The runtime can insert a check every \(\CheckWindowTurnCount\) turns that catches any violation in its window. Each check also saves a durable checkpoint of the sandbox (Checkpoints), which takes 3 s, and a caught window is restored from the previous checkpoint and rerun. Operations wants at least 95 percent of windows to pass on the first attempt. When a window fails, finding the responsible step means replaying the alternatives inside it: the agent chooses among 3 candidate actions per turn, each replayed path takes 0.5 s to check, and the attribution budget is 10 minutes.
Use equation 1.1 to compute the probability that the whole trajectory runs free of violations with no intermediate checks, and the per-turn rate a better model would need to lift it to 95 percent. Find the largest \(\CheckWindowTurnCount\) that meets the first-pass target and the largest \(\CheckWindowTurnCount\) that fits the attribution budget. Which constraint binds, and how many checks does the trajectory need? At that spacing, compute the expected wall-clock time including checkpoint saves and reruns, and compare it with rerunning the unchecked trajectory from scratch until it passes. Turn 40 pushes a branch to a shared remote, an effect the sandbox checkpoint cannot restore. Using intent before effect (principle 11) and the commitment region of The commitment region, explain where a check must sit relative to that turn, what the runtime must record before dispatching it, and why the restore-and-rerun arithmetic above no longer covers a violation found after it.
Where the accepted-task cost goes (Basic). A repair attempt runs 30 turns. Each turn reads 8,000 input tokens at $2 per million and generates 400 output tokens at $10 per million, and each attempt also pays $0.25 of sandbox time, $0.15 of test runs, and $1.00 of human review. Half of the attempts are accepted.
Compute the attempt cost and \(\CostPerAcceptedTask\) from equation 1.2. Two changes are proposed. Prefix caching bills each turn’s reused prefix at one-fifth of the input price, and without other changes the reused prefix is 7,200 of the 8,000 input tokens. A context fix adds a 2,000-token block of current test output to every turn and raises acceptance to 65 percent. Compute \(\CostPerAcceptedTask\) for each change alone. Then compute it for both together, with the block placed after the stable content so that the 7,200-token reused prefix survives and the block is billed at the full input price. Using principle 5, explain why the two changes compound only under that placement (Exercise 7 prices the alternative), and which term of the attempt cost neither change touches.
Pricing a blind retry (Basic). A fleet of remediation agents issues 25,000 mutating configuration calls per day. Requests are lost with probability \(\RequestLossProbability = 0.001\) and replies with probability \(\ReplyLossProbability = 0.004\), and a duplicated change costs $2,000 in expected damage. An operation-status lookup costs $0.01.
Compute the expected number of timeouts per day and, using the settlement relation of Idempotent actions, the expected duplicates per day under a blind retry with a fresh identity, and their daily cost. Compare that cost with a status lookup on every timeout. An endpoint offers neither deduplication by operation identity nor a status lookup. Using the contract of section 20.3, decide whether the task can be admitted against that endpoint and what the runtime must do after a lost reply.
Serving memory for a support fleet (Intermediate). This problem transfers the chapter’s whole-trajectory sizing to the support agent, whose binding constraint is serving capacity under concurrency rather than the authority that binds the remediation agent. Sessions arrive at 20 per second and last 180 s on average, mostly waiting on the customer. Each session makes 6 model calls of 2 s each, every call reads a 6,000-token context, and attention state costs 0.1 MB per token. The pool has 1,000 GB of accelerator memory for attention state, prefills 400,000 tokens/s in aggregate, and moves 200 GB/s between host memory and accelerators.
Use equation 19.4 to compute the concurrent sessions and the attention state needed to hold every session resident, and compare it with the pool. Compute the state held by sessions inside a model call. Evaluate two policies: evicting a session’s state between calls and recomputing it at the next call, and keeping it in a host-memory tier so that only about 500 new tokens per call are prefilled. For each, compute the load on the binding resource. Choose a policy, and state which workload and fleet measurements are needed before concluding that the remediation agent would encounter a different binding constraint.
Buying release evidence for an adapted model (Intermediate). A team proposes an adapted model for a repair workload that lowers the cost per assigned task from $1.20 to $0.80. Development and training cost $6,000, each release-evaluation task costs $3 per arm, both arms run the same number of tasks, and the workload runs 10,000 tasks per month. Acceptance is about 70 percent. The release test uses \(\SignificanceCriticalQuantile = 1.96\) and \(\StatisticalPowerQuantile = 0.84\).
Use equation 13.3 to compute the tasks per arm needed to detect a 5-point and a 2.5-point acceptance difference, and fold each evaluation bill into \(\AdaptationInvestmentCost\). Use equation 19.8 to compute \(\BreakEvenTaskCount\) and the payback in months for each. Leadership requires payback within 6 months. Which detectable difference can the team afford? A 3-point regression can hide inside the 5-point test; compute \(\CostPerAcceptedTask\) for both configurations under that regression and decide whether the cost case survives it. Unlike the code-review notebook in Payback of an adaptation investment, the evaluation bill here depends on the decision the team wants to defend; explain why.
Delegating a research question under a deadline (Intermediate). A research agent answers a question in 40 minutes as a single controller, at a cost of $4.00. The analyst needs the answer in 20 minutes. Of the work, 80 percent splits across workers, and coordination overhead grows as \(\CoordinationOverhead(\WorkerCount) = 0.02\,(\WorkerCount-1)\) of the single-controller time. A design with \(\WorkerCount \ge 2\) workers costs $4.00 plus $0.60 per worker for repeated context and result integration. The question’s allowance is $10.
Use equation 18.1 to find the smallest \(\WorkerCount\) that meets the deadline, the \(\WorkerCount\) that maximizes speedup, and the cost multiple of each over the single controller. A proposed runtime gives each worker its own $10 ceiling. Compute the worst-case spend at the chosen \(\WorkerCount\) and state the admission rule of principle 10 that prevents it. Three workers agree on a source’s claim after reading the same page. Explain why that agreement is weaker evidence than three independent checks, and what the integrating controller must record before accepting it.
Where the fresh evidence goes in the prompt (Intermediate). A remediation trajectory runs 20 turns. Its context starts at 20,000 tokens and grows by 1,000 tokens per turn. A 500-token status block, refreshed from the source on every turn, sits at token position 2,000, right after the system instructions. Uncached input costs $2 per million tokens, a reused prefix costs $0.20 per million, and prefill runs at 10,000 tokens/s.
Using principle 5, compute the tokens recomputed on turns 2 to 20 with the block at position 2,000 and with the block moved to the end of the prompt, where each turn recomputes only the block and the new turn’s tokens. Compare the input cost and the average per-turn prefill time of the two layouts. The team then proposes to stop refreshing the block and rely on the cached copy, since a cache hit shows the text is unchanged. Use principle 6 to decide whether the saving is admissible, and state which check at the endpoint still protects the change if the status block is stale.
Sizing a whole remediation trajectory (Advanced). A remediation service runs 400 attempts per day, and 80 percent are accepted. An attempt spends $0.60 on 12 model turns, $5.00 on operator approval, and $0.40 on the health-window check. Three failure paths remain.
- Each turn proposes a violating action with probability 0.01; the action gate rejects 98 percent of them, and an escaped violation causes $20,000 of damage, of which compensation recovers 75 percent.
- In 5 percent of attempts the baseline changes before dispatch. Without an expected-version check the stale change applies, causing $3,000 of damage with 50 percent recovered; with the check the change is rejected and redone at a cost of 4 turns and a new approval.
- Each attempt makes one mutation, and its reply is lost with probability 0.005. A blind retry duplicates the change, causing $10,000 of damage with 50 percent recovered; a settlement query costs $0.01.
Use equation 12.5 to compute the expected loss per attempt from each path with all mechanisms in place, then fold it into the attempt cost and compute \(\CostPerAcceptedTask\). Recompute \(\CostPerAcceptedTask\) with the gate, the version check, and settlement each removed in turn, and rank the mechanisms by what they save per day. With every mechanism present, which term dominates the cost per accepted task? Find the gate miss rate that brings that term below $1 per attempt and the resulting \(\CostPerAcceptedTask\). Compare the model’s token spend with the expected loss, and explain what the comparison says about where engineering effort on this lighthouse belongs.
Critiquing a research system end to end (Advanced). A team deploys a research agent that answers 20,000 analyst questions per day. Its design is as follows.
- A planner fans each question out to 8 workers, each granted its own copy of the question’s $12 allowance.
- Workers fetch 40 pages in total per question and concatenate them into the planner’s context without source labels, and the planner holds a
send_reporttool that can address any email named in its context. - An answer is accepted when 3 of the 8 workers agree.
- A retrieval cache keyed by URL never expires its entries.
- Every night, each accepted trajectory becomes a supervised fine-tuning target, and the acceptance grader is reused both as the reward for reinforcement learning and as the release evaluation.
- After 24 hours, a generated summary replaces the raw trajectory log.
Assume that 1 page in 500 carries an injected instruction, the planner acts on 2 percent of the injected instructions it reads, and each disclosure costs $5,000. Each question produces one final answer candidate. Before grading, 60 percent of candidates are valid, with \(\VerifierFalseRejectionRate = 0.10\) and \(\VerifierFalseAcceptanceRate = 0.20\). Compute the expected disclosures and their daily cost, the worst-case spend on one question against its allowance, and, by equation 3.3, the invalid share of accepted answers and the invalid answers accepted per day. Map every design choice to the principle of table 2 it violates, rank the defects by expected harm, and propose a redesign that names the owner and the record at each seam of table 4. Defend which two changes ship first, and identify one defect that no number above prices but that would still block a safety case.