4 Context Engineering

Purpose

Agent stack focus for this chapter.

Why can an agent lose the thread of a task even when its entire history still fits?

An agent’s context window is a finite and expensive resource, not an unbounded repository for raw text. Naively accumulating tool outputs, conversational turns, and retrieved snippets exhausts token capacity, inflates prefill latency, and spreads attention across material the next decision does not need. Fitting is not the same as being usable. A request that fits can still place a refuted hypothesis beside the test that refuted it, or present a result from an earlier revision as a fact about the current one. Effective context engineering treats each request as a deliberately selected working set, in which relevant state is compacted without losing its provenance, superseded observations are withdrawn, and bulky artifacts stay in durable storage behind handles. Stale or truncated evidence that does reach the model produces errors that compound across later turns. The systems challenge is to carry an accurate, minimal projection of the environment within the token budget, so that the state an agent carries bounds its uncertainty rather than compounding it.

Learning Objectives

• Distinguish nominal request capacity from usable working context and explain why admitted tokens dilute needed evidence
• Allocate the next request’s input budget while preserving output headroom and task-relevant evidence
• Design a context layout that orders entries by change rate for prefix reuse and binds each observation to its evidential scope
• Compare compaction policies by information loss, recovery requirements, and the cost of repeated transformations
• Audit a progress record against runtime-owned sources and quantify how unchecked per-round constraint loss compounds across compaction cycles
• Apply dependency-aware invalidation without treating historical observations as proof of current state
• Evaluate context policies by decision quality, accepted outcomes, and total resource use under common limits

4.1 Carrying a Repair Across Turns

The repair agent (Four lighthouse workloads) has spent several turns investigating a connection-pool defect. It has inspected the pool implementation, executed a reproduction test, and revised its initial diagnosis, and deliberation has left behind candidate versions, a refuted timeout hypothesis, test observations, and the dependency records that tie them together (Revising Candidate Plans). None of that progress lives inside the model. Because foundation models are stateless functions across turns, every call begins and ends with the payload assembled by the host runtime. The agent can use the test trace, honor public API constraints, and avoid revisiting a rejected hypothesis only if the runtime includes them in the upcoming request, structured so that the status and role of each artifact remain unambiguous.

Working context governs the within-session state exposure by selecting, representing, and refreshing what each call carries.

The token accounting of Tokenizing the Request settles two questions for an individual request: whether the payload satisfies the admission bound, and what its tokens cost (The Cost of a Model Call). Those bounds, however, do not determine whether the prompt helps the model make a sound decision. A prompt can fit well within token limits and still fail to guide the agent toward a valid repair. The simplest agent architectures evade this problem by treating context as an append-only log, concatenating every observation, tool execution, and partial response into an expanding transcript. That policy breaks down almost immediately. Even while the transcript fits within physical limits, the runtime pays repeatedly for obsolete tokens that the next decision no longer requires. Stale code snippets sit alongside their replacements, and refuted hypotheses linger alongside the test results that disproved them, diluting the model’s attention. When an extended trajectory inevitably exhausts the context window, an unmanaged runtime is forced into blind truncation, evicting vital instructions or early constraints without any principled criteria for what to preserve.

↩︎ The quadratic input cost of resending an append-only history on every call is established in How multi-turn history drives up input cost.

The agent’s working context is the runtime subsystem that decides what each model call receives. Rather than treating context as a passive transcript, this subsystem constructs a bounded working set tailored to a single decision. It establishes token budgets for each distinct role in the prompt, arranges data layouts so unchanged prefixes can be reused across turns, compacts historical evidence without distorting the facts, maintains structured progress records that resist summarization drift, and enforces invalidation rules whenever external actions alter environment state. The host runtime owns assembly, token budgets, records, and execution gates. Selection and compaction can use heuristics or model-generated candidates, which the runtime must check against the task’s requirements and retained evidence before admission.

The chapter follows the repair agent through its next decision, and each section answers the question the previous one leaves open. The runtime first decides which evidence the next step needs and how the input allowance divides among competing roles (section 4.2). Admitting that evidence does not mean the model can use it, so section 4.3 separates the window an interface accepts from the context a model reliably acts on. The selected items then need a layout that keeps their sources distinct and their stable prefix reusable across turns (section 4.4). When history outgrows the budget, compaction (section 4.5) and audited progress records (section 4.6) shorten it without erasing causal evidence or refuted hypotheses. When the agent’s own edits change the workspace, invalidation (section 4.7) withdraws the observations that no longer describe it. Evaluation (section 4.8) then asks how to show that a complete policy lowers the cost per accepted task.

4.2 The Working-Set Decision

As our connection-pool investigation unfolds, the runtime accumulates far more information than was present at initialization. A controlled test has shown that a connection returned to the pool while a waiting client remained blocked, so the initial timeout hypothesis has lost support and missed notification is now the leading candidate explanation. The next model call must build on these findings. If the runtime replays the entire conversational history without distinguishing raw observations from superseded hypotheses, it forces the model to weigh discounted failure modes alongside the evidence that disproved them.

Consider the codebase at version r17, where the relevant files are src/pool.py and tests/test_pool.py, and the failure is captured by the reproducer test_waiter_notified_on_return. Before prompting the model for a candidate patch, the runtime must decide which source excerpts, test results, interface constraints, and open questions belong in the upcoming prompt. This choice is the working-set decision, the runtime’s selection of the bounded subset of task information needed to run the next concrete step. Making that choice is the first obligation of treating context as a selected working set (principle 4).

The term draws on Denning’s classic working-set model of program behavior (Denning 1968). Over any time window, a running program references only a fraction of its total address space, so an operating system avoids thrashing by keeping those active pages in physical memory while leaving dormant pages on backing storage. An agent runtime faces a similar residency problem when it stages tokens in context to support the next step without exhausting its prompt budget, yet one structural difference separates context engineering from virtual memory. Denning’s working set is measured from page references the program has already issued, whereas an agent runtime must assemble the context working set before calling the model, predicting what the upcoming decision will need.¹ Moreover, standard foundation model interfaces report only generated tokens and emit no hardware-style reference bits to show which staged context tokens the model actually relied on during inference.

¹ Optimal Replacement (MIN): Belady (1966) evaluated page-replacement algorithms against an optimal offline baseline that evicts the page whose next reference lies farthest in the future. Practical operating systems approximate this unattainable ideal using historical access patterns such as least recently used (LRU). An agent context runtime lacks even that access history. The model interface reports no reference bits for the tokens it processes, so working-set selection must rely on explicit domain heuristics and structured task state rather than on observed cache reuse.

Belady, Laszlo A. 1966. “A Study of Replacement Algorithms for a Virtual-Storage Computer.” IBM Systems Journal 5 (2): 78–101. https://doi.org/10.1147/sj.52.0078.

In context management, relevance to the next step must govern selection rather than chronological recency. Core user constraints set at the start of a session remain binding after dozens of intermediate tool calls, whereas a lengthy test log produced moments ago may contribute only a single exit code. Candidate information also comes from well beyond the dialogue history: system instructions, tool declarations, newly read source files, and retrieved reference documents (Lewis et al. 2020) all compete for space in the assembled request. The runtime must build a working representation from these sources without letting any staged copy outrank its origin, because authority stays with the source (principle 6). The environment remains the only authority on system state, and runtime policy, not the model, decides which actions are permitted.

To organize this selection, the runtime builds a context manifest, a structured inventory that lists each item chosen for the upcoming request and why the next step needs it. The manifest for the connection-pool repair’s next request (table 1) reads like a specification rather than a raw byte stream, naming each entry and the exact content it supplies. Code snippets and test logs keep their source and version labels, and their contents enter the prompt only where the next decision requires them.

Table 1: Selecting the next request: Each entry supplies a distinct part of the illustrative connection-pool decision. Version labels identify the example’s artifacts, not measured repository revisions.

Selected entry	Content needed for the next step
Goal and constraints	Repair waiter notification; preserve the public API; remain within the approved workspace; do not deploy to production.
Current artifacts	Relevant source and test excerpts from src/pool.py and tests/test_pool.py at r17, together with the configuration used by the controlled test.
Observed result	On r17, the controlled waiter/return test recorded an available connection while a waiter remained blocked. Keep the test identity and conditions attached to the result.
Hypothesis status	The timeout explanation has lost support. Missed notification remains a hypothesis, not an established cause.
Repair and check status	No candidate patch has yet been checked. Behavior outside the controlled reproducer remains unresolved.
Next bounded step	Propose a candidate change to the notification path using the current excerpts; do not report the defect as repaired.

Building this manifest requires careful filtering. Keeping a failure log without tying it to its source commit (r17) can mislead later turns into reading an old error trace as evidence against a newly patched file, just as recording a short label like “notification bug” without noting its uncertainty turns a working hypothesis into a settled root cause. Conversely, keeping the full dialogue of rejected theories wastes space once the runtime records which test refuted the timeout hypothesis. While a foundation model can help summarize earlier turns to draft this manifest, the runtime must check those candidate summaries against recorded environment observations before admitting them into the prompt.

The active working context must also stay separate from durable records in external storage. Rather than inlining a long test log into the prompt, the runtime can supply a raw-payload handle, a short reference to an external record that an authorized tool can read on demand under access and retention rules. This design enforces the handle rule: once a payload moves behind a handle, it re-enters the working context only through an explicit, checked read. If a later step needs a specific stack-trace line or configuration value omitted from the initial prompt, the model calls a retrieval tool to bring that slice into context.

Every working-set candidate must fit within the endpoint’s context admission bounds. When an endpoint shares one context window across inputs and outputs, the serialized prompt length \(\SequenceLength\) must satisfy \(\SequenceLength \le \ContextWindow - \OutputLimit\), where \(\OutputLimit\) is the token budget reserved for generation; when providers enforce separate input and output limits, \(\SequenceLength \le \ContextWindow\) applies directly. The runtime must count tokens on the fully formatted wire request (system envelopes, chat delimiters, and JSON tool schemas included) rather than summing isolated strings, because tokenizer merges and framing overhead can easily erase the savings of a shorter evidence block.

↩︎ The admission bound and the output headroom \(\OutputLimit\) it reserves are established in Output headroom.

While this inequality sets the overall ceiling, the runtime must divide the available input space across the four roles in the manifest:

\[ \SystemInstructionTokenCount + \ToolDeclarationTokenCount + \TaskStateTokenCount + \ObservationTokenCount \le \ContextWindow - \OutputLimit. \tag{1}\]

Here \(\SystemInstructionTokenCount\) counts the static system instructions, \(\ToolDeclarationTokenCount\) counts API schemas and tool declarations, \(\TaskStateTokenCount\) counts active constraints, progress records, and staged source or test excerpts, and \(\ObservationTokenCount\) counts recent tool outputs and user steering. Because all four terms share a fixed ceiling, growth in any one term squeezes the others. A runtime that appends every raw tool output² lets \(\ObservationTokenCount\) grow on every turn until command-line logs crowd out \(\TaskStateTokenCount\), the constraints and code excerpts the next decision needs. Whereas appending lets observations expand until they displace task state, selection holds each role’s share steady by moving older results into external records that a handle can still reach (figure 1).

² Interleaved Observations: ReAct prompting interleaves reasoning traces, tool actions, and observation payloads within an ever-expanding prompt transcript (Yao et al. 2023). While keeping full execution history visible simplifies early prototyping, it carries a steep cost, because the unchecked growth of \(\ObservationTokenCount\) consumes prompt capacity and raises input token costs on every subsequent call.

Yao, Shunyu, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. “ReAct: Synergizing Reasoning and Acting in Language Models.” International Conference on Learning Representations (ICLR).

Equation 1 is a planning budget rather than an exact identity, because sub-word tokenization across joined strings does not equal the sum of separately tokenized parts. The runtime must therefore check the assembled wire request against the provider’s limit, and when the request runs over, the budget partition tells the runtime which share (usually the observation buffer or older history) to trim or compact first. Prompt staging (section 4.4) then orders these same four components by how often they change so the serving engine can reuse their cached prefix.

Figure 1: Selection Protects the State Budget: Each bar is one serialized request, stacked by the four terms of equation 1 beneath the input limit \(\ContextWindow-\OutputLimit\), with the output reserve \(\OutputLimit\) hatched above it. Appending every tool result (A) lets \(\ObservationTokenCount\) grow until it displaces the task state the next decision needs. Selection (B) keeps each term’s share by moving older results into retained records outside the request, where a handle can reach them if a later decision needs them. Segment heights are schematic, not measured token counts.

Napkin Math 4.1: How much of the window is evidence

Problem: In our reference repository-repair harness, an agent runs inside a 128,000-token context window under a 30-turn step budget. The runtime reserves 8,000 tokens for generation (\(\OutputLimit\)) and assigns 3,000 tokens to system instructions (\(\SystemInstructionTokenCount\)) and 12,000 tokens to tool declarations (\(\ToolDeclarationTokenCount\)). Dynamic task state (constraints, progress records, and active code excerpts) takes 24,000 tokens (\(\TaskStateTokenCount\)). If every raw tool result is appended directly at 4,000 tokens per turn, how many turns fit within the window, and what maximum per-turn observation size lets the agent finish the full step budget?

Variables: The context window and step horizon come from the registry’s repository-repair harness profile, while component sizes and per-turn observation counts are scenario values for fully serialized wire requests.

Math: By equation 1, the usable input allowance is 128,000 − 8,000 = 120,000 tokens. Subtracting system instructions and tool declarations leaves 105,000 tokens (82.0 percent of the context window) for state and observations, and deducting the 24,000 tokens of state leaves an observation budget of 81,000 tokens. Under naive appending, this budget holds 81,000 / 4,000 = 20 turns. Lasting all 30 turns requires keeping the per-turn observation payload to at most 81,000 / 30 = 2,700 tokens.

Result: The appending policy overflows the context window at turn 21, well short of the 30-turn budget.

Systems insight: Because the number of feasible turns equals the observation budget divided by the per-turn observation size, observation volume is the primary design lever: compacting each result into an informative 2,000-token digest extends the run to 40 turns and clears the step budget. Fitting inside the window, however, is only the first cost. Because an appending policy re-reads every accumulated token on each call, its cumulative input bill grows quadratically in \(\TurnHorizon\), following the re-send pattern of equation 2.9 with a per-turn append \(\TurnAppendLength\) of 4,000 tokens (where the first request already holds one tool result, so the appended sum runs to \(\TurnHorizon(\TurnHorizon+1)/2\)). Selection keeps the bill linear. Across this step budget, and even granting the appending policy a window large enough to admit its final request, appending bills 3,030,000 input tokens against 1,410,000 for selection, 2.1× as many. That extra spend enters the numerator of \(\CostPerAcceptedTask\) (equation 1.2) on every attempt.

Appending re-reads every result, so its bill bends upward; selection stays linear.

The notebooks throughout this chapter reuse this baseline scenario, adopting the 128,000-token window, the 30-turn step budget, and the component token allocations defined above. Whereas the repair agent binds on the growth of \(\ObservationTokenCount\) against \(\TaskStateTokenCount\), the other three lighthouses stress different terms of the same partition.

Lighthouse 4.1: Context by archetype

Each recurring workload places pressure on a distinct component of the context budget in equation 1, calling for a different runtime mechanism (table 2).

Table 2: Context by archetype: The four recurring workloads bind on different terms of the same input budget.

Lighthouse	What fills the request	Binding constraint	Mechanism that answers it
Repair agent	Source excerpts, test logs, and a progress record from its own workspace, accumulating over a 30-turn step budget	Growth of \(\ObservationTokenCount\) displacing \(\TaskStateTokenCount\)	Compaction and progress records (section 4.5)
Research agent	Retrieved pages and search results written by third parties	Third-party text that dominates \(\ObservationTokenCount\) and may carry instructions	Staging that keeps each span’s source attached (section 4.4.2); enforcement in Quarantined Readers and Ingestion Isolation
Remediation agent	Readings of shared production state that other actors change	Staleness between the read and the action	Invalidation and action-time preconditions (section 4.7)
Support agent	Fixed policy instructions and tool declarations, one customer record, a short session	\(\SystemInstructionTokenCount+\ToolDeclarationTokenCount\) repeated across many short sessions	A stable leading block of instructions and declarations that later calls reuse (section 4.4.3)

Systems insight: The same budget equation binds on different constraints across workloads: token growth for the repair agent, source tracking for the research agent, freshness for the remediation agent, and prefix reuse across short sessions for the support agent. As a result, a policy tuned for one workload rarely transfers directly to another. Compaction rules that keep a repair agent within its token budget do not solve the main risk of a research agent, where untrusted external text matters far more than transcript length.

Every one of these budgets governs admission, ensuring that a serialized payload fits inside the provider’s context limit, yet admission alone does not mean the prompt supports the next decision. A prompt can obey the budget while omitting the return path needed to write a patch, or it can include every relevant code excerpt while burying version boundaries under old error logs. Sizing a context budget is therefore only the first step; the runtime must also ask how well the model can find and use the tokens admitted into that window.

4.3 Usable Working Context

In notebook 4.1, the 128,000-token window admits the investigation’s full transcript for its first 20 turns, yet fitting an entire history inside the window does not mean the runtime should send all of it. What matters is what each item contributes to the pending decision: an omitted synchronization contract can ruin a proposed repair, whereas ten pages of repeated timeout speculation only add noise. Raw request length cannot separate essential evidence from distraction.

The nominal context capacity is the maximum sequence length that the configured model interface accepts under its input and generation limits. How well the model reasons over a chosen set of instructions and evidence is a separate question, answered only by testing concrete tasks against explicit quality criteria (principle 4).

Definition 4.1: Usable working context

Usable working context is the context content, organization, and length with which a specified model and invocation configuration meet a chosen quality criterion on a stated workload.

1. Significance: Usable working context, rather than the nominal window, determines how much of the input allowance \(\ContextWindow-\OutputLimit\) effectively supports decisions. It dictates how many turns an execution trajectory can sustain before selection or compaction must intervene, directly setting the input token cost per accepted task in \(\CostPerAcceptedTask\).
2. Distinction: Nominal capacity is an architectural boundary of the serving interface that admits a request; usable working context is an empirical property of a task, model, and representation that only evaluation can establish.
3. Common pitfall: Treating a fixed fraction of the nominal window, or success on an isolated synthetic retrieval probe, as a universal guarantee of usable context for a production workload.

For a software repair workload, that criterion might require candidate patches to pass designated regression tests while staying within a permitted edit scope, whereas an information extraction workload requires high precision and recall against a target schema. Because failure modes differ across workloads, usable working context must be measured on each workload rather than read off the model card.

4.3.1 Testing usable working context

Testing whether a context length is usable starts with what the task asks the model to do with the staged text. A model that scores nearly perfectly on an isolated retrieval probe³ by finding a single planted fact in a long input can still fail a repair decision at the same length, because the repair requires relating observed symptoms to source files, honoring API invariants, and separating a measured failure from an unverified hypothesis (table 3). Synthetic retrieval probes quickly flag gross blind spots, but validating a context policy requires testing actual task decisions under realistic conditions.

³ Needle-in-a-Haystack Probes: The best-known isolated retrieval probe plants a short fact, the needle, in long distractor text and asks the model to return it. Hsieh et al. (2024) found that models scoring nearly perfectly on this probe degraded sharply as length grew on tasks that require multi-hop tracing and aggregation, and that only about half of the models tested held satisfactory performance at the context length all of them claimed to support. The gap between the probe and the harder tasks is the gap between nominal capacity and usable working context.

Hsieh, Cheng-Ping, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. “RULER: What’s the Real Context Size of Your Long-Context Language Models?” Conference on Language Modeling (COLM).

Table 3: Retrieval and decision scope: A simple retrieval probe and the running repair example test different capabilities.

Test obligation	Isolated key-retrieval probe	Connection-pool repair decision
Identify relevant content	Return the requested planted value.	Find the source, test conditions, and observation relevant to the notification path.
Interpret relationships	Match the queried key to its value.	Tie the failure to r17 and distinguish the missed-notification hypothesis from the observed blocked waiter.
Respect task constraints	Meet the probe’s answer requirements.	Propose a scoped change while preserving the public API and leaving untested behavior unresolved.
Establish success	Compare the returned value with the probe’s target.	Apply the chosen candidate checks and scope criteria; report only what they establish.

Context organization directly affects task success even when all required facts are present in the prompt. Evaluating multi-document question answering and key-value retrieval, Liu et al. (2024) reported marked sensitivity to the position of relevant information across model architectures, with accuracy dropping substantially when key facts appeared in the middle of the input. This position sensitivity, known as lost in the middle, means systems engineers cannot treat context as an unordered bag of tokens. Placement policies must be tested on the target workload, because how accuracy varies with position depends on the model family and prompt layout.

Length degrades comprehension even when every required fact remains in the input. One long-context benchmark isolates that effect by rewriting each of its stories at lengths from a few hundred tokens to the full text while preserving the details its questions depend on, so a change in score across lengths reflects the surrounding material rather than missing evidence (Fiction.live 2025). Answering requires following the order of events and drawing inferences from implicitly stated information, a task closer to a repair decision than to returning a planted value. Figure 2 plots the scores against the length of story in context, as aggregated by Epoch AI (2026).

Figure 2: Comprehension Falls with Context Length: Accuracy on the same stories as the version placed in context grows from the shortest rewrite to 192K tokens. The shaded band spans the middle half of the 53 models scored at every length through 60K tokens; colored lines trace six individual models. A few models stay near the top at every length, while the median model loses more than a third of its accuracy by 60K tokens. Each point rests on 36 questions, so single-length wiggles are within sampling noise. Data from Fiction.live (2025) via Epoch AI (2026).

Fiction.live. 2025. Fiction.LiveBench.

Epoch AI. 2026. Capabilities and Benchmarking.

Two features of figure 2 bear directly on the working budget: the decline starts early and runs deep, and models differ widely at any fixed length. Across the 53 fully scored models, the median falls from 100 percent on the shortest version to 58 percent at 8K tokens and 56 percent at 60K, with only 3 models staying at or above 90 percent at 60K. At 60K tokens, scores also span from 22 to 100 percent, which is the empirical content of the definition above: usable working context belongs to a specified model, configuration, and workload, so the budget must be found by evaluation rather than read from the window.

A first-order mechanism explains why appending distractor tokens cannot be treated as free. Within an individual attention head, a query assigns weight to each visible key through a softmax operation (Vaswani et al. 2017). For a target key with scaled logit \(\TargetKeyLogit\) alongside \(\CompetingKeyCount\) other keys with logits \(\CompetingKeyLogit_j = \QueryVector\cdot\KeyVector_j/\sqrt{\AttentionHeadDimension}\) for head dimension \(\AttentionHeadDimension\), the target’s weight is

\[ \TargetKeyAttentionWeight = \frac{e^{\TargetKeyLogit}}{e^{\TargetKeyLogit} + \sum_{j=1}^{\CompetingKeyCount} e^{\CompetingKeyLogit_j}}. \tag{2}\]

Fixed-logit distractors shrink the target’s share of one normalizer.

All weights share a single normalizer. Because each additional key contributes a strictly positive term \(e^{\CompetingKeyLogit_j}\) to the denominator of equation 2, expanding the sequence length while holding logits constant depresses the target key’s share of attention. This structural decay is attention dilution. In a toy head with a target logit of 6 and every background logit at -2, the target commands 96.8 percent of the weight among 100 keys. When the context expands to 100,000 keys, that same target’s share falls to 2.9 percent, even though no single background key is individually competitive.

In agent execution, this effect is often worse than unrelated filler would produce.⁴ Stale tool outputs, superseded logs, and earlier source reads share identifiers such as waiter, connection, and timeout with the evidence that matters, so they produce large inner products with query vectors and keep elevated logits. This dynamic is a sufficient condition for dilution under fixed logits, not a deterministic forecast of failure: production models sharpen logits, route evidence across multiple attention heads and layers, and learn position-dependent retrieval behaviors, which is why Liu et al. (2024) measured degradation empirically rather than deriving it analytically. Trained models also show a second consequence of the shared normalizer. Because the weights must sum to one, a query with no strong match deposits its leftover weight on the first few positions, which act as attention sinks regardless of their content (Xiao et al. 2024). Sinks change where leftover weight goes, not the dilution argument, and every staged token still adds a term to the normalizer that the evidence for the next decision must compete against.

⁴ Distracting Context: Shi et al. (2023) added irrelevant sentences to grade-school arithmetic problems and found that model accuracy fell dramatically, although the added text changed nothing about the answer. Because the distractors were irrelevant by construction, the drop measures interference rather than missing information. That interference is the cost an agent pays for every superseded log line and stale file read it keeps in the request.

Shi, Freda, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023. “Large Language Models Can Be Easily Distracted by Irrelevant Context.” International Conference on Machine Learning (ICML), Proceedings of machine learning research, vol. 202: 31210–27.

Xiao, Guangxuan, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. “Efficient Streaming Language Models with Attention Sinks.” International Conference on Learning Representations (ICLR).

↪ Why cache eviction policies must retain attention-sink tokens is developed in Reclamation vs. approximation.

4.3.2 Choosing a working budget

Sizing a working budget from the window’s admission bound alone can still leave a request too long for the model to use reliably. Choosing a working budget therefore takes two steps, and only the first can be computed from the window. In notebook 4.1, the admission bound leaves 81,000 tokens for observations in any one request: an appending policy must hold each result to 2,700 tokens to last all 30 turns, whereas a selecting policy can spend up to that whole share on every request because it does not accumulate. The ceiling says what fits, not what the next decision can use, so the second step searches beneath it empirically.

To find that working point, engineers test complete context policies on representative task suites (section 4.8) and choose the budget that meets the workload’s quality criterion at the lowest effective cost. Because a context policy sets a token limit alongside what material is retrieved, how structured data is formatted, and where each piece sits in the prompt, decision quality must be weighed alongside input size, latency, and token spend. Additional context can supply needed grounding, add harmless noise, or degrade reasoning through dilution, which means the quality curve is rarely monotonic and seldom has a single sharp peak. A documented operating envelope must therefore name the workload, model configuration, quality threshold, and test conditions.

These choices carry direct resource consequences under model-call cost accounting. Uncached tokens add prefill latency and compute charges, whereas aligned prefix caching spreads prompt costs across repeated turns. High cache hit rates or discounted per-token prices mean little if stale context hurts decisions, yet pruning context too sharply to save prefill cost can strip away required constraints. The engineering goal is to meet the task quality bar at the lowest effective cost per accepted task, \(\CostPerAcceptedTask\) (equation 1.2): trimming unneeded tokens lowers the attempt cost in the numerator, whereas dropping a needed fact lowers the acceptance probability \(\AcceptanceProbability\) in the denominator and drives up total spend across failed runs.

In the connection-pool repair example, keeping the full transcript of early exploratory turns wastes space once distilled test observations, code snippets, interface invariants, and active hypotheses are ready to guide the next patch. Stripping away a required interface contract or error log merely to fit an arbitrary sub-budget, however, is just as harmful. When no candidate policy meets both the task criterion and the available budget, the runtime must fetch the missing information, narrow the scope of the next permitted action, or report the limit; passing a request-admission check never certifies that the staged context is adequate. Once the runtime selects that working set and bounds its token budget, it must arrange those items into a concrete prompt layout that keeps sources and evidence relationships distinct while separating stable prefix content from turn-by-turn updates.

4.4 Staging the Next Invocation

In our connection-pool repair task, the selected manifest holds several distinct kinds of material: public API constraints, source excerpts from src/pool.py at revision r17, test results from test_waiter_notified_on_return, and the missed-notification working hypothesis alongside an abandoned timeout explanation. Concatenating these items into a flat, unlabeled string would blur the distinctions that guide the model’s next decision. Once a selection policy chooses the working set for an upcoming call, staging turns those entries into a structured request payload. That assembly step records the source, authority, and revision of every item so the model can tell policy instructions apart from test observations and working hypotheses, while ordering the content to minimize token churn across consecutive calls.

4.4.1 Ordering by change rate

Minimizing token churn across calls means ordering the prompt by how often each piece of context changes, placing stable material ahead of volatile updates. This layout partitions the serialized prompt into three regions: the root, the trunk, and the leaf, which group content strictly by its update cadence during a trajectory rather than by message role or authority (table 4). In the repair scenario from notebook 4.1, the root holds 3,000 tokens of system instructions and 12,000 tokens of tool declarations, the trunk holds 24,000 tokens of evolving task state, and the leaf receives 8,000 tokens of newly staged observations.

Table 4: Context ordered by change rate: The zones help preserve a stable request prefix. Message roles and source metadata remain separate from these placement choices.

Zone	Contents in the repair request	When it changes
Root	Application instructions and available tool definitions	The application contract or tool interface changes
Trunk	Task constraints, the current progress record, and selected source and test content at r17	A task constraint changes, an artifact is edited, or the progress record is refreshed
Leaf	The latest scoped test observation and the next bounded step, here proposing a candidate change to the notification path	New evidence or a new next-step decision arrives

Prefix stability must never override correctness. The root stays fixed only while its declarations remain accurate; if a tool definition changes during a session, the runtime updates the root on the next call and accepts the cache miss. Likewise, when a candidate patch advances the repository from r17 to r18, the runtime cannot keep r17 source code in the trunk just to keep the prefix warm. The earlier test failure remains valid historical evidence about r17, but checking whether it persists on the patched workspace requires a fresh run (section 4.7).

Information can also move between zones as a trajectory advances. An observation first staged in the leaf can fold into the trunk’s progress record during a scheduled refresh so long as its source, version, and scope stay attached; conversely, a per-step instruction injected into the leaf keeps its full authority despite sitting at the end of the prompt. Zone placement sets the physical order of tokens to match their change rate, whereas schemas and message roles carry provenance and meaning.

4.4.2 Preserving source and scope

As observations move from the leaf into the trunk or across compaction passes, each entry must keep the revision and execution conditions that produced it so the model does not read a local result as a universal fact about the codebase. For our failing test case, the host runtime attaches the test name, the workspace revision (r17), the test configuration, and a run identifier directly to the observation payload. This discipline enforces the scope rule:⁵ an observation supports claims only for the artifact versions and execution conditions under which it was gathered, and every later context transformation (summarization, compaction, and invalidation alike) must preserve those bounds.

⁵ Why- and Where-Provenance: Database research distinguishes why-provenance, the source records that justify a result’s existence, from where-provenance, the source locations its values were copied from (Buneman et al. 2001). The scope rule demands both for every observation: which execution and artifact versions justify its existence, and which payload spans supplied its values. A result that loses either becomes an unattributed claim that later compaction can silently generalize.

Buneman, Peter, Sanjeev Khanna, and Wang-Chiew Tan. 2001. “Why and Where: A Characterization of Data Provenance.” Database Theory – ICDT 2001, Lecture notes in computer science, vol. 1973: 316–30. https://doi.org/10.1007/3-540-44503-X_20.

Greshake, Kai, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. “Not What You’ve Signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.” Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, 79–90. https://doi.org/10.1145/3605764.3623985.

The runtime builds this framing on the outside rather than reading it from text inside the observation. A serializer places raw tool output strictly as a literal data value inside a structured envelope so that mismatched quotes, markdown delimiters, or control characters cannot corrupt the surrounding schema. Even with that framing, the payload itself remains an untrusted span: arbitrary text whose arrival the runtime can record, but whose claims it cannot vouch for. Careful encoding stops format injection, yet it cannot stop the language model from reading and following adversarial instructions hidden inside the text (Greshake et al. 2023).

Provenance labels therefore guide interpretation rather than enforcing security. Marking a tool output as untrusted data helps the model weigh its source, but if an observation tells the agent to drop a database or overwrite production credentials, the model may still propose that call. Security boundaries (filesystem sandboxes, read-only mounts, and deployment blocks) must run in host code outside the model’s prompt interpretation: staging keeps instructions and observations distinct in the request, while host gates enforce what may run.

↪ The host-side information-flow policy that limits what untrusted context may influence and where private data may flow is enforced in Information-Flow Policy.

How much weight this distinction carries depends on the workload. The repair agent’s observations come mostly from its own workspace, whose files change only when the agent or its tests change them. The research agent fills its requests with third-party text, so its \(\ObservationTokenCount\) consists mostly of untrusted spans (lighthouse 4.1). For that workload the untrusted-span boundary, not token growth, is the binding concern. Once each entry’s source and scope are framed, the serializer must emit the root and trunk in a repeatable token sequence so the serving engine can reuse their cached attention state.

4.4.3 Preserving an exact prefix

Because causal self-attention computes each token’s keys and values from all preceding tokens, a single changed token near the beginning of a request invalidates the cached key-value (KV) attention state for every later position, even when thousands of trailing tokens are identical. Serving engines equipped with prefix caching⁶ skip redundant prefill computation only across an identical leading token sequence, because attention state is derived from the token prefix (principle 5). Ordering the zones by change rate is what turns this rule into savings. A call that changes only the leaf reuses the root and trunk, whereas a compaction that rewrites the trunk ends reuse at the root boundary and forces recomputation of every later token (figure 3).

⁶ Automatic Prefix Reuse: Serving systems such as SGLang index cached attention state by token prefix within a radix tree, allowing an incoming request that shares an existing token prefix to reuse cached KV blocks transparently (Zheng et al. 2024). Because radix tree lookups evaluate exact token equality rather than semantic equivalence, two requests that diverge on an early timestamp share zero cache entries past that offset. Layout discipline directly determines cache hit rates.

Zheng, Lianmin, Liangsheng Ying, Zhenghao Yin, Cody Hao Liu, Eric Yu, Prakhar Patel, Zhipeng Yang, Anthony K. Joseph, and Ion Stoica. 2024. “SGLang: Efficient Execution of Structured Language Model Programs.” arXiv Preprint arXiv:2312.07104, ahead of print. https://doi.org/10.48550/arXiv.2312.07104.

Figure 3: Reuse Ends at the First Changed Token: Three consecutive requests follow the root, trunk, and leaf order of table 4, with the output reserve \(\OutputLimit\) beyond the input limit \(\ContextWindow-\OutputLimit\). A new observation changes only the leaf, so the root and trunk remain a reusable prefix. Refreshing the trunk to r18 moves the first changed token to the end of the root, and every later token is processed again. The figure assumes compatible cached state on the serving side; an identical prefix alone does not guarantee reuse.

Cache equality depends on the final serialized token sequence rather than on high-level application state. Small serialization changes (JSON key order, whitespace padding, prompt template edits, or token merges across boundaries) can quietly alter the token stream; for example, byte-pair encodings (BPE) often merge trailing whitespace with the leading characters of the next string, producing different token IDs at zone joins. Prefix stability must therefore be checked on the assembled, tokenized wire request.

An identical token prefix is necessary for cache reuse, but it is not sufficient on its own. The serving engine must also still hold compatible cache blocks in accelerator memory under the same model weights, positional encoding offsets, and attention configuration. Memory pressure, LRU eviction, or cluster autoscaling can drop cached blocks between turns even when the request prefix is byte-for-byte identical. While the serving engine manages and evicts those internal tensors, the agent runtime is responsible for keeping static material in a repeatable prefix and refreshing whatever state the next decision requires.

↪ Prefix compatibility and the retention and eviction of cached KV blocks in the serving engine are developed in Prefix Caching Across Turns.

Placing fast-changing metadata near the start of a request forfeits prefix reuse for no benefit, whereas moving per-call fields (request IDs, wall-clock timestamps, step counters) into the leaf preserves reuse across the root and trunk. Whenever an early field changes, cache sharing stops at the first differing token. The following calculation takes the extreme case where the very first token changes on every call, bounding the extra cost of putting volatile fields at the front of a request.

Napkin Math 4.2: Pricing a context layout across one trajectory

Problem: The repair agent of notebook 4.1 runs its full 30-turn step budget under the selecting policy, making one call per turn. Each request carries a 15,000-token root of instructions and tool declarations, a 24,000-token trunk of task state, and an 8,000-token leaf of selected observations, measured after serialization. The leaf is replaced on each call rather than appended, and the trunk is rewritten on 2 calls after the first, one for each compaction pass that notebook 4.3 finds this step budget triggers. Input tokens cost a hypothetical $3 per million tokens, with cached reads billed at 10 percent of that price. What input bill follows from each layout?

Variables: Both layouts send 47,000 tokens per request and 1,410,000 input tokens across the trajectory, matching the selecting policy budget from notebook 4.1. In the unstable layout, the first token changes on every call and no prefix is reused. In the stable layout, the first call is cold, the 2 trunk rewrites first differ right after the root, and the remaining 27 calls first differ right after the trunk, with compatible cached state kept for every matching prefix. This calculation isolates input token pricing, omitting generation charges and cache storage fees.

Math: The unstable layout processes all input without reuse, billing $4.23. In the stable layout, the initial cold call processes the whole request, each trunk rewrite prefills the trunk and leaf, and every warm call processes only the leaf, for a total uncached volume of \(47{,}000 + 2 \times 32{,}000 + 27 \times 8{,}000 = 327{,}000\) tokens. The remaining 1,083,000 tokens hit the prefix cache (76.8 percent of all input tokens). Billing uncached tokens at full price and reused tokens at the discounted rate gives $1.31 for the trajectory.

Result: Average input cost drops from $0.141 per call in the unstable layout to $0.044 in the stable layout. An unstable layout bills 3.2× the stable layout and requires 4.3× more uncached prefill tokens. These figures are billing and token-volume ratios, not end-to-end execution speedups.

Systems insight: With a stable prefix, the size of the volatile leaf sets the marginal input cost of each warm call, whereas rewriting the trunk prefills 32,000 tokens, the cost of 4 leaf-only calls. Engineers should keep per-call dynamic fields in the leaf and rewrite the trunk only when workspace files or active task constraints change. That discipline lowers the input share of attempt cost in \(\CostPerAcceptedTask\) by the ratio above, while keeping task accuracy (\(\AcceptanceProbability\)) intact because the staged context still reflects current environment state. A high cache hit rate never justifies keeping stale context.

Checkpoint 4.1: Checking the staged request

A candidate patch has just moved the workspace from r17 to r18, and the runtime is staging the next request.

• Provenance after promotion: If a test result moves from the leaf into a refreshed progress record, what source, version, and scope must remain attached to it?
• Volatile fields: Why does a changing field near the start shorten prefix reuse without necessarily eliminating it, and where should a per-call request identifier go?
• Trunk refresh estimate: With the layout notebook’s zone sizes, how many uncached input tokens does the trajectory pay if the trunk is rewritten on every fourth call after the first instead of twice? Compare the result with the unstable layout’s count.
• Stale trunk: After the source changes from r17 to r18, why is keeping the old source in the trunk an incorrect optimization even when it raises the hit rate?

Staging puts the selected evidence into a clear schema and an order built for prefix reuse, yet ordering alone does not shrink long tool outputs: an oversized test log or large source file still takes its full share of the budget. When the needed working set outgrows the available budget, the runtime must shorten the representation itself while keeping the evidence later decisions need.

4.5 Context Compaction

As the connection-pool repair runs across turns, it accumulates repeated source listings, boilerplate test-setup output, and long execution traces. Keeping every line in every prompt eats up space that later turns need for patch inspection and regression checks, yet collapsing that trace into a bare note like “the test failed” drops the clues that matter: which revision failed, which thread schedule ran, and which assertion tripped. Compaction must shorten the working context without erasing the causal links future decisions rely on.

Definition 4.2: Context compaction

Context compaction replaces material in a working context with a shorter representation chosen for the task’s next decisions.

1. Significance: It creates room for new observations inside the fixed input allowance \(\ContextWindow-\OutputLimit\), allowing an agent trajectory to run past the turn at which appending would overflow while retaining critical evidence, constraints, and unresolved obligations.
2. Distinction: A compressed representation is inherently lossy or access-deferred. Keeping a handle to omitted material makes recovery conditional on an explicit retrieval step.
3. Common pitfall: Assuming that a deterministic extractor, or a summary mentioning all required field names, guarantees that no decision-critical information was lost.

4.5.1 Choosing a compaction method

Different compaction methods trade token savings against validation work and the risk of silent information loss in different ways. A regular-expression filter trims log lines deterministically but can drop the one line that explains an intermittent fault, whereas a source-code outline keeps API signatures while stripping the locking logic needed to spot a missed wakeup; at the other end of the spectrum, an LLM-written summary can preserve high-level diagnostic intent, yet models often invent details or misstate whether a check passed.⁷ Because every shorter representation risks dropping something a later turn needs, no single compaction method fits all runtime data.

⁷ Unfaithful Summaries: In a human evaluation of neural abstractive summarizers, Maynez et al. (2020) found hallucinated content (statements unsupported by the source document) in the output of every model studied. A progress record written by a model inherits this failure mode. The record audit of section 4.6.2 therefore compares each claimed milestone with its recorded event instead of trusting the summarizer’s account.

Maynez, Joshua, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. “On Faithfulness and Factuality in Abstractive Summarization.” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 1906–19. https://doi.org/10.18653/v1/2020.acl-main.173.

Table 5 compares the choices by what they must preserve rather than by fixed reduction ratios. The right representation depends on the next step: checking an interface takes only an outline, whereas debugging a race condition requires full function bodies and thread-interleaving logs.

Table 5: Compaction representation choices: Trade-offs between prompt reduction, residual prompt contents, and the downstream validation required to prevent silent information loss.

Representation	What remains in the request	What must be checked
Normalized or deduplicated text	Selected text with repeated output or formatting removed	Whether repetition, ordering, timing, or formatting carries task evidence
Tool-result digest and raw handle	Status, selected evidence, source identity, and a retrieval reference	Whether the digest supports the next decision and the omitted payload remains accessible
Deterministic extraction	Matched log records, selected fields, source excerpts, or code outlines	Whether the extraction rule omitted relevant context, relationships, or implementation details
Model-written summary	A generated account of observations, hypotheses, and progress	Whether each retained claim is supported and required details and uncertainty remain explicit

The first three classes in table 5 use deterministic rules that run in host code without calling a language model. Table 6 breaks those three rows into the concrete techniques a runtime builds and asks what specific evidence each one can drop without raising an error. Deterministic filtering guarantees repeatable output, but it cannot guarantee that the remaining lines suffice for the task; a filter is safe only when later decisions do not depend on what it cut.

Table 6: Deterministic compaction techniques: Specific evidence dropped by programmatic pruning methods. The trace in listing 1 demonstrates how naive failure-marker slicing discards the causal ordering behind a test failure.

Technique	What it keeps	What it can silently drop
Repeated-poll deduplication	One copy of an unchanged status output and a repetition count	A brief change between polls, or the timing of repetitions
Terminal-control stripping	Plain text without escape codes, progress redraws, or spinners	Timing or ordering signals carried only by the redraws
Diff hunk folding	Changed lines, their location, and a marker for unchanged context	Surrounding code needed to judge whether the change is correct
Superseded-read replacement	A pointer from an old file read to the edit that replaced its source	Old text a later decision needs to explain what changed
Head, tail, and failure-marker slicing	Setup lines, final lines, and lines matching failure patterns	An intermediate warning or event ordering that matches no pattern
Source outline	Signatures, types, and docstrings, with bodies folded	The implementation a repair must read, such as the notification path
Tool-result digest	Status, exit code, payload size, and a handle to the retained output	Every detail of the payload until the handle is retrieved

Lossless byte compression offers no shortcut, because the model reads only decoded tokens. The alternative that loses nothing is the handle rule from the working-set decision, which pairs a digest in the request with a handle to the retained payload. The raw payload stays in an external artifact store, and the cost of that choice falls on the later step that needs the payload, which pays a read’s latency and the tokens of whatever it brings back. In our connection-pool repair task, running the controlled regression test at revision r17 produces the log excerpt in listing 1 (with repeated polling lines already stripped).

Listing 1: Observation before compaction: The event ordering and failed assertion together identify what the controlled test observed at r17.

source=src/pool.py@r17 test=tests/test_pool.py@r17
test_name=test_waiter_notified_on_return
configuration=controlled waiter-before-return schedule
... repeated worker progress messages ...
waiter_registered_before_return=true
connection_available_after_return=true
waiter_resumed_at_notification_check=false
FAILED: assert waiter_resumed_at_notification_check

A simple filter that keeps only lines matching FAILED captures the final assertion failure while dropping the event sequence that explains it. Without seeing that waiter_registered_before_return and connection_available_after_return both evaluated to true, the model cannot tell whether the failure came from a registration race, an exhausted pool, or a missed notification. A compact, structured digest keeps that causal chain intact: “At r17, test_waiter_notified_on_return registered the waiter before returning a connection. The connection became available, but assert waiter_resumed_at_notification_check failed at the notification check.” Alongside this digest, the context keeps the revision tags (src/pool.py@r17, tests/test_pool.py@r17), the test schedule, and a handle to the full trace for on-demand inspection.

This short record supports the missed-notification hypothesis under the tested conditions while keeping measured facts separate from diagnostic guesses. When a candidate patch produces revision r18, the runtime records the new test outcome against r18 rather than overwriting or merging it with the r17 failure; collapsing both runs into an unqualified note such as “the bug is fixed” would erase the provenance needed to check what the patch actually changed.

4.5.2 Protecting task-scoped fields

Keeping that provenance intact across compaction passes requires naming which fields a pass is never allowed to drop. In the connection-pool trace, losing the source revision (pool.py@r17), the test name (test_waiter_notified_on_return), or the concurrency schedule leaves the next turn without the conditions needed to reproduce the fault. The runtime therefore defines a set of protected fields that must survive every compaction transform, and checks before committing an updated working context that those fields remain present and unchanged. Alongside test identity, revision, scheduling flags, failed assertion, and exit status, global task constraints stay pinned as well: preserving the public API, keeping edits inside the approved repository directory, and blocking production deploys. Specific function bodies or configuration lines omitted from the digest can then be paged back in through handles when a later step needs them.

Which fields need protection depends on the current phase of the task. Pinning every historical line number, diff hunk, and identifier defeats compaction by filling the budget, whereas a rigid boilerplate schema can drop the exact relationship a task hinges on, such as an assertion message detached from its commit hash or concurrency schedule. At the same time, runtime filters must scrub credentials, tokens, and private keys so that secret strings are never kept merely because they matched a literal-retention rule.

Protected fields are not limited to static parameters known at the start of a run, because agents often uncover new conditions during execution. Suppose our concurrency reproducer fails reliably under the controlled waiter-before-return schedule but passes most of the time under the default operating-system thread scheduler. That dependency does not appear in the initial task prompt; it emerges only in tool output. A summarizer that records “the waiter test failed” while dropping the schedule flag lets a later turn rerun the test under the default scheduler, see a chance pass on broken code, and declare the repair done. Whenever tool output shows that an outcome depends on a run parameter, the runtime promotes that parameter into the protected-field set alongside the observation it qualifies, subjecting it to the same checks as the original contract constraints.

Automated checks catch missing test names, altered literals, or results attached to the wrong revision, though schema checks alone cannot prove that the inventory covers every needed fact or that the model will use what survives (section 4.8). Before evicting any raw payload, the runtime records what happened to the omitted material: either it is no longer needed, it stays reachable behind a handle, or it has been permanently dropped and leaves an explicit gap in the context. If a later step needs an evicted detail and retrieval fails, the agent must rerun the tool or report the gap rather than making up plausible history.

4.5.3 Scheduling compaction within the request budget

Because each compaction pass folds new leaf observations into the trunk’s progress record, compacting after every small tool output would rewrite the trunk on every turn and forfeit prefix reuse across the trajectory. To avoid paying that trunk-rewrite cost on every step, runtimes separate the trigger threshold from the post-compaction target length. This two-threshold rule, hysteresis,⁸ lets observations build up in the leaf over several turns before the runtime rewrites the trunk. A narrow gap between the two watermarks causes frequent, costly rewrites, whereas a wider gap lets the agent run longer between passes at the cost of deeper compression when a pass finally triggers. Sizing the gap balances token headroom against information loss: compressing too far risks stripping needed detail, while compressing too little causes thrashing.

⁸ Watermark Pairs: Operating-system page reclaim uses the same two-threshold pattern. A reclaim thread wakes when free memory falls below a low watermark and stops only after restoring a higher one, allowing reclamation to run in batches rather than on every allocation. The gap balances batch size against invocation frequency, a trade-off that equation 3 quantifies for context compaction.

Let \(\CompactionTriggerLength\) and \(\CompactionTargetLength\) denote the upper trigger threshold and the post-compaction target, measured in total serialized input tokens. Because the runtime must keep room for generation, these watermarks obey \(\CompactionTargetLength < \CompactionTriggerLength \le \ContextWindow - \OutputLimit\), where \(\ContextWindow\) is the context window limit and \(\OutputLimit\) is the reserved output allowance. Both thresholds are runtime policy choices rather than physical model limits.

In an idealized cycle where each compaction pass resets the request to \(\CompactionTargetLength\) and each turn adds \(\TurnAppendLength > 0\) net input tokens, the runtime checks the request length after each addition before calling the model. The integer number of turns between compaction passes is

\[ \CompactionCycleTurnCount = \left\lceil \frac{\CompactionTriggerLength - \CompactionTargetLength}{\TurnAppendLength} \right\rceil. \tag{3}\]

Hysteresis turns compaction into a sawtooth whose gap sets the cadence.

Because context length is checked at turn boundaries, the serialized length just before compaction is \(\CompactionTargetLength + \CompactionCycleTurnCount\TurnAppendLength\), which can overshoot \(\CompactionTriggerLength\) by up to \(\TurnAppendLength - 1\) tokens. That possible overshoot makes \(\CompactionTriggerLength\) a soft threshold: when tool outputs vary widely in size, the runtime must either trim oversized observations or stage them behind handles so a single large output cannot cross the hard ceiling \(\ContextWindow - \OutputLimit\). If the runtime calls a model to write the summary, that compaction call also needs its own input and output budgets.

Compaction directly affects prompt-cache spend. A pass that rewrites the trunk invalidates cached KV blocks from the end of the root onward, incurring the trunk-prefill cost shown in Call \(\Glyph{n}+2\) of figure 3. Over a cycle of \(\CompactionCycleTurnCount\) task turns and one compaction pass, let \({\TaskTurnCost}_{\CompactionCycleTurnIndex}\) be the total cost of task turn \(\CompactionCycleTurnIndex\) (cached and uncached input tokens plus generated output tokens) and \(\CompactionPassCost\) be the full cost of the compaction pass under the same billing rules. The amortized cost per task turn is

\[ \AmortizedTurnCost = \frac{\sum_{i=1}^{\CompactionCycleTurnCount} {\TaskTurnCost}_{i} + \CompactionPassCost}{\CompactionCycleTurnCount}. \tag{4}\]

Any cache misses caused by rewriting the trunk already appear in the uncached token charges of the following \({\TaskTurnCost}_{\CompactionCycleTurnIndex}\) terms, so adding a separate cache-invalidation penalty would double-count them. Beyond prompt tokens, a full cost comparison must also count external storage, handle-retrieval latency, repeated checks, and runs that fail because needed context was dropped too early. A schedule that minimizes per-turn token spend \(\AmortizedTurnCost\) can still raise the effective cost per accepted task (\(\CostPerAcceptedTask\)) if heavy compression sends the agent down dead ends. The watermark gap \((\CompactionTriggerLength - \CompactionTargetLength)\) sets \(\CompactionCycleTurnCount\) and therefore how many turns share \(\CompactionPassCost\).

Napkin Math 4.3: Spacing compaction passes

Problem: The repair agent from notebook 4.1 appends each 4,000-token tool result and compacts instead of overflowing. Its 128,000-token context window, minus the 8,000-token output reserve, leaves 120,000 tokens for serialized input. Each compaction pass resets the prompt back to 47,000 tokens (matching the baseline selecting policy’s request size), and the next pass triggers at 92,000 tokens. How often does compaction run, and how many passes does the agent execute across its 30-turn step budget?

Variables: Every compaction pass resets the prompt to the target \(\CompactionTargetLength\), and per-turn additions \(\TurnAppendLength\) stay constant at 4,000 tokens. The runtime checks request length after each tool addition before calling the model. The watermarks are runtime policy settings rather than hardware limits.

Math: Applying equation 3 gives \(\CompactionCycleTurnCount = \lceil (92{,}000 - 47{,}000) / 4{,}000 \rceil = 12\) additions between compaction passes. The request reaches the trigger at 95,000 input tokens, overshooting \(\CompactionTriggerLength\) by 3,000 tokens, and adding the output reserve brings the total to 103,000 tokens, still inside the window before compaction runs. Across the 30-turn step budget, the runtime runs 2 compaction passes, each rewriting the trunk; those are the two trunk rewrites priced in notebook 4.2 (where the leaf was replaced on every call rather than growing between passes). Raising the post-compaction target to 81,000 tokens under the same arrival rate shrinks the interval to 3 additions.

Systems insight: Compaction frequency grows with the token arrival rate divided by the watermark gap, so narrowing the gap from 12 arrivals to 3 triggers compaction 4× as often and raises the per-turn compaction overhead in equation 4 by the same factor. Setting a gap smaller than a single turn’s arrival causes thrashing by compacting on every step. Designers should size the gap to span several arrivals and compress toward \(\CompactionTargetLength\) only as far as protected-field rules allow, testing on real tasks where the line between needed context and removable detail lies.

A shortened observation captures only one piece of the repair’s state; finishing the task also requires carrying forward the current candidate patch, the checks already run on it, the remaining test obligations, and the next bounded step. Because free-form prose summaries easily blur those links during compaction, the runtime keeps them in a structured progress record.

4.6 Structured Progress Records

By the time the candidate patch is tested, the repair has moved through three linked events: at r17, the controlled test failed; the agent’s patch then wrote r18; and the reproducer passed on r18. All three events must survive the next compaction with their relationships intact. A naive summary that says only “fixed the notification bug” drops the distinctions the agent needs to choose its next step: the r17 test saw a returned connection leave a waiting thread blocked, whereas the r18 run showed only that the waiter wakes under that single test schedule, proving nothing yet about the broader regression suite or public API compatibility. Once raw diagnostic turns are evicted from the window, the next call still needs to tell verified observations apart from working hypotheses and an incomplete repair.

A structured progress record carries this state across compactions in named fields, holding active task constraints, verified observations, working hypotheses, artifact versions, and open obligations instead of replaying raw dialogue. A fixed schema lets host code flag missing fields and internal contradictions automatically, though checking whether the recorded claims match what actually happened still requires an audit against host logs. This record is a continuation manifest for the next prompt rather than an execution checkpoint: snapshotting processes, tracking dirty files on disk, and recovering in-flight tool calls belong to execution persistence.

↪ Execution checkpoints, which bind workspace snapshots and in-flight operations to a committed log position, are developed in Checkpoints.

4.6.1 Recording facts and open questions

Keeping a continuation record trustworthy starts with separating facts the host runtime can verify from hypotheses proposed by the model, so a compaction pass cannot turn an instruction like “test this repair” into a claim that “this repair passed.” Active task constraints come from the task contract rather than from a model’s summary, artifact versions and tool outcomes come from host execution logs rather than free-form prose, and a diagnosis stays marked as a hypothesis even when it explains the symptoms seen so far. In the connection-pool repair, the agent cannot loosen public API rules, widen its allowed workspace, or lift deployment blocks merely by rewriting its progress record. When an operator updates the task contract, the runtime writes the new terms directly into the prompt rather than keeping old instructions to save a prefix hit, and each progress record names the contract version it was built against so the runtime can check its fields against current rules (table 7).

↩︎ The five-part task specification contract, the source of a task’s binding constraints, is established in The five-part task contract.

Table 7: A progress record for continuation: Named fields preserve the difference between historical evidence, the current candidate, and work still needed. Field presence alone does not establish semantic fidelity.

Record field	Repair information to retain	Basis for checking it
Task and constraints	Repair waiter notification; preserve the public API; approved workspace only; no production deployment	Current task contract and explicit authorized updates
Artifact identities	src/pool.py at r18, candidate patch identity, and the version of tests/test_pool.py and configuration used by each check	Recorded reads, applied diff, and check inputs
Supported observations	Waiter remained blocked at r17; test_waiter_notified_on_return passed on r18 under the recorded conditions	Corresponding tool results, not the model’s account of them
Hypotheses	Timeout explanation lacks support from the controlled trace; missed notification is the working explanation	Relevant observations and a retained distinction between diagnosis and observation
Unresolved obligations	Broader behavior and public API compatibility are not yet established by the reproducer	Required acceptance evidence compared with checks already performed
Next bounded step	Inspect the patch and relevant tests to choose the next permitted regression check	Remaining task needs, permissions, and resource allowance
Raw-payload handles	References to the original trace, patch, and test output when later inspection may require more detail	Retained payload identity, access permissions, and retrieval availability

Every observation entry in table 7 follows the scope rule, so the r17 failure and the r18 pass each speak only for their own revision. Milestones record the exact check run and its result (“reproducer passed on r18”) rather than a blanket claim (“the race condition is gone”), while the handle row applies the handle rule so the agent can pull back an exact assertion diff or full trace on demand without keeping raw logs in every prompt.

4.6.2 Auditing before replacing active history

Because a model-written progress record is an untrusted candidate to replace active history, it can satisfy every schema type check (supplying a valid status string, artifact tag, and event ID) while still reporting a test run that never happened. Before swapping the record into the prompt, the runtime must therefore check both its structure and its factual claims against host logs, applying the same split between schema validity and factual correctness that governs tool calls.

↩︎ The gap between a schema-valid payload and a correct one is established for tool calls in What a grammar mask guarantees.

To avoid circular validation, the audit checks candidate records against a required-field inventory owned by the host runtime. If the runtime trusted a checklist generated by the summarizer itself, a hallucinating model could drop the no-deploy rule from both the summary and the checklist while passing self-consistency checks. Instead, the runtime reads active constraints straight from the task contract and checks artifact hashes and test outcomes against its own execution log. While that check guarantees that known contract terms and logged events are accurately reported, deciding which code excerpts the next step will need remains a selection decision that no schema check can settle automatically.

For every claimed milestone, the runtime looks up the referenced event in the execution log and compares the recorded command, exit code, artifact version, and test scope. An unlogged tool result stays marked unknown, a failed check stays failed regardless of how the summary phrases it, and a nonzero exit code records failure without proving why it failed; open questions stay marked as hypotheses until a logged check settles them. If a candidate record claims that the full regression suite passed on r18 when the event log shows only test_waiter_notified_on_return, the audit catches the mismatch and rejects the replacement. Broader questions (whether that one test covers enough behavior, whether edge cases remain in other modules, or whether the next action is permitted) belong to test execution and policy gates rather than the summary audit.

Version checks are how the audit enforces the scope rule: a historical milestone may cite r17, whereas any claim about the current pool must point to evidence recorded at r18. Separating a missing log entry, a valid historical observation, and an unsupported claim about current code lets the runtime keep useful r17 history without mistaking it for proof that r18 works. Listing 2 puts these three checks into one host-side function, where every compared value comes from state the summarizer did not write: the runtime’s required-field inventory, the logged tool events, and the current workspace versions.

Listing 2: Auditing a candidate progress record: Every comparison uses evidence the summarizer did not produce. An empty result permits replacement; any finding rejects the candidate.

def audit_record(record, inventory, events, workspace):
    findings = []

    # 1. Required fields: contract constraints plus promoted discoveries,
    #    such as the controlled waiter-before-return schedule.
    for field in inventory.required_fields():
        if record.value_of(field.name) != field.value:
            findings.append(("missing_or_altered", field.name))

    # 2. Historical and current claims must match recorded evidence.
    for claim in (*record.milestones, *record.current_state_claims):
        event = events.get(claim.event_id)
        if event is None:
            findings.append(("no_evidence", claim))
        elif (claim.operation, claim.result, claim.artifact,
              claim.artifact_version, claim.scope) != \
             (event.operation, event.result, event.artifact,
              event.artifact_version, event.scope):
            findings.append(("contradicts_event", claim))

    # 3. Scope: history may name r17; a current-state claim needs support
    #    recorded at the version the workspace holds now.
    for claim in record.current_state_claims:
        if claim.artifact_version != workspace.version_of(claim.artifact):
            findings.append(("needs_current_evidence", claim))

    return findings

On our candidate record, Check 2 rejects the claim that the full regression suite passed on r18 because the event log records only the single reproducer test, just as Check 1 catches any dropped no-deployment rule or test-schedule flag and Check 3 keeps the r17 failure as history while blocking any claim that it describes the r18 workspace. Because the audit can reject a candidate, the runtime keeps raw tool logs and source diffs in storage until the candidate record is accepted or rejected; evicting turns from the prompt never means deleting raw payloads from backing storage. When the audit finds a mismatch, the runtime can either feed the specific findings back to the model for a bounded correction pass or build a minimal record directly from its own event log. Because correction calls spend both tokens and wall-clock time, the runtime caps repair attempts, handle reads, and elapsed time rather than looping until a candidate happens to pass. If the cap is reached without a verified record, the runtime keeps a larger slice of raw history (if space permits), pulls needed files on demand, or stops with an explicit handoff error; running out of retries never makes an unverified summary safe to admit.

4.6.3 Continuing from the record

Once the r18 progress record passes the audit, it anchors the next call without dragging along pages of refuted timeout speculation. The runtime assembles the request from the active contract, the verified record, and the fresh observations or code excerpts needed for the next step, placing them in root, trunk, and leaf order so the audited record sits in the reusable trunk until a constraint change or new trunk refresh replaces it. The assembled request, including framing and any payloads pulled back through handles, must still fit inside \(\ContextWindow - \OutputLimit\).

In the connection-pool repair, the next call sees the r18 patch alongside the passing reproducer and the still-open API compatibility checks. Whenever a later step needs details that compaction omitted, the runtime reads the original artifacts through their handles or reports the missing evidence explicitly. Compaction acts like a reversible cache rather than a one-way loss only when two conditions hold: the raw payloads remain in storage, and each new compaction pass is built from those raw records rather than by summarizing the previous summary. Summarizing an earlier summary carries forward every earlier omission, compounding small per-round losses across the run (section 4.8.2).

Writing a structured record to disk lets summary state survive a process restart, yet a saved context record alone does not make an interrupted run resumable: restoring workspace files, syncing git worktrees, and reconciling in-flight tool calls with unknown outcomes belong to execution recovery. Context continuation has a narrower job, ensuring that the next model call receives a checked record of task constraints and observations under their stated conditions.

↪ Restoring workspace state and reconciling in-flight tool calls on resumption are developed in Resuming a Trajectory.

Checkpoint 4.2: Continuing after compaction

A candidate progress record says the repair is complete, while the retained evidence contains only a passing reproducer on r18.

• Evidence scope: Which claim should the audit reject, and which historical result should it retain?
• Constraint preservation: How can the runtime check for a missing no-deployment requirement without relying on the summary’s own checklist?
• Missing detail: What conditions must hold before a raw-payload handle can supply an omitted assertion to the next call?
• Correction budget: If each rejected record costs one more call as large as the compaction pass and the policy caps corrections at two, what is the worst-case multiple on \(\CompactionPassCost\) in equation 4, and which outcomes remain permissible once the cap is reached?

A structured progress record preserves the distinctions an agent needs after old dialogue turns are evicted, yet those recorded observations support current decisions only while the workspace still matches what they read. Whenever an edit changes a source file, updates a configuration, or modifies a test fixture, the runtime must decide which findings still apply to the present state without erasing what was observed earlier.

4.7 Context Invalidation

Invalidation acts on the second event of the repair’s timeline, the write that turns r17 into r18, before the reproducer runs again. At that moment the working context holds the r17 source excerpt, the r17 test failure, and the missed-notification hypothesis built on both. While that failure still describes how the unpatched code behaved under the earlier source, test harness, and schedule, it says nothing about whether the r18 patch fixes the bug.

The same boundary separates historical source excerpts from current code. An excerpt of src/pool.py read at r17 explains why the agent formed its missed-notification hypothesis, whereas planning another edit against r18 requires reading src/pool.py at r18. When building the next request, the runtime must choose whether to keep the r17 excerpt as labeled history, replace it with the r18 code, or drop it because the next step no longer needs it.

Definition 4.3: Context invalidation

Context invalidation withdraws a staged observation’s authority to support claims about current state whenever an underlying artifact or operational precondition changes, while preserving that observation as historical evidence of the state it originally recorded.

1. Significance: When a superseded observation leaks into a decision prompt, it introduces an unchecked step error that compounds across multi-step execution horizons (equation 1.1). Invalidation intercepts this drift by converting stale state into an explicit refresh operation scheduled before the model acts.
2. Distinction: Invalidation alters an observation’s applicability to the present, not its historical truth. It also differs from context deletion, which merely removes tokens from the request payload. The test failure at r17 remains an immutable, accurate record of revision r17.
3. Common pitfall: Treating a fresh timestamp or an unexpired polling interval as a guarantee that an in-context observation matches the current artifact on disk.

When a runtime neglects context invalidation, stale observations mislead decisions in three ways. First, in offset drift, the model plans an edit against line numbers or byte offsets read at r17, so after r18 shifts the file, the patch tool either rejects the hunk (listing 3) or fuzzy-matches it into the wrong function. Second, in ghost-symptom chasing, the r17 failure trace stays unmarked in the prompt, leading the model to keep diagnosing a bug that the candidate patch already fixed. Third, in regression from old negative evidence, a long early failure log outweighs a short passing notice from a later run, prompting the model to revert or complicate a working edit. Each failure confuses historical observation with current state, a confusion that dependency tracking turns into an explicit refresh step.

Listing 3: An edit planned from a superseded read: The hunk’s line numbers and context come from the r17 excerpt, and the patch tool rejects it against r18. Illustrative output, not a recorded trace.

--- a/src/pool.py   (hunk planned from the r17 excerpt)
+++ b/src/pool.py
@@ -87,3 +87,4 @@
     def release(self, conn):
         self._idle.append(conn)
+        self._wake_one_waiter()
         return None
error: hunk 1 does not apply at line 87 of src/pool.py (r18)
error: expected context 'def release(self, conn):', found a different function

4.7.1 Tracking claim dependencies

Turning those failure modes into explicit refresh steps requires recording what each entry in the prompt depends on. Test-time deliberation solved the planning version of this problem by binding each observation to what it read and propagating lost support through a candidate plan’s dependency graph (Invalidating stale and refuted evidence). Context invalidation applies the same graph machinery to the working context, with two changes: the graph now covers the observations staged in the request rather than uncommitted plan nodes, and its usual trigger is a write the agent itself ran against the workspace. Observations reached by that write are marked historical rather than retracted. In the working context, a source excerpt records its file path, repository revision, and line span or abstract syntax tree (AST) symbol, while a test run records the source commit, test-suite revision, selected test case, runtime environment, and exit status. Derived conclusions inherit dependencies from their supporting observations, so once r18 replaces r17, the behavior of the patched pool is an open question that needs fresh test runs (table 8).

Table 8: Dependencies and current applicability: Each entry has a different reason to remain in, leave, or be refreshed within the repair’s next request.

Manifest entry	Relevant dependencies	Treatment after the candidate patch
Source excerpt used to plan the edit	src/pool.py at the recorded revision	Re-read the needed region at r18 before planning another edit.
Controlled waiter/return failure	Source, test, and configuration used at r17	Retain as historical evidence; do not report it as a test of r18.
Missed-notification hypothesis	The r17 source and observed failure	Keep its status as a hypothesis; reassess using the patch and new checks.
Subsequent passing reproducer	r18, the executed tests/test_pool.py, selected test, and configuration	Record this bounded result; broader behavior remains unresolved.

The runtime maintains this dependency graph in host code rather than leaving bookkeeping to model attention. Every staged observation \(\TrajectoryObservation_{t}\) binds to its target artifact through an observation binding:

\[ \ObservationBinding(\TrajectoryObservation_{t}) = \langle \text{artifact}, \text{kind}, \text{revision}, \text{range}, \text{content id} \rangle, \tag{5}\]

where \(\ObservationBinding(\TrajectoryObservation_{t})\) records the artifact identifier, its kind (source file, test run, daemon process, or build output), the revision observed, the byte range or symbol read, and a cryptographic hash of the content. Derived entries (working hypotheses and test summaries) hold directed edges to each supporting observation, forming the in-memory dependency graph.⁹ Propagation follows the plan-graph traversal with one difference made possible by the binding: a tool write marks its directly bound observations historical rather than unsupported, and only the conclusions downstream of them are flagged for reassessment. Running this propagation when the patch writes r18 produces the right-hand column of table 8, while an excerpt of docs/pool.md and the API usage note built from it keep their status because nothing they depend on changed (figure 4).

⁹ Invalidation Is Not Retraction: A truth maintenance system (Doyle 1979) retracts a belief when its premise fails, which suits a candidate plan whose refuted diagnosis should simply disappear. An executed observation is different, because the r17 failure stays true of r17. The runtime graph therefore marks an affected observation historical, removing only its support for current-state claims, and reserves retraction for derived conclusions such as a hypothesis.

Doyle, Jon. 1979. “A Truth Maintenance System.” Artificial Intelligence 12 (3): 231–72. https://doi.org/10.1016/0004-3702(79)90008-0.

Figure 4: Invalidation Follows Recorded Dependencies: Each staged observation carries the binding \(\ObservationBinding(\TrajectoryObservation_{t})\) of equation 5, and each derived entry holds an edge to the observations it depends on. When the patch writes src/pool.py from r17 to r18, propagation marks the source excerpt and the controlled test run historical and flags the missed-notification hypothesis for reassessment. The documentation excerpt and the note derived from it keep their status because nothing they depend on changed. A historical observation remains evidence about r17; the marking removes only its support for claims about r18.

Selective invalidation is only as accurate as its recorded edges. Editing a test assertion, changing a timeout setting, or modifying a core struct immediately invalidates dependent test claims, and whenever dependency edges are coarse or incomplete, safety favors over-invalidation: marking a wider set of entries for refresh avoids acting on silent regressions, because a missing edge never proves freshness.

Cryptographic content hashes verify whether input files stay identical across steps.¹⁰ When checking a derived result, the runtime compares the current content hashes of its direct dependencies against the hashes recorded when the tool ran. Even when all input hashes match, nondeterministic behavior (thread scheduling or network timing) means that a single passing run is still only one sample, so the manifest tracks artifact versions, execution scope, and sampling uncertainty separately.

¹⁰ Timestamps vs. Content Hashes: Make’s timestamp rule (Feldman 1979) rebuilds a target whose file was touched but not changed and can miss a real change whose timestamp moved backward. Modern build engines record cryptographic hashes of input contents and rebuild only when a dependency hash diverges (Mokhov et al. 2018). Context runtimes face the same trade-off, since a timestamp records when an inspection occurred, while a content hash confirms whether the artifact is unchanged.

Feldman, Stuart I. 1979. “Make — a Program for Maintaining Computer Programs.” Software: Practice and Experience 9 (4): 255–65. https://doi.org/10.1002/spe.4380090402.

Mokhov, Andrey, Neil Mitchell, and Simon Peyton Jones. 2018. “Build Systems à La Carte.” Proceedings of the ACM on Programming Languages 2 (ICFP): 79:1–29. https://doi.org/10.1145/3236774.

4.7.2 Detecting change and bounding uncertainty

The clearest signal that an artifact has changed is a write completed through a runtime tool. When a file-editing tool writes r18, the runtime marks all r17 source excerpts historical and records open test obligations for the modified file. Because a tool call that fails or times out can still leave partial edits on disk, the runtime checks the affected files alongside exit codes and return payloads, marking dependent observations for refresh until the on-disk state is known. All of this tracking belongs in the runtime’s dependency graph rather than in prompt reminders asking the model to remember what changed.

External processes can also change workspace files outside the agent’s tool calls. Cheap filesystem metadata (file size and modification time) offers a quick hint to re-check a file, while content hashes or git commit IDs confirm whether the bytes actually changed. Kernel filesystem watchers such as Linux’s inotify cut detection delay, yet queue overflows, unmonitored mounts, and dropped events can leave a cached view stale (Linux man-pages project 2026), so runtimes treat watcher notifications as hints to re-verify hashes rather than as a complete log of changes.

Linux man-pages project. 2026. Inotify(7): Monitoring Filesystem Events.

¹¹ Leases: Gray and Cheriton (1989) converted temporal bounds into formal consistency guarantees through leases. In a distributed cache, a lease grants a client exclusive or shared read rights over a datum for a fixed term, during which the server guarantees it will not permit conflicting writes until every lease expires or is explicitly released. This guarantee hinges on server-coordinated consensus. Because standard filesystems and external web services provide no such cooperative write locks to an agent runtime, a local observation TTL remains a heuristic rather than a consistency guarantee.

Gray, Cary G., and David R. Cheriton. 1989. “Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency.” Proceedings of the Twelfth ACM Symposium on Operating Systems Principles (SOSP), 202–10. https://doi.org/10.1145/74850.74870.

To bound staleness on external resources that emit no change events, runtimes often set a time-to-live (TTL) that forces a fresh read once an observation passes a chosen age. A TTL is a practical rule of thumb tuned to how fast the environment changes and what a stale decision costs, not a guarantee of consistency:¹¹ an observation can go stale microseconds after it is read, and an expired entry may still match an unchanged resource. Before any high-consequence action, the runtime bypasses TTL caches and reads the current state directly.

Even when the runtime refreshes context right before building a prompt, the external system can still change between prompt assembly and tool execution. Context engineering cannot close that race window inside the prompt; closing it requires action-time preconditions at the tool boundary, such as atomic compare-and-swap writes or revision checks.¹² Context assembly contributes by recording the exact revision ID on which the model planned its action so the tool runner can check that revision before applying the write.

¹² Optimistic Validation: In optimistic concurrency control, transactions execute reads and computations without acquiring locks, validating prior to commit that no conflicting transaction mutated their read set (Kung and Robinson 1981). If the validation check fails, the transaction aborts. Preconditions at the agent tool boundary follow the same structure. The assembled prompt supplies the version snapshot the model evaluated, and the tool runner verifies that version’s integrity before applying side effects to disk.

Kung, H. T., and John T. Robinson. 1981. “On Optimistic Methods for Concurrency Control.” ACM Transactions on Database Systems 6 (2): 213–26. https://doi.org/10.1145/319566.319567.

↪ Execution-time preconditions enforced as atomic conditional updates at the tool boundary are developed in Conditional updates.

4.7.3 Removing, replacing, or appending

Once dependency tracking or a version check marks an in-context observation outdated, the runtime must decide how to reflect that change in the next request (table 9): remove the outdated payload outright, replace it in place with a short marker or fresh read, or append a superseding notice at the end of the prompt. Each choice trades KV-cache prefix reuse against the risk of leaving stale tokens visible to attention.

Table 9: Invalidation representation choices: Removing a payload, replacing it in place, and appending a notice are different operations, even when each is described informally as tombstoning, the practice of leaving a marker where deleted content stood.

Policy	What the next request contains	Limitation
Remove the old payload	Other selected material, without the outdated excerpt	The request no longer supplies that historical detail; provenance must be retained elsewhere if needed.
Replace the old payload	Current evidence, or a short marker that identifies the superseded entry and pending refresh	A marker is not fresh evidence; changing earlier tokens also changes the reusable prefix boundary.
Append a superseding notice	The unchanged old payload plus a notice describing its status	The outdated material remains visible and occupies capacity; the notice does not enforce correct use.

In the connection-pool repair, an in-place marker can state directly that the earlier excerpt came from r17 and that the agent must read r18 before planning another edit, while keeping a short note on the r17 failure if that history still informs the diagnosis. That replacement keeps the investigation coherent without presenting old code as current state, and a handle to the full r17 excerpt preserves auditability without spending working-set tokens.

Appending a notice at the end of the prompt preserves the KV-cache prefix across the turn, yet it leaves the model to weigh the earlier excerpt against the trailing correction.¹³ Placing the notice at the end of the request or phrasing it strongly does not guarantee that attention will favor the correction over the detailed code or error trace above it. Conversely, removing the payload drops the stale tokens from the prompt, though it can leave downstream hypotheses dangling unless the runtime also propagates the change through the dependency graph. Both removal and in-place replacement end KV-cache reuse at the first modified token, forcing the serving engine to prefill the entire suffix that follows.

¹³ Tombstones in Storage Engines: Log-structured storage engines like Bigtable record deletions by writing tombstones, which actively mask superseded values during read scans until major compaction discards both the tombstone and the deleted record (Chang et al. 2008). The storage engine’s read path mechanically enforces this masking. By contrast, an appended advisory notice in a language model prompt has no read filter, so the superseded text remains visible in the attention matrix, leaving the model to resolve the contradiction on its own.

Chang, Fay, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber. 2008. “Bigtable: A Distributed Storage System for Structured Data.” ACM Transactions on Computer Systems 26 (2): 1–26. https://doi.org/10.1145/1365815.1365816.

Napkin Math 4.4: Token accounting for invalidation

Problem: How do payload removal, in-place replacement, and appended notices change resident token counts and the uncached suffix that the serving engine must prefill?

Variables: The baseline prompt is the 47,000-token request under the selecting policy of notebook 4.1. An outdated 850-token source excerpt from r17 sits at the start of the trunk, leaving the 15,000-token root (system instructions and tool declarations) as the unchanged prefix. An in-place tombstone marker takes 24 tokens, and an appended notice, including role and message framing, takes 128 tokens. These counts are serialized token lengths assuming no cross-boundary BPE re-tokenization. To isolate the invalidation edit itself, the leaf is held fixed from the previous call and the notice is appended at the end of the request after the leaf; both removal and in-place replacement diverge from the previous request at the end of the root prefix, ending KV-cache reuse there.

Math: Removing the excerpt leaves 47,000 − 850 = 46,150 resident tokens, and subtracting the reusable root leaves 46,150 − 15,000 = 31,150 tokens in the uncached suffix.

Replacing the excerpt with a tombstone leaves 46,150 + 24 = 46,174 resident tokens and an uncached suffix of 46,174 − 15,000 = 31,174 tokens. Appending a notice leaves all earlier tokens untouched, giving 47,000 + 128 = 47,128 resident tokens with only the new 128-token tail beyond the cached prefix.

Systems insight: For the invalidation edit alone, appending preserves the cached prefix and prefills only 128 tokens against 31,174 under in-place replacement, roughly 244× fewer. On a real next call that also replaces the 8,000-token leaf, both choices prefill that new leaf as well, narrowing the per-call difference to the trunk that replacement recomputes. Moreover, the cache saving from appending recurs as a cost on every later call because all 850 superseded tokens remain in the prompt, diluting attention on current evidence (equation 2) and inviting ghost-symptom chasing. Runtimes therefore use the dependency graph to combine both approaches: append a notice for historical observations that the agent cites only as background, but replace or tombstone code regions that upcoming edits target, or batch that replacement into the next scheduled trunk rewrite when the trunk prefix will be recomputed anyway (notebook 4.2). Because none of the three options loads the new code, the read of r18 and the output reserve must be budgeted separately.

Context invalidation keeps historical observations separate from claims about present state by combining dependency tracking, bounded freshness checks, and deliberate replacement rules. Whether the complete context policy actually supplies enough evidence for valid actions while lowering token spend and prefill latency is an empirical question that only end-to-end evaluation on realistic tasks can answer.

4.8 Evaluating Context Policies

Evaluation returns to the end of the repair’s timeline. The reproducer has passed on r18, the history has been compacted, and the next request must carry the task constraints, the r17 failure of test_waiter_notified_on_return, and the r18 pass, each with its scope. At this point, the context policy succeeds only if the next prompt gives the model enough evidence to distinguish the active candidate patch from the earlier failure while showing which verification checks remain open. Token reduction alone is never evidence of success. When compaction erases version boundaries, an agent can mistake an obsolete diagnostic failure for current reality, or mistake a partial reproducer pass for complete task success, which example 4.1 dissects.

Example 4.1: A passing test without its version

Scenario: The repair stands where this chapter left it. test_waiter_notified_on_return failed at r17 and passes at r18 under the recorded configuration, and broader behavior is unverified. Before the next call, a naive compaction policy replaces these entries with one summary: “The waiter test passed; the repair is complete.” The API constraint and the deployment prohibition survive verbatim.

Diagnosis: A schema validator checking for a string-valued test_status accepts the summary without error, and a rule-based constraint check confirms that both negative constraints remain present. Neither check, however, detects that a single-case observation was turned into a claim of complete task success. An evidence probe closes this gap by freezing the history and inspecting both the assembled prompt and the model’s next decision, checking candidate and test identities, reproducer scope, and open acceptance checks. In a controlled variant where the evaluator edits a relevant test or source file after the passing run, a sound policy leads the agent to see that the earlier pass no longer covers the modified code and to run fresh checks before declaring the repair done; an edit to an unrelated file serves as a negative control to confirm that the policy does not invalidate context indiscriminately.

Systems lesson: A context policy must preserve causal relationships and version bindings, not merely isolated field names. A test result has meaning only relative to the exact artifact and execution conditions under which it was produced, and subsequent decisions must respect those boundaries. Targeted evidence probes pinpoint semantic information loss, while repeated dynamic trials verify whether the agent acts on those preserved distinctions during execution.

Every mechanism in this chapter meets at failures of this kind: selection decides what enters the prompt, staging sets representation and zone order, compaction and progress records govern what survives across trunk rewrites, and invalidation decides when stale support must be refreshed. Judging a policy by summary length alone misses the failures that happen at those joins, so evaluation must test the complete context policy (external reads, schema checks, and fallback paths included) under the target workload’s rules and budgets.

4.8.1 Comparing complete policies

Testing the complete policy means accounting for every read, summary call, and retry it triggers outside the main prompt, because comparing main prompt lengths alone can credit a “compact” policy that quietly shifts token spend into extra tool reads or helper calls. A fair comparison starts every repair trial from the same source snapshot, task contract, and initial observations, and charges every auxiliary read or model call to the policy that issued it. The harness holds the model, sampling settings, tool definitions, and acceptance suite fixed while varying only the context policy under shared ceilings such as the 128,000-token window and 30-turn step budget.

A complete policy specification must also state the operating rules that would otherwise hide costs or failure paths. A sliding-window policy must state how it pins system instructions and task constraints; a summarizing policy must state its trigger and target watermarks, protected-field inventory, and fallback path when a summary fails the audit; and any policy that moves raw payloads behind handles must state what happens when a handle read fails. Labels like “sliding window,” “summarizer,” or “structured record” leave out these rules, even though fallback behavior often decides whether an agent recovers or fails when context is missing.

Repeated runs across task cohorts remain necessary even under greedy decoding with fixed seeds, because distributed serving engines, kernel nondeterminism, and asynchronous tool scheduling all introduce run-to-run variation. Evaluation logs must record model versions, serving-engine builds, sampling settings, sandbox images, and KV-cache policies so that infrastructure changes are not mistaken for context-policy gains. A single passing run shows only that a trajectory can succeed once, not how reliably the policy performs.

↩︎ Why greedy decoding with a fixed seed still varies across served calls, and the difference between replaying a trajectory and re-querying the model, are established in Repetition and replay.

↪ How many tasks and repeated runs an estimate needs, and the confidence intervals on acceptance rates, are developed in Statistical Evaluation Rigor.

4.8.2 Measuring retention, behavior, and cost

To tell whether a failed trial came from a compaction pass that dropped a required constraint or from a model that ignored a constraint still in the prompt, the evaluation harness defines a required-information inventory against the task fixture before inspecting any runs. For the connection-pool repair, that inventory lists the public API rules, the approved workspace path, the no-deploy rule, the source and test revisions, and the scope of each test run.

Required-field retention is the fraction of required items whose value, scope, and provenance match the ground-truth fixture; merely keeping a field named test_status does not count. Exact identifiers and compiler messages can be checked by string equality, whereas paraphrased claims need semantic checks that account for grading uncertainty. Recording omitted, altered, and unsupported fields separately tells engineers whether the failure came from aggressive pruning or from hallucinated rewriting.

Retention across repeated compaction passes is much stricter than retention across a single pass, because an item can drop out during an intermediate rewrite and only reappear later through an expensive re-read. The benchmark must therefore state whether it checks retention at every compaction boundary, uninterrupted survival across the whole run, or presence just before a decision step. In equation 6, \(\TaskContract\) is the set of critical constraints from the task contract (the negative scope of the goal and the limits on permitted actions) rather than the five-part contract tuple itself, \(|\TaskContract|\) is their count, and \(\CompactionBoundaryCount\) counts compaction boundaries rather than turns or retries. For uninterrupted retention across \(\CompactionBoundaryCount\) compaction boundaries, the retention fraction is

\[ \ConstraintRetentionFraction(\CompactionBoundaryCount) = \frac{1}{|\TaskContract|} \sum_{\CriticalConstraint \in \TaskContract} \prod_{i=1}^{\CompactionBoundaryCount} \IndicatorFunction\left[\CriticalConstraint \text{ correctly retained at boundary } \CompactionBoundaryIndex\right]. \tag{6}\]

A small per-round loss compounds geometrically over repeated compactions.

The product in equation 6 shows why summarizing an earlier summary is so hazardous. When a required constraint survives each compaction round independently with probability \(1-\CompactionLossProbability\), its survival probability across \(\CompactionBoundaryCount\) rounds compounds to \((1-\CompactionLossProbability)^\CompactionBoundaryCount\), following the same compounding law as equation 1.1 with compaction passes in place of decision steps. Under a seemingly small per-round loss rate of 5 percent, a constraint survives 10 compaction rounds with probability 59.9 percent, but survives 40 rounds with probability only 12.9 percent. That rapid decay is why production runtimes check protected fields in host code at every compaction boundary rather than trusting an unconstrained model summary; extending the calculation to the full working set requires joint survival estimates because real compaction losses often correlate across fields within the same pass.

Behavioral metrics likewise need explicit denominators. A constraint-violation rate counts violating actions against the decision points where that constraint applied, logging blocked proposals separately from executed violations so generation errors are not confused with missing host gates. In the same way, the stale-support rate

\[ \StaleSupportRate = \frac{\text{decisions that treat superseded support as current}}{\text{decisions that require current evidence}} \tag{7}\]

restricts its denominator to decisions where evidence freshness affects validity. Under the scope rule, citing the r17 failure as history after moving to r18 is valid, whereas citing it as proof of how r18 behaves is a stale-support error.

End-to-end task acceptance estimates the acceptance probability \(\AcceptanceProbability\) in equation 1.2:

\[ \EstimatedAcceptanceProbability = \frac{\AcceptedTasks}{\EvaluationTrialCount}, \tag{8}\]

where \(\AcceptedTasks\) counts the trials that pass the full verification standard within their resource budgets and \(\EvaluationTrialCount\) counts all assigned trials. Passing the single-case reproducer is only an intermediate milestone, and timed-out runs, tool crashes, or context overflows must stay in the denominator \(\EvaluationTrialCount\) rather than being dropped. Table 10 brings these representation, behavioral, and outcome metrics together under one measurement contract.

↪ The task acceptance contract that decides which trials count as accepted is developed in The Task Acceptance Contract.

Table 10: Context-policy measurement contract: Representation checks, behavioral observations, and task outcomes answer different questions. Their denominators and coverage must remain explicit when comparing policies.

Measurement	What the evaluator records	What the result cannot establish
Required-field retention	Correct values and evidence scope at each specified compaction boundary	That the inventory includes every fact needed later
Constraint compliance	Violating proposals among applicable proposals, separately from executed violations	That prompt text enforces an action boundary
Current-evidence use	Decisions using superseded support as current, among eligible decisions	That every old observation is irrelevant
Task acceptance	Trials meeting the common evidence standard, including every assigned trial in the denominator	Correctness beyond that standard’s coverage
Request footprint	Fully serialized input length, output reservation, and rejected oversized requests	Lower total task cost or adequate information
Resource use	All calls, source reads, compaction, checks, and elapsed time charged to the cohort	A benefit unless accepted outcomes justify the expenditure

Compression measures representation size rather than decision quality. To compute compression at a decision step, serialize both an uncompacted reference prompt containing the full raw history and the candidate prompt produced by the context policy with the same tokenizer, including system instructions, tool declarations, schemas, and framing delimiters in both. The request compression ratio is

\[ \RequestCompressionRatio = \frac{\ReferenceRequestLength}{\PolicyRequestLength}, \tag{9}\]

where \(\ReferenceRequestLength\) and \(\PolicyRequestLength\) are the token counts of the uncompacted reference request and the policy-generated request. Because \(\ReferenceRequestLength\) is an analytical baseline, it can exceed the serving engine’s context window without needing to be run; values of equation 9 above \(1.0\) indicate token savings, provided both numerator and denominator count complete serialized requests rather than comparing raw observations against a framed prompt.

Compression ratios also depend on how a run unfolds. On fixed-trace probes where every policy processes the same recorded transcript, request lengths compare directly; in live closed-loop trials, however, a high compression ratio can simply mean that a weak policy wandered through extra exploratory turns that then had to be compacted. Reports should therefore pair compression ratios with total token counts and full system spend (including summarization calls, search queries, and handle reads) so a policy cannot look cheap merely by moving tokens out of the main prompt into helper calls.

Checking that an agent actually uses what the prompt retains requires causal interventions rather than static inspection, because token presence, attention weights, or copied identifiers show only correlation. By removing or altering a key variable in a controlled fixture (such as the branch tag or test status) while holding the rest of the prompt fixed, an evaluator can see whether the model’s next tool call changes accordingly; if the call stays the same, the model is ignoring that field under the current layout. Those static interventions still need to be paired with live task runs, where an agent can recover from an omitted prompt field by reading the workspace through a tool call. Counterfactual probes test omitted payloads in the same way: if a compaction policy replaces raw test logs with an external handle, temporarily disabling the handle store shows whether the agent falls back cleanly or guesses when retrieval fails, a failure mode that field-presence checks never see.

Choosing a production context policy means weighing all of these measurements together across task acceptance, failure modes, latency, and total dollar cost under shared limits. A structured-record policy that keeps every schema field yet still leads the model to declare victory on an incomplete check fails its purpose. The right production policy is the one that minimizes the effective cost per accepted task, \(\CostPerAcceptedTask\), across the evaluation cohort: shrinking the prompt lowers per-call spend, but if aggressive pruning lowers \(\AcceptanceProbability\), the cost per accepted task rises sharply.

Checkpoint 4.3: Testing what survives compaction

A candidate policy preserves the manifest’s field names but drops specific values and version relationships. Its resulting request is measurably shorter than the uncompacted reference prompt.

• Retention: How does an automated validation check distinguish mere field key presence from the semantic retention of the r18 reproducer test outcome?
• Evidence scope: How does an intervention probe distinguish a valid historical reference to the r17 failure from an unsupported assertion about the r18 workspace?
• Cost boundary: If the runtime fetches an evicted diagnostic log on every subsequent turn via tool calls, which metrics reveal whether compaction achieved a net resource reduction?
• Retention estimate: Using the hypothetical per-round loss rate from equation 6, after how many compaction rounds does a single constraint’s survival probability fall below even odds (50 percent), and what does this imply for where protected-field validation must execute?
• Controlled comparison: Why can a fixed-trace probe isolate representation defects without predicting dynamic end-to-end task completion rates?

Every evaluation result is only as strong as its test fixtures, evidence rules, and cost accounting. The most common evaluation mistakes come from treating a single intermediate metric as proof of overall quality, such as mistaking a shorter prompt for better context or mistaking safety instructions in the prompt for an enforced execution gate.

4.9 Fallacies and Pitfalls

A prompt can be compressed until it drops a required constraint, or left as an uncurated transcript where superseded tool outputs sit beside current state. Both failures happen when engineers ask the prompt payload to do the work of state tracking, durable storage, and execution enforcement all at once.

Fallacy: If every token fits, every needed fact is usable.

Admitting a prompt within the interface’s context window does not ensure that the model can attend to and extract every critical fact (section 4.3). Token capacity sets only an upper bound on input size, whereas evidence retrieval depends on how attention weight is distributed. For example, under fixed logits, the toy attention head in equation 2 assigns 96.8 percent of its attention weight to the target token across a sequence of 100 keys, but this share declines to 2.9 percent when distributed across 100,000 keys, even though all 100,000 keys fit within the context limit. Validating a context layout therefore requires varying information density and placement alongside sequence length, measuring actual task completion on realistic workloads rather than relying on raw token budgets.

Pitfall: Treating deterministic extraction as lossless task understanding.

Deterministic filters and parsers guarantee reproducibility, but reproducibility is not task understanding. A heuristic parser consistently extracts the same error lines while silently dropping the warning that preceded them. For example, filtering exclusively for FAILED status lines in listing 1 isolates the failing assertion while discarding the event sequence that makes the failure actionable. Deterministic pipelines make transformations inspectable and debuggable, yet they remain vulnerable to silent data loss. Reliable runtimes guard against this by marking protected fields that summarization passes cannot discard, noting omitted spans, and retaining content-addressed handles to raw payloads so that an agent can re-read original data when needed.

Fallacy: A valid progress-record schema guarantees continued compliance.

A structured schema verifies syntactic presence, but it cannot guarantee that task constraints survive repeated compaction cycles. Under the hypothetical 5 percent per-round loss rate modeled in equation 6, an operational policy constraint survives 40 iterative compaction steps with only 12.9 percent probability, even though every intermediate serialized record validates cleanly against the schema. Systems engineers must treat prompt context as an advisory channel rather than an enforcement layer. Runtimes must validate protected fields against ground-truth sources at every compaction boundary (section 4.6), check subsequent agent actions against security policies, and enforce hard execution boundaries in host code rather than relying on in-context reminders.

Pitfall: Keeping obsolete support solely to preserve a prefix hit.

Appending corrections to the end of a transcript preserves prefix-cache hits for the next call. In notebook 4.4, the invalidation edit alone makes in-place replacement reprocess roughly 244× as many tokens beyond the prefix, a gap that narrows to the size of the trunk once the next call’s new leaf is counted. However, prioritizing short-term cache hits retains 850 superseded tokens across all subsequent requests, forcing the model to repeatedly reconcile stale text with appended errata. The dependency graph decides which rule applies. Evidence in any region that a later edit or current-state decision targets should be replaced, and appended notices reserved for inert history that the agent cites only as rationale (section 4.7).

Fallacy: A larger context window removes the need to select and compact.

Expanding context windows raises physical token limits, but it does not eliminate the compounding latency, cost, and attention dilution of unmanaged histories. As demonstrated in notebook 4.1, an append-only transcript policy running across a 30-turn horizon requires 167,000 tokens by its final request. While an expanded window accommodates this trajectory, it bills 3,030,000 input tokens compared to 1,410,000 tokens for a selective policy that maintains a compact working set of 8,000 observation tokens, a factor of 2.1× in token volume. The extra tokens also dilute attention over the evidence required for subsequent tool calls. Structured selection and compaction remain necessary, reducing both the effective trial cost \(\CostPerAcceptedTask\) and the distraction within the request regardless of window capacity.

Pitfall: Placing unnecessary volatile fields before reusable content.

Prefix-caching engines match sequence tokens strictly from left to right. Inserting dynamic values (timestamps, request identifiers, ephemeral counters) near the start of a prompt breaks cache reuse across turns, even when the system prompt, schemas, and task instructions that follow are completely unchanged. In notebook 4.2, this volatile layout increases the input bill so that it is 3.2× that of the stable layout under explicit reuse and pricing assumptions. Exact-prefix matching halts at the very first differing token, regardless of message roles or field meaning. System architects should structure prompts hierarchically, pinning invariant instructions and tools at the root prefix and deferring volatile, per-call parameters to the leaves.

These pitfalls share one root cause: treating the prompt window as a single store for evidence, task state, and execution enforcement. Sound context engineering divides those jobs across the runtime, using the working context to stage task-relevant evidence for the next decision, external stores to hold verified history, and host gates to enforce action boundaries.

4.10 Summary

Fitting an execution trace within a model’s context window does not ensure that the model will make a sound decision. Throughout this chapter, the repair agent’s most instructive failures rarely stemmed from context window exhaustion. Instead, they occurred when the context representation severed causal relationships: a discarded hypothesis remained displayed alongside contradictory evidence, a passing reproducer lost its revision metadata, an automated summary silently dropped an operational constraint, and a cached code excerpt persisted after the underlying file had changed. Each case reflects a failure of context state management rather than token capacity. Solving these breakdowns requires runtime bookkeeping outside the model (explicit working-set management, dependency tracking, cache-aware layout, and artifact versioning), not a larger context window or more elaborate prompt wording. Table 11 summarizes the quantitative relations developed throughout this chapter to budget context tokens, schedule compaction passes, invalidate stale observations, and evaluate end-to-end task efficiency.

Table 11: Key equations: Each relation counts complete serialized requests under the chapter’s stated assumptions; the attention share uses fixed toy logits and bounds no accuracy.

Equation	What it bounds or decides	Where derived
\(\SystemInstructionTokenCount + \ToolDeclarationTokenCount + \TaskStateTokenCount + \ObservationTokenCount \le \ContextWindow - \OutputLimit\)	Partitioning of input token capacity across roles after reserving output generation headroom	equation 1
\(\TurnHorizon(\SystemInstructionTokenCount + \ToolDeclarationTokenCount + \TaskStateTokenCount) + \TurnAppendLength \TurnHorizon(\TurnHorizon+1)/2\) against \(\TurnHorizon(\SystemInstructionTokenCount + \ToolDeclarationTokenCount + \TaskStateTokenCount + \ObservationTokenCount)\)	Cumulative input token scaling: quadratic growth for naive appending vs. linear growth for active selection	Adapted from equation 2.9 in notebook 4.1
\(\TargetKeyAttentionWeight = e^{\TargetKeyLogit} / \left(e^{\TargetKeyLogit} + \sum_{j=1}^{\CompetingKeyCount} e^{\CompetingKeyLogit_j}\right)\)	Attention weight retained by a target token as competing staged keys accumulate	equation 2
\(\CompactionCycleTurnCount = \lceil (\CompactionTriggerLength - \CompactionTargetLength) / \TurnAppendLength \rceil\)	Turn count between compaction passes under high and low watermarks, setting rewrite frequency	equation 3
\(\AmortizedTurnCost = \left(\sum_{i=1}^{\CompactionCycleTurnCount} {\TaskTurnCost}_{i} + \CompactionPassCost\right) / \CompactionCycleTurnCount\)	Amortized per-turn cost accounting for periodic compaction passes across an execution cycle	equation 4
\(\ObservationBinding(\TrajectoryObservation_{t}) = \langle \text{artifact}, \text{kind}, \text{revision}, \text{range}, \text{content id} \rangle\)	Observation provenance tuple used to identify and invalidate stale entries upon mutation	equation 5
\(\ConstraintRetentionFraction(\CompactionBoundaryCount)\); \((1-\CompactionLossProbability)^{\CompactionBoundaryCount}\) under independent per-round loss	Fraction of a constraint set retained across \(\CompactionBoundaryCount\) consecutive compaction boundaries	equation 6
\(\StaleSupportRate\) = superseded-support decisions / decisions requiring current evidence	Proportion of decisions that erroneously rely on superseded observations rather than current evidence	equation 7
\(\EstimatedAcceptanceProbability = \AcceptedTasks/\EvaluationTrialCount\), entering \(\CostPerAcceptedTask = \mathbb{E}[\AttemptCost]/\AcceptanceProbability\)	Effective deployment cost per accepted task completion across all assigned trials	equation 8
\(\RequestCompressionRatio = \ReferenceRequestLength / \PolicyRequestLength\)	Token reduction relative to an uncompacted reference; reflects size rather than reasoning quality	equation 9

Key Takeaways: Context is selected information for the next decision

• Context budgets require explicit working-set management: Every serialized request must preserve sufficient headroom for generation, but staying within token limits does not guarantee that the prompt supplies the necessary evidence. Naively appending every tool observation quickly exhausts the context window. In the reference repair agent, uncontrolled logging overflows the context window at turn 21 of 30. Effective runtimes actively bound and select per-turn observations to match the execution horizon while measuring decision quality directly.
• Prefix stability governs cache reuse rather than semantic authority: Placing static instructions and tool definitions in a stable prefix eliminates redundant prefill computation through KV-cache reuse. Across the repair trajectory, under the chapter’s stated reuse and price assumptions, an unstable layout’s input bill came to 3.2× that of a cache-aligned layout. Cache layout, however, should never be confused with authority. Message roles, cache boundaries, and data provenance are separate architectural concerns, and staging content in a privileged prefix does not validate untrusted tool outputs.
• Compaction mechanisms must enforce strict schema retention: Truncation and summarization inevitably discard detail. Because small per-round losses compound geometrically over long trajectories, compaction policies must explicitly safeguard the exact fields required by downstream tools and decisions. Runtimes should validate generated summaries against their original records at every compaction boundary, retaining raw-payload handles so the agent can deterministically retrieve omitted evidence on demand.
• Observation validity must be bound to artifact versions: Every tool observation reflects the specific system environment and artifact revision present when the tool ran. When an agent modifies source code or configuration files, earlier test results cease to validate claims about the current state, even though they remain historically accurate records of prior turns. Runtime managers must track revisions explicitly, invalidating stale claims while preserving observations that remain tied to untouched components.
• Context policies must be evaluated on end-to-end task outcomes: Intermediate metrics such as compression ratio, token count, and field retention provide useful diagnostics, but they do not measure decision quality. A compaction or retrieval strategy improves a system only if it reduces the effective cost per accepted task completion, accounting for the overhead of compaction calls, retries, and failed trajectories across all evaluated trials.

Treating prompt context as an active working set (principle 4) turns context engineering from ad hoc prompt assembly into a repeatable systems discipline. Rather than assuming that a larger window will resolve ambiguity, engineers must verify whether an assembled prompt contains the dependencies needed for the next decision step. Scoping observations by artifact revision reinforces the rule that authority remains with the external source of truth (principle 6), enabling the runtime to invalidate superseded evidence without corrupting historical audit logs. Working together, these principles ensure that context preparation is transparent about its cache costs, accountable for the evidence it provides, and explicit about unresolved dependencies.

4.11 Historical Perspective and Further Reading

Managing bounded fast memory against a much larger backing store is one of the foundational challenges of computer systems. When virtual memory decoupled address spaces from physical DRAM, Belady established the theoretical ceiling for page replacement by comparing online policies against an optimal offline algorithm with clairvoyant knowledge of future memory accesses (Belady 1966). Denning grounded this theory in measurable program behavior with the working-set model (Denning 1968), defining an empirical working set over a sliding time window to prevent thrashing. Context engineering confronts a similar capacity limit under one sharp systems difference: unlike hardware with memory-management units, a transformer forward pass emits no access traces or page faults, so the runtime cannot passively observe token reuse and must instead predict and assemble the relevant working set before issuing each model call. Virtual context systems such as MemGPT (Packer et al. 2023) adopted this hierarchical model directly, paging structured dialogue and retrieved facts between the active context window and external storage.

Packer, Charles, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. 2023. “MemGPT: Towards LLMs as Operating Systems.” arXiv Preprint arXiv:2310.08560, ahead of print. https://doi.org/10.48550/arXiv.2310.08560.

O’Neil, Patrick, Edward Cheng, Dieter Gawlick, and Elizabeth O’Neil. 1996. “The Log-Structured Merge-Tree (LSM-Tree).” Acta Informatica 33 (4): 351–85.

The mechanisms required to maintain consistency across derived agent state draw from decades of research in automated reasoning, software build systems, and distributed storage. Classical truth maintenance systems established early formalisms for tracking dependencies. Doyle (Doyle 1979) recorded explicit justifications for each maintained belief, automatically propagating invalidations whenever an underlying premise collapsed. Software build tools addressed the same synchronization challenge at the filesystem boundary. Feldman’s make (Feldman 1979) reconciled targets with dependencies using file modification timestamps, a strategy that modern build architectures subsequently generalized by tracking cryptographic content hashes and execution traces (Mokhov et al. 2018). In distributed systems and databases, optimistic concurrency control (Kung and Robinson 1981) protected state validity by verifying at commit time that underlying read sets remained unaltered, while cache leases (Gray and Cheriton 1989) bound staleness through time-limited writer contracts. Similarly, data provenance research clarified how derived views inherit lineage from source records (Buneman et al. 2001), and log-structured storage engines established how to compact and tombstone superseded records without stalling active reads (O’Neil et al. 1996; Chang et al. 2008). Agent runtimes encounter the same dilemma, because intermediate summaries and cached observations outlive their inputs unless governed by explicit dependency tracking and proactive eviction.

The foundation model layer introduced its own distinct set of systems bottlenecks. Transformer attention (Vaswani et al. 2017) structured every input sequence as an all-to-all competition for weight, making sequence length a quadratic compute and linear memory cost during inference. Retrieval-augmented generation (Lewis et al. 2020) brought dynamic document chunks into this input space, while reasoning-and-acting frameworks such as ReAct (Yao et al. 2023) established the now-standard agent loop of interleaved reasoning, tool invocation, and environment observations. As multi-turn interaction traces expanded, empirical evaluation began to distinguish nominal context capacity from usable context quality. Liu et al. (2024) documented the U-shaped “lost-in-the-middle” phenomenon, showing that retrieval performance falls when critical evidence sits in the interior of a long prompt. Shi et al. (2023) demonstrated that even benign, irrelevant context distracts model attention and degrades task accuracy, while Hsieh et al. (2024) revealed that basic needle-in-a-haystack benchmarks substantially overestimate real-world usable context on complex multi-hop queries. At the serving layer, systems such as vLLM (Kwon et al. 2023) and SGLang (Zheng et al. 2024) introduced paged attention and prefix caching to share key-value tensors across requests, turning prompt structure into a primary determinant of accelerator memory footprint and time-to-first-token latency. Finally, because naive history compaction can distort the evidence it keeps, findings by Maynez et al. (2020) on widespread hallucination in abstractive summaries show why runtime compaction requires verifiable provenance audits.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. “Attention Is All You Need.” Advances in Neural Information Processing Systems (NeurIPS) 30: 5998–6008.

Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, et al. 2020. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Advances in Neural Information Processing Systems 33: 9459–74.

Kwon, Woosuk, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” Proceedings of the 29th Symposium on Operating Systems Principles, 611–26. https://doi.org/10.1145/3600006.3613165.

• Denning (1968) introduces the classical working-set model based on temporal locality, illustrating both how the working set maps to context management and why the lack of observable memory reference strings in model inference requires proactive curation.
• Doyle (1979) establishes the foundational treatment of truth maintenance systems, formalizing how explicit dependency records enable non-monotonic invalidation whenever underlying assumptions change.
• Mokhov et al. (2018) formalize build systems by cleanly separating staleness detection from execution scheduling, providing a principled blueprint for decoupling dependency invalidation from prompt reconstruction in agent runtimes.
• Liu et al. (2024) isolate prompt position effects to demonstrate that transformer attention degrades when target information appears in the middle of long sequences, establishing an experimental methodology for evaluating context placement on realistic workloads.
• Hsieh et al. (2024) demonstrate why isolated needle-in-a-haystack tests overestimate usable context windows, offering a comprehensive benchmarking framework to evaluate multi-hop tracing, aggregation, and variable tracking across expanded context lengths.
• Maynez et al. (2020) quantify factual unfaithfulness and hallucination in abstractive summarization, providing direct empirical motivation for rigorous provenance tracking and audit mechanisms when compacting agent interaction histories.

Denning, Peter J. 1968. “The Working Set Model for Program Behavior.” Communications of the ACM 11 (5): 323–33. https://doi.org/10.1145/363095.363141.

Liu, Nelson F., Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. “Lost in the Middle: How Language Models Use Long Contexts.” Transactions of the Association for Computational Linguistics 12: 157–73. https://doi.org/10.1162/tacl_a_00638.

4.12 Exercises

Unless a problem states otherwise, use the reference repair harness and the hypothetical prices in table 12. Token counts are for complete serialized requests, prices are problem inputs rather than any provider’s schedule, and a reused prefix is assumed to have compatible cached state available.

Table 12: Givens for the exercises: The window and step budget come from the registry’s repair-harness profile; every price is hypothetical.

Quantity	Symbol	Given
Context window of the repair harness	\(\ContextWindow\)	128,000 tokens
Step budget of the repair harness	\(\TurnHorizon\)	30 turns
Uncached input price		$3 per million tokens
Reused-prefix input price		$0.30 per million tokens (10% of uncached)
Output price		$15 per million tokens

1. Tool catalogs and the output reserve (Basic). The repair harness gains a larger tool catalog and longer patch outputs. Its tool declarations grow to 30,000 tokens, and the runtime raises \(\OutputLimit\) to 16,000 tokens. Application instructions stay at 3,000 tokens and task state at 24,000, and every tool result is appended at 4,000 tokens.

1. Using equation 1, compute the input limit, the observation budget, the number of appended turns that fit, and the largest per-turn digest that carries the full step budget \(\TurnHorizon\).
2. A proposal loads tool declarations on demand, the on-demand staging that Static and on-demand staging weighs against static staging, and keeps only 6,000 tokens of core declarations resident. Repeat (a).
3. Show that no tool-declaration policy lets unmodified 4,000-token results be appended for all \(\TurnHorizon\) turns under the 16,000-token reserve. Which term must the design change instead? Compare the per-turn digest each declaration policy allows, and state what the on-demand policy adds to a request whenever a turn needs a non-core tool.

2. When appending stops being cheaper (Basic). Every request carries a fixed 39,000 tokens of instructions, declarations, and task state. The appending policy adds \(\TurnAppendLength = 4{,}000\) tokens of tool results per turn and keeps them all; the selecting policy keeps \(\ObservationTokenCount = 8{,}000\) tokens of observations on every call. Assume no prefix reuse and a window large enough to admit every request.

1. Write the cumulative input tokens of each policy over \(\TurnHorizon\) turns. At which call does the appending policy’s single request first exceed the selecting policy’s, and at which horizon does its cumulative bill first exceed?
2. A longer task runs a 60-turn step budget. Compute both cumulative bills in tokens and at the uncached input price, and their ratio.
3. Show that the ratio of the two bills is exactly linear in \(\TurnHorizon\) and find its slope. At what horizon does appending cost three times as much as selecting? Use the result to critique a plan to “add selection later, once tasks get long.”

3. Look-alike evidence versus bulk (Basic). In the toy head of equation 2, the target key has logit \(\TargetKeyLogit = 6\). The request holds 40,000 unrelated keys at logit \(-2\) and a superseded 850-token r17 excerpt whose keys share identifiers with the evidence, giving each a logit of 2. All logits stay fixed as keys are added or removed.

1. Compute \(\TargetKeyAttentionWeight\) without the excerpt and with it.
2. How many unrelated keys contribute as much to the normalizer as the 850 excerpt keys?
3. The runtime can either drop half of the unrelated keys or drop the excerpt. Compute \(\TargetKeyAttentionWeight\) for each choice, and for dropping every unrelated key while keeping the excerpt. Which removal should the runtime make first, and which measurement from section 4.8 would confirm the choice on a trained model?

4. Choosing a watermark gap (Intermediate). A noisier test suite raises arrivals to \(\TurnAppendLength = 6{,}000\) tokens. Compaction returns the request to \(\CompactionTargetLength = 47{,}000\) tokens: a 15,000-token root, a 24,000-token trunk, and an 8,000-token leaf. The output reserve is \(\OutputLimit = 8{,}000\). Two triggers are proposed, \(\CompactionTriggerLength = 92{,}000\) and \(\CompactionTriggerLength = 68{,}000\). Model each cycle as \(\CompactionCycleTurnCount\) ordinary calls at request lengths \(\CompactionTargetLength + \CompactionCycleTurnIndex \, \TurnAppendLength\) for \(\CompactionCycleTurnIndex = 1, \ldots, \CompactionCycleTurnCount\), the last of which triggers a compaction pass over the same request. The first ordinary call after a pass reuses only the root, because the pass rewrote the trunk; every later call reuses the whole previous request. Each ordinary call generates 1,000 output tokens. The compaction pass processes its entire input uncached and generates a 6,000-token record.

1. For each trigger, compute \(\CompactionCycleTurnCount\) from equation 3, the trigger length and overshoot, whether the trigger request plus \(\OutputLimit\) fits the window, and the number of passes within \(\TurnHorizon\).
2. Compute \(\CompactionPassCost\), the mean ordinary-call cost, and \(\AmortizedTurnCost\) from equation 4 for each trigger.
3. Each pass independently loses each of 10 unprotected constraints with probability 0.02. Compute the probability that all 10 survive every pass of the step budget under each trigger.
4. Choose a trigger and explain why the narrow gap loses on both cost and retention even though each of its passes is cheaper. What measured property of the workload would justify the narrow gap anyway?

5. Replace now, or ride the next rewrite? (Intermediate). Start from the request of notebook 4.4: 47,000 tokens behind a 15,000-token root, with an 850-token r17 excerpt opening the trunk, a 24-token replacement marker, and a 128-token appended notice. Twenty calls remain, and each later call reuses the whole previous request; the new leaf content is the same under both policies and can be ignored.

1. Compute the input cost of the next call under replacement and under an appended notice.
2. How many more tokens does the appended policy keep resident on every later call, what do they cost per call at the reused-prefix price, and after how many calls would the two policies break even on price alone?
3. A compaction pass is scheduled six calls from now and will rewrite the trunk regardless. Propose an invalidation policy that uses this fact, and state the condition under which it must replace the excerpt immediately anyway. Using equation 7, identify which of the next six decisions belong in the denominator and how the policy keeps the numerator at zero.

6. Paying for a protected-field check (Intermediate). A model-written progress record loses each of 12 required constraints independently with probability \(\CompactionLossProbability = 0.03\) per compaction round, and the trajectory runs 3 rounds. A lost constraint makes the task unacceptable; with every constraint intact, \(\AcceptanceProbability = 0.6\). An attempt costs $2. A protected-field check in the style of listing 2 detects 90 percent of losses and repairs each detected loss; budget it as one $0.05 correction call per round.

1. Without the check, compute the probability that all 12 constraints survive, the resulting \(\AcceptanceProbability\), and \(\CostPerAcceptedTask\).
2. Repeat with the check, treating undetected losses as the new per-round loss.
3. What per-round loss would an unchecked summarizer need to keep all-constraint survival at or above 95 percent over 3 rounds?
4. With the check in place, after how many rounds does all-constraint survival first fall below 50 percent? The runtime in section 4.6 copies contract constraints directly from the task contract. What does that change, and which protected fields remain exposed to \(\CompactionLossProbability\)?

7. A stable root for short sessions (Advanced). The support agent serves 50,000 sessions a day. Each request carries a 20,000-token root of policy instructions and tool declarations and a 2,000-token customer record. A session makes 6 calls; before call \(\TurnIndex\) the request has grown by 1,500 tokens per turn to \(22{,}000 + 1{,}500\,\TurnIndex\) tokens, and each call generates 300 output tokens. Sessions arrive often enough that compatible root state is always resident. Layout A places the customer record and a session timestamp ahead of the root; Layout B places the root first. Within a session, every call after the first reuses the whole previous request under either layout.

1. Compute the input and output cost per session, the daily bill, and the token-weighted prefix hit rate under each layout.
2. Show that the entire difference arises on the first call of each session. Which term of equation 1 binds here, and how does that differ from what binds the repair agent over its step budget?
3. A developer templates the customer’s name into the first line of the policy instructions to personalize the tone. Which layout does that recreate, and what does it cost per day?
4. A cost review proposes cutting the root to 10,000 tokens by removing policy text. Under Layout B, compute the saving per session. With a 90 percent resolution rate at the full root, and each unresolved session escalating to a human at $4, how many percentage points of resolution can the cut lose before it costs more than it saves? What evidence should the proposal present before it ships?

8. Freshness through a human approval (Advanced). The remediation agent decides on a production change from 5 observations of shared state. Other actors change each observed resource independently, as a Poisson process with mean rate \(\ResourceChangeRate\) of one change per 20 minutes. The runtime refreshes an observation only when it is older than a 10-minute time-to-live, so at decision time each observation’s age is uniformly distributed between 0 and 10 minutes.

1. Compute the probability that one observation still describes its resource at decision time and the probability that all 5 do. If every decision requires current evidence for all 5, estimate \(\StaleSupportRate\) from equation 7.
2. A teammate proposes refreshing all 5 observations immediately before each model call. The call takes 30 seconds, and the action executes after a human approval that takes 10 minutes. Compute the probability that at least one of the 5 resources changes between refresh and execution, first ignoring the approval and then including it. What does the result show about the limit of any context-refresh policy for this workload?
3. Only one observation, the configuration version the change edits, determines whether the action is safe; the other four were diagnostic. Redesign the policy around the binding \(\ObservationBinding(\TrajectoryObservation_{t})\) of equation 5 and an action-time precondition at the tool boundary that re-observes and re-requests approval when the precondition fails. Compute the probability that the precondition holds at execution and the expected number of approval rounds, and compare with a precondition on all 5 observations. Contrast the change signal available here with the one the repair agent uses after its own patch.

What’s Next: From selected tokens to physical attention state

The runtime has chosen the token sequence for the next call, and the prefix-layout calculation priced its reused prefix as a discounted line on the bill. Behind that discount sits physical state. The serving system must decide whether to retain the attention state computed from earlier compatible prefixes, and holding it occupies accelerator memory even while the agent generates nothing and waits on a tool, memory that competing requests could use. KV Cache Management sizes that state and develops the storage, sharing, and retention decisions for it. Those decisions can change execution cost without changing which observations belong in the logical working context.