Research / Keeping pace

Keeping pace: changing the parts of an agent without losing what you know

A year after "Context as a Service", three projects, one lesson: you do not keep up with AI by reading faster, you keep up by making every change cheap to try and quick to judge.

Ezekias Bokove15 min read UrithiContext Lifecycle EngineHarness cache study

In August 2025 I wrote about Context as a Service. The idea was to build agents from versioned blocks, each one holding a model, tools, memory, a prompt and a suite of evals, and to promote those blocks from dev to prod like any other release. One scenario in that post read "a month later, a new version of an LLM is released". I wrote it as an edge case.

It is not an edge case anymore. It is a normal week. Since then I have built three systems:

In all three, the first version was never the hard part. The hard part was the next model, the next embedding model, the provider that changed its behaviour overnight, and the paper that made me rethink a design. This post is about the habits that let those systems absorb change, and the measurements that taught me each one.

What moves, and how often

Here is what changed under these projects in the last few months, without a single line of my code changing:

What moved under one agent block: six dated changes pointing at the model, the embedder and the memory design
Figure 1. Everything a block depends on keeps moving. Each red arrow is a change I did not choose and was not warned about.

Some of these are large. Two Gemini embedding models, gemini-embedding-001 and gemini-embedding-2, agree at a cosine of 0.040 on the same texts: they are not two versions of one space, they are two spaces. Others are small and sneaky: a provider that served part of my prompt from cache one day and none of it three days later.

You cannot plan for this list. You can only decide what a change costs you when it arrives. Here is the loop I ended up with.

The change loop: a change arrives, goes behind an interface, is replayed for free, then run on a small budget, frozen and compared, then promoted; drift sends it back to the start
Figure 2. The change loop. Each step below is a habit, and each habit comes from something that broke or something I measured.

Habit 1put every moving part behind an interface, and record what it was

In Urithi, every store sits behind an interface with a local implementation and contract tests any production version must pass (decision 0005). That sounds like ordinary engineering. What made it matter was the model calls.

Urithi first called Gemini through its OpenAI compatible endpoint. On a real call, that endpoint reported 2 completion tokens for a call whose total was 121: the thinking tokens were billed but not shown. A cost meter reading that response would have counted a fraction of the bill. Measured later through the provider's own SDK, wording a single question used 623 thinking tokens for a 17 token answer. So Urithi now calls each provider through its own SDK, behind the same two functions, and every call hands its billed usage to whoever listens:

apu-urithi · apu/inference/llm.py

def usage_of(response, role: str, provider: str, model: str) -> Usage:
    """Tokens billed, from the response. Output is total minus prompt: thinking included."""
    meta = getattr(response, "usage_metadata", None)
    prompt = getattr(meta, "prompt_token_count", None) or 0
    thinking = getattr(meta, "thoughts_token_count", None) or 0
    total = getattr(meta, "total_token_count", None) or 0
    answer = getattr(meta, "candidates_token_count", None) or 0
    total = total or prompt + answer + thinking
    return Usage(role=role, provider=provider, model=model, prompt_tokens=prompt,
                 output_tokens=max(total - prompt, 0), thinking_tokens=thinking,
                 total_tokens=total)

Output is computed as total minus prompt, so whatever a provider bills as thinking is counted even if it is not reported as an answer.

The second half of the habit is provenance. In the CLE, an embedder is a substrate: every trigger records the embedder_id of the space that made it, and comparing vectors from two spaces raises an error instead of returning a number. In the harness study, every frozen run carries a pinning key, the full description of what produced its numbers. This is a real one, trimmed to the fields that matter here:

{
  "run_id": "gemapi-g35lite-lf500-compact-pin-v1",
  "model": "gemini-3.5-flash-lite", "llm": "gemini", "prefix_cache": "implicit",
  "embedder": "onnx", "mode": "compact", "compact_at": 6000, "compact_keep": 4,
  "verbatim_pin": true, "verbatim_recall": false,
  "scenario": "/scenarios/long_facts_en_v1.jsonl", "scenario_sha": "819650542c17",
  "pricing_hash": "0c644146e92f", "code_sha": "83e7b89+h10",
  "cpus": 1.0, "memory_bytes": 1073741824, "date": "2026-11-16"
}

The memory switches (verbatim_pin, compact_at) were added to that key only recently. Before, two runs with different memory designs had the same key, and the dashboard reported them as "same setup". A key that misses a moving part will hide exactly the change you are trying to judge.

Habit 2a threshold belongs to its space

This one cost me months.

The CLE groups a person's recurring intents into clusters, then validates a candidate agent by replaying the person's own history: does its trigger fire on the intent it was built for, and on nothing else? On a Stack Overflow corpus, with moderator duplicates as ground truth, it grouped 32 of 48 attested intents in one cluster, against about 2% at random. But the same run reported a false trigger rate of 0.580, and for months I read that as "the detector fires on almost everything".

It was not the detector. It was one line.

131 unrelated episodes on a similarity axis: at a routing bar of 0.6, 76 fire; at the space's own 0.775, none fire
Figure 3. Two halves of one measurement read at two different bars. In this space, 0.6 sits below the median similarity between different intents (0.61), so most unrelated episodes cleared it. The counts are measured; the position of each point is illustrative.

The fix makes the bar travel with the space:

context-lifecycle-engine · cle/detect/clusters.py

# The clustering threshold is a property of the VECTOR SPACE, not a global
# default: bag-of-tokens puts same-domain text at ~0.2-0.4, a real sentence
# embedder at ~0.7-0.9, so one number cannot serve both.
CLUSTER_THRESHOLD_BY_EMBEDDER: dict[str, float] = {
    "stub:hashed64": 0.6,
    "google:gemini-embedding-2:768": 0.775,
    "onnx:embeddinggemma-300m-fp16:query:768": 0.46,
}

def cluster_threshold_for(embedder_id: str | None, default: float) -> float:
    return CLUSTER_THRESHOLD_BY_EMBEDDER.get(embedder_id or "", default)

context-lifecycle-engine · cle/build/replay.py

routing_bar = cluster_threshold_for(
    getattr(embedder, "embedder_id", None), config.cluster_similarity_threshold
)

Why did the whole offline suite never catch it? Because the test embedder, stub:hashed64, is calibrated at 0.6, the very value the broken line used. Every fixture figure was unchanged by the fix. Only a space calibrated somewhere else could reveal the defect. Adding a second, local embedding space (EmbeddingGemma 300m on CPU, 40 of 48 intents grouped, false trigger rate 0.000 to 0.023) is what made the first space's bug visible.

The harness study hit the same wall from the other side. When I added EmbeddingGemma 2 next to MiniLM for memory search, the search thresholds had to come with it, set per embedder, or the better model would have looked worse. So when you swap a model, list every number in your code that was tuned on the old one. They are part of the swap.

Habit 3proof expires

The CLE builds each agent into an image and freezes a fingerprint of the model it was validated on: twelve probes drawn from the user's own history, and a hash of each output. A re-validator replays them against whatever model is served now:

context-lifecycle-engine · cle/lifecycle/revalidator.py

current_output_hashes = tuple(
    content_hash(output) for output in fingerprinter.outputs(image.probe_set)
)
# `zip_longest`, not `zip`: a probe whose output went missing, or an output
# with no frozen counterpart, is a probe that moved.
probe_deltas = tuple(
    f"probe-{index}"
    for index, (frozen, current) in enumerate(
        zip_longest(image.probe_output_hashes, current_output_hashes)
    )
    if frozen != current
)
oplog.emit("revalidation_failed" if probe_deltas else "revalidate", ...)

A failed re-validation can demote an agent, never promote one. The comment about zip_longest is itself a lesson: with zip, a truncated answer changed the fingerprint while naming no probe that moved, and the proof was reported as holding.

The harness study shows why proof has to expire. I ran the same 1500 turn conversation, with 24 personal facts hidden in everyday chatter, on four models. Here is the share of the prompt the provider served from its cache, for the same memory design:

Share of the prompt served from the provider's cache: GLM-5.3 96.5%, DeepSeek V4 Flash 96.0%, Gemini 3.5 Flash-Lite 58.4%, Nemotron 3 Super 1.6 to 3.0%
Figure 4. Same harness, same prompts, same conversation. On Nemotron 3 Super the summary runs never finished: they stopped between turns 1142 and 1227 of 1500, and their reruns failed at the first call when my Nebius quota ran out. Its bar shows the extracted memory runs, which completed all 1500 turns: 1.6 to 3.0% from cache, at the same provider as GLM-5.3.

The cheapest way to remember moved with the model:

model running summary extracted notes, written only for statements cheaper
DeepSeek V4 Flash (Nebius) 0.131 USD, 24/24 facts 0.173 USD, 24/24 summary
GLM-5.3 (Nebius) 3.421 USD, 24/24 3.126 USD, 24/24 notes

Table 1. One run per cell for GLM-5.3, so read the gap as a direction, not a size. Source: docs/measured/decisive-1500.md and docs/measured/models-1500-glm53.md, each with its reproduce command.

And a setting that was safe on one model was not safe on another: extracting memory once every two exchanges lost facts with GLM-5.2 (8 of 13 kept) and GLM-5.3 (11 of 13), where extracting every turn kept them all. A result is a fact about a model, a provider and a day. When any of the three moves, the result is a hypothesis again.

Habit 4re-measure, never relabel

For months, every CLE benchmark ran in gemini-embedding-001, because BigQuery could not host the model the engine actually uses. On October 5, 2026, BigQuery made gemini-embedding-2 available, and the two surfaces finally shared one space:

remote model                 raw norms             cosine against the engine's cache
emb_gemini_embedding_2       1.000000 to 1.000001  median 1.000000, 20/20 identical to 1e-6
emb_gemini_embedding_001     0.582974 to 0.599283  median 0.040290, 0/20

The tempting move was to keep the published figures and update their labels. I re-ran them instead. Some held. Some did not:

published claim in gemini-embedding-001 in the engine's own space
raw text beats facets (cross-user bench) 15.0 points 5.7 points
cost of the facet boundary, strict 12.0 points 14.5 points
amplification on real WildChat users 7x between 7x and 33x, depending on the space

Table 2. The direction held in every case. The size did not. Source: CLE docs/findings/, each figure with its pinning key and reproduce command.

"Raw text beats facets by 15.0 points" was a sentence I had repeated in talks. In the space the engine runs on, it is 5.7. The old number was not wrong; it was true somewhere else. So the rule in both repositories is the same: a number carries the key of the setup that produced it and the command that reproduces it, and a new measurement is added beside the old one, never over it.

Habit 5keep a cheap bench of your own usage, so a new idea gets a verdict in days

Reading a paper does not tell you whether it helps your agent. Measuring it does, and that is only realistic if measuring is cheap. Here is a full cycle, from reading to verdict.

Timeline from November 11 to 16: read the post, build pinned statements, free replay, deliberate defects, 500 turns on Flash-Lite, one variant rejected, 1.41 USD spent
Figure 5. Five days and about 1.4 USD from a research post to a measured answer.

The idea came from Anthropic's post on the components of agent memory. In one of their tests, several agents summarising contract terms into one shared table kept 41 of 57 graded facts; one agent per contract, quoting the terms word for word, kept all 57. My harness folds old conversation into a running summary, so I tried the obvious version: at each fold, copy the user's own statements of fact, word for word, into a small block after the summary.

harness cache study · src/harness_cache/engine.py (branch consolidation-ideas)

def pin_statements(pinned: dict[str, int], rows: list[Message], first_turn: int) -> dict[str, int]:
    """H10: the folded user messages that state a fact, word for word. A statement said again
    moves to its latest turn, so the newest value of a corrected fact is read last."""
    out = dict(pinned)
    for i, m in enumerate(r for r in rows if r.role == "user"):
        if states_a_fact(m.text):
            out.pop(m.text, None)
            out[m.text] = first_turn + i
    return out

No model call, no search: it runs on the cold path, while nobody waits, and the block changes only when a fold happens, so the start of the prompt stays cacheable.

Step one, free. I replayed the 1500 turn scenario with a stand-in model that answers instantly and costs nothing. It cannot tell me whether answers are right, but it can tell me whether the right sentence reaches the prompt:

arm (stand-in model, 1500 turns) fact's sentence in the prompt, word for word turns that got a search quote prompt tokens memory time per turn, median / p95
summary only 4/24 0 4.95 M 1.9 / 4.2 ms
summary + search over past messages (MiniLM) 24/24 1273 5.26 M 3.9 / 57 ms
summary + search (EmbeddingGemma 2) 24/24 1273 5.26 M 4.1 / 170 ms
summary + pinned statements 24/24 0 5.92 M 1.9 / 3.4 ms

Table 3. The pinned block puts every fact in the prompt with no search and no latency tail, for about 20% more prompt tokens than the summary alone. Source: docs/measured/verbatim-pin-1500.md.

Step two, about 1.4 USD. Then 500 turns on a real model:

arm (Gemini 3.5 Flash-Lite, 500 turns) facts recalled cost prompt tokens served from cache
summary only 23/24 0.455 USD 1.84 M 32.9%
summary + pinned statements 24/24 0.478 USD 2.08 M 36.2%
pinned, kept out of the summary 24/24 0.479 USD 2.07 M 36.0%

Table 4. Source: docs/measured/verbatim-pin-flashlite-500.md, one run per arm.

Two lanes over 150 turns: the sentence said at turn 37 is folded into the summary by turn 114; at turn 148 the summary-only agent answers that it has no note, the pinned agent answers July 9
Figure 6. The same sentence in two runs. The answer in red is quoted from the summary-only run.

The summary alone lost one fact. "My sister's birthday is on July 9" was said at turn 37, folded into the summary, and gone by turn 148, when the model answered that it had no note about it. With the pinned block, the sentence was in the prompt and the fact came back, for about 5% more on the bill: 13% more tokens, but the block sits at the start of the prompt, where part of it is read from cache.

The variant was rejected. Leaving pinned sentences out of the summary did not shrink it; its median grew from 453 to 545 estimated tokens. The model filled the space with something else.

This is one run per arm, so that birthday is an observation, not a rate. The compact arm also ran on a different API key and day than the pinned arms, so the cache share is not attributable to the design alone. Those limits are written into the figure itself, next to the numbers.

A bench also tells you when nothing changed, and that is a result too. In Urithi I compared two speech recognition models on ten clips, then on ten French questions carrying 22 numbers. Both kept 16 of 22, failing on the same three clips in the same places. An error both models make on the same clip is the clip, not the model. Nothing separated them, and saying otherwise would have been inventing a result.

And sometimes the bench says the obvious improvement is a regression. Urithi ranks the parts of a knowledge by meaning, and fusing that with word matching (BM25, reciprocal rank fusion) looked like a free gain. On 38 questions written before anything was built, it hurt the French ones: 8 of 12 found first, against 10 of 12 by meaning alone. Short words matched by chance across languages, French "on" for English "on". Counting words only when the question is in the knowledge's language gave the best of both: 32 of 38 found first, against 30 before.

Habit 6make sure your tests would notice

A swap is only safe if a mistake in it would turn something red. Coverage does not tell you that. Deliberate defects do: change the code on purpose and check that a test fails. Urithi's v1 caught 285 of 285 such defects before release, run in parallel copies of the repository. In the harness study, the pinned statements were not called done until every hand-made defect on the new lines was caught. Three survived at first, and each pointed at a test that checked too little: one never verified that the score counted only sentences that state a fact, one never verified that the assistant's replies still reached the summary, and one never verified that the dashboard actually showed the new score.

The CLE adds a rule I now use everywhere: a non-measurement is never a verdict. A check that cannot run in the configured space reports unavailable, and the gap is disclosed instead of passing silently.

Back to the block

So what would I change in Context as a Service today? The structure holds. A block still has a model, tools, a prompt and evals, and promotion still goes from dev to prod. What changes is what the block declares, and what its evals are allowed to conclude.

The Context Block in 2025 and today: Model gains a fingerprint, Tools gain cache classes, Memory becomes Harness, Evals re-run on drift
Figure 7. The block grows a harness section, and its evals stop being a gate passed once.
Context as a Service, 2025 The block I would write today
Memory a scope (short term, conversation) an architecture with measured trade-offs, and the embedder that serves it, with its exact revision and its own thresholds
Evals pass once to lock a version recall, latency, cost and the provider's cache share, re-run whenever the model, the embedder or the provider changes
Promotion a button a bench on the user's own history that has to agree
latest the default for continuous delivery only where a re-run happens before the label moves

The last line is the one I got most wrong. "No need to touch the code, it will use the latest version" is exactly how a better model ends up behind a worse threshold.

What I do not know

These habits come from three systems built mostly around one person's usage and generated scenarios, often with one run per setup. They kept those projects honest; I have not measured them on a team or across organisations. Some things I still cannot measure well: the CLE validates when an agent should trigger, never the quality of its answer, and the harness study scores recall by matching strings in the answer. The pinned statements code is on a branch, not yet merged.

The short version

You cannot keep up by reading faster. You keep up by making each change cheap to try and quick to judge:

  1. every moving part behind an interface, with a recorded identity;
  2. thresholds that travel with their space;
  3. proof that expires when the model under it moves;
  4. figures that are re-measured, never relabelled;
  5. a small bench of your own usage, free first, then cheap;
  6. tests that would notice a mistake.

Then a new model, a new provider rule or a new paper is not a rewrite. It is a run.


Code and measurements: Urithi (apu-urithi), the Context Lifecycle Engine and the harness cache study. Every number in the tables comes from a figure generated from frozen runs, with the command that reproduces it.

Context Lifecycle Engine Urithi, step by step Context as a Service, 2025