Swapping Tools Mid-Conversation Without Losing Your Prompt Cache

Identical content, different fingerprint

The second finding was the one that unsettled me.

Two tool definitions can be logically identical and still fingerprint differently, purely because of the order in which they were assembled. Python dictionaries preserve insertion order, so the sequence in which you populate properties changes what json.dumps emits.

All 36 orderings hashed differently. The false-invalidation rate was 35/35 — 100%.

This is not a hypothetical. If you assemble tool definitions with conditionals — add this property for admins, include that field in debug mode — then branch ordering decides how the tool serializes. Your logs will insist the tools are unchanged while the cache quietly dies.

The fix is one line.

json.dumps(tool, sort_keys=True, separators=(",", ":"), ensure_ascii=False)

sort_keys=True pins key order, separators pins whitespace, and ensure_ascii=False keeps output stable for tools carrying non-ASCII descriptions. The important part is sending the canonicalized dict, not merely hashing it. Canonicalizing only the fingerprint while shipping the raw dict changes nothing.

Stable core, volatile tail

That settled the design. Stop treating the tool array as one slab and split it in two.

• Stable core: never changes for the life of the session, with the cache_control breakpoint on its last element
• Volatile tail: everything after the core, where adds, removes, and reorders are all fine

As long as the core is untouched, anything done in the tail leaves the prefix up to the breakpoint intact. Changing "remove the tool" into "move it to the tail, then remove it" contains the blast radius.

import hashlib
import json
from dataclasses import dataclass
 
class ToolPrefixViolation(Exception):
    """Raised when the core would change mid-session."""
 
def canon(o):
    return json.dumps(o, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
 
@dataclass(frozen=True)
class ToolSegments:
    core: tuple   # immutable for the session; breakpoint goes on its last element
    tail: tuple   # free to mutate
 
    @property
    def core_fingerprint(self) -> str:
        h = hashlib.sha256()
        for t in self.core:
            h.update(canon(t).encode())
        return h.hexdigest()[:16]
 
    def to_request_tools(self) -> list:
        # ship the canonicalized dicts; hashing them alone accomplishes nothing
        tools = [json.loads(canon(t)) for t in self.core]
        tools[-1]["cache_control"] = {"type": "ephemeral"}   # breakpoint at core tail
        tools += [json.loads(canon(t)) for t in self.tail]
        return tools

Exactly one element in the returned array carries cache_control: the last tool of the core. That is the segment boundary. However many tools hang off the tail, nothing before that boundary moves.

A guard that refuses core edits

Discipline that can be broken will be broken, so I made the core hard to touch.

class ToolSetSession:
    """Tool configuration for one session, with the core fingerprint pinned."""
 
    def __init__(self, segments: ToolSegments):
        self._pinned = segments.core_fingerprint
        self.segments = segments
 
    def mutate_tail(self, new_tail) -> ToolSegments:
        nxt = ToolSegments(core=self.segments.core, tail=tuple(new_tail))
        if nxt.core_fingerprint != self._pinned:
            raise ToolPrefixViolation(
                f"core changed: pinned={self._pinned} got={nxt.core_fingerprint}"
            )
        self.segments = nxt
        return nxt

Running it:

initial core fingerprint: daf5a27a5273fe97
cache_control position in request tools: [2]
core fingerprint after tail swap: daf5a27a5273fe97 -> match
core edit detected: core changed: pinned=daf5a27a5273fe97 got=8ac9bd8f8b1072d2

frozen=True and tuples are there so that segments.core.append(...) simply does not work. Left unlocked, a future version of me forgets the rule.

Raising is deliberate. Failing loudly is cheaper than degrading silently, because a cache miss is functionally correct behavior — tests will never catch it. You find out from the invoice.

Comparing across a 40-turn session

To check whether the split actually pays, I generated 40 turns of random tool operations and replayed the identical operation sequence against two strategies.

• A: single segment. Each turn, collect the active tools, sort them by name, and rebuild the array — a very common pattern
• B: six-tool stable core held fixed, mutations confined to the tail

The operation mix was 10 adds, 12 drops, 9 swaps, and 9 no-ops.

The six-tool core serialized to 990 bytes. Strategy A resent all of it 39 times.

What drives the gap is not the split so much as A's tidy-looking "sort by name and rebuild" step. Add or drop a single tool and the sorted positions shift, moving the core. My intuition said sorting is deterministic and therefore stable. Determinism and stability under mutation are not the same property.

One caveat on the numbers: I am measuring serialized bytes of the tool block, not billed tokens. What you actually save depends on how much conversation history sits behind the core. The longer the session, the more those 990 bytes drag down with them.

Moving the tail takes the history with it

For a while after the split went in, I felt safe. The core fingerprint matched for all forty turns and the guard never fired once.

The bill did not come down the way I expected.

The reason looks elementary in hindsight. The prefix is linear. tools comes first, then system, then the accumulated conversation. A breakpoint at the end of the core protects everything up to that breakpoint — which is the core's 981 bytes and nothing more. The moment I appended one tool to the tail, the system block and every turn of history behind it stopped matching.

The segment I had been guarding was the one least worth guarding.

I re-measured across a forty-turn session, mutating the tail every third turn. Each turn adds one user and one assistant message; by turn forty the history serializes to 46,262 bytes.

The split rescued less than four percent of what moved. The other ninety-six percent was re-sent every time I touched the tail.

So I compared it against a setup with no tail at all (C): the tool block never changes during the conversation, unused tools stay in the array, and narrowing happens through tool_choice and through the tools themselves replying that they are not available yet.

Carrying eight unused tools to the end of the session costs 12.9% of what B was throwing away. Leaving things you do not need in the array turns out to be cheaper than removing them — which runs directly against the instinct that got me here. I resisted it for a while.

What decides it is how often you touch the tail. Same forty turns, varying only the interval:

They break even somewhere around one mutation per fifteen turns. Touch the tail more often than that and freezing the array is cheaper. Swap your toolkit once at the midpoint of a session and the split earns its keep.

Only move the things that are not carrying anything behind them. The tool block carries the entire conversation. The split limits how badly a change hurts; it is not a way to make changes free, and I had been reading it as the latter.

These are serialized byte counts, not billed tokens — a tokenizer will shift the ratios. What it will not shift is the two-order-of-magnitude gap between the segment you protect and the segment you lose, which only widens as the history grows.

What I watch in production

Running scheduled jobs as an indie developer means failures often stay invisible until the invoice arrives, and I could not reconstruct how long mine had been broken. A few habits have helped.

1. Have a rule for what enters the core. Anything that might be called even once belongs in the core; "probably unused" goes in the tail. When in doubt, tail. A smaller core breaks less often.
2. Do not assemble tool definitions inline. Keep declarations as module constants and let conditionals choose among them, never build them. The moment a branch composes the body, ordering drifts.
3. Log cache_read_input_tokens per turn alongside the tool fingerprint. When it drops to zero, diffing two adjacent fingerprints identifies the cause in minutes. Without it, you cannot even tell which turn broke.
4. Recycle the session when the tail grows. Past a certain size, rebuilding a fresh core and folding the conversation is cheaper. Tail tools are still tokens.
5. Prefer disabling over removing. Withholding a tool via tool_choice, or having the tool itself return "not available at this stage," is often safer than pulling it out of the array.

The third one has already earned its keep. Put the fingerprint from the broken turn next to the one before it, and the difference is obvious at a glance.

Here is how I decide per setup.

Where to start

Serialize your current agent's tool array once, and hash it both with and without sort_keys=True. If those two differ, you may be losing the cache on turns where nothing changed at all.

If they match, look at breakpoint placement next — and at how many bytes sit behind it. Print which entries carry cache_control, then log the serialized size of system and the history on that same turn. Those two numbers side by side show you what you are protecting and what you are re-paying for.

Trimming tools to save context can cost more than it saves. What made it click was writing the preserved bytes and the lost bytes on the same line — until then I kept trusting the instinct over the measurement. If you have been stuck at the same spot, I hope this saves you the evening I lost.