⬡ API & SDK/2026-08-01Advanced

Swapping Tools Mid-Conversation Without Losing Your Prompt Cache

Tools can now be added and removed between turns, but the tool block sits at the very front of the cache prefix. I measured 12 mutation patterns with fingerprints and built a stable-core plus volatile-tail registry with a guard that refuses unsafe changes.

Claude API¹²¹ Prompt Caching⁶ Tool Use⁹ Architecture³ Cost Optimization⁹

✦ Premium Article

I was trimming unused tools from a long-running session.

Twelve tools were still attached, several of which clearly would not be called again in the back half of the conversation. The beta for adding and removing tools between turns had landed, so I dropped four of them. Input tokens should have gone down.

They did not. They went up.

cache_read_input_tokens fell to zero, and the entire accumulated conversation history was being resent. What I thought was a savings had quietly invalidated the cache and made me pay for everything again.

The tool block sits at the very front

Prompt caching works on prefix matching. Everything from the start of the request up to the first divergence is eligible for a cache hit. Change one byte anywhere, and everything downstream misses.

The trouble is where tool definitions live in that ordering.

Position	Block	Expected churn	Blast radius
1	`tools`	Should be low	All of system and messages
2	`system`	Low	All of messages
3	`messages` (history)	Append-only	Later turns
4	`messages` (current)	Every turn	None

The tool array has the widest blast radius in the whole request. It does not matter whether a hundred turns have piled up behind it — swap one tool and all hundred turns go with it.

The beta unlocking mid-conversation tool changes and those changes being cache-safe are two separate things. The API accepting the mutation does not mean the prefix survives it.

I had conflated the two. Reading the beta note, I took "prompt cache is maintained" to mean maintained across any change at all.

Measuring 12 mutations by fingerprint

Rather than reason about it, I measured it. Build a rolling hash chain over the tool array — one hash per prefix length — then compare the chains before and after a change.

# toolprefix.py — measure prefix fingerprints over a tool array
import hashlib
import json
 
def canon(obj):
    """Deterministic serialization: sorted keys, fixed separators."""
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
 
def prefix_chain(tools):
    """Return the rolling hash after each successive tool."""
    chain, h = [], hashlib.sha256()
    for t in tools:
        h = h.copy()               # without copy(), later updates bleed backwards
        h.update(canon(t).encode())
        chain.append(h.hexdigest())
    return chain
 
def shared_prefix_len(a, b):
    """How many leading elements two hash chains agree on."""
    n = 0
    for x, y in zip(a, b):
        if x != y:
            break
        n += 1
    return n

That h.copy() matters. Without it you keep mutating a single hasher and every entry ends up as the final digest. I spent an evening staring at a meaningless 100% match before spotting it.

The baseline is five tools shaped like a real setup.

def mk(name, props):
    return {
        "name": name,
        "description": f"{name} tool",
        "input_schema": {
            "type": "object",
            "properties": props,
            "required": sorted(props.keys()),
        },
    }
 
BASE = [
    mk("search_docs", {"query": {"type": "string"}, "top_k": {"type": "integer"}}),
    mk("read_file",   {"path": {"type": "string"}}),
    mk("write_file",  {"path": {"type": "string"}, "body": {"type": "string"}}),
    mk("run_tests",   {"suite": {"type": "string"}}),
    mk("post_report", {"channel": {"type": "string"}, "text": {"type": "string"}}),
]
 
base_chain = prefix_chain(BASE)
for label, mutate in CASES:
    n = shared_prefix_len(base_chain, prefix_chain(mutate(BASE)))
    print(f"{label:<26} {n}/{len(BASE)}  {n / len(BASE):.0%}")

Here is what twelve patterns produced on my machine.

Mutation	Shared prefix	Retention	Verdict
Append one tool at the tail	5/5	100%	Safe
Rebuild with identical content	5/5	100%	Safe
Remove the last tool	4/5	80%	Partial
Edit the last tool's description	4/5	80%	Partial
Add an optional property to the last tool	4/5	80%	Partial
Insert a tool in the middle	2/5	40%	Partial
Remove a middle tool	2/5	40%	Partial
Rename a middle tool	2/5	40%	Partial
Swap the second and third tools	1/5	20%	Partial
Prepend one tool at the head	0/5	0%	Total loss
Remove the first tool	0/5	0%	Total loss
Edit the first tool's description	0/5	0%	Total loss

Two of twelve preserved the prefix completely. Average retention was 48%.

The "Partial" rows are the trap. A 4/5 prefix match is worth nothing unless a cache_control breakpoint sits exactly at that boundary. Cache granularity is the breakpoint-delimited segment, not the individual tool.

If your only breakpoint is at the end of the tool array — the common setup — then only the two "Safe" rows actually survive. The other ten collapse to zero whether they scored 80% or 20%.

That was precisely my failure. When I trimmed unused tools, post_report had been sitting in the middle of the array.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦12 tool mutations measured by prefix fingerprint: only 2 of 12 preserved the prefix completely, average retention 48%

✦36 logically identical tool definitions produced 36 distinct fingerprints under plain JSON serialization — a 100% false-invalidation rate

✦Measured: the core split rescues only 3.69% of the bytes that actually move, and freezing the tool block breaks even against it at one mutation per fifteen turns

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Identical content, different fingerprint

The second finding was the one that unsettled me.

Two tool definitions can be logically identical and still fingerprint differently, purely because of the order in which they were assembled. Python dictionaries preserve insertion order, so the sequence in which you populate properties changes what json.dumps emits.

Condition	Assembly orderings	Plain `json.dumps`	Canonicalized
Field order and property order both vary	36	36 distinct fingerprints	1
Field order fixed, property order varies	6	6 distinct fingerprints	1

All 36 orderings hashed differently. The false-invalidation rate was 35/35 — 100%.

This is not a hypothetical. If you assemble tool definitions with conditionals — add this property for admins, include that field in debug mode — then branch ordering decides how the tool serializes. Your logs will insist the tools are unchanged while the cache quietly dies.

The fix is one line.

json.dumps(tool, sort_keys=True, separators=(",", ":"), ensure_ascii=False)

sort_keys=True pins key order, separators pins whitespace, and ensure_ascii=False keeps output stable for tools carrying non-ASCII descriptions. The important part is sending the canonicalized dict, not merely hashing it. Canonicalizing only the fingerprint while shipping the raw dict changes nothing.

Stable core, volatile tail

That settled the design. Stop treating the tool array as one slab and split it in two.

Stable core: never changes for the life of the session, with the cache_control breakpoint on its last element
Volatile tail: everything after the core, where adds, removes, and reorders are all fine

As long as the core is untouched, anything done in the tail leaves the prefix up to the breakpoint intact. Changing "remove the tool" into "move it to the tail, then remove it" contains the blast radius.

import hashlib
import json
from dataclasses import dataclass
 
class ToolPrefixViolation(Exception):
    """Raised when the core would change mid-session."""
 
def canon(o):
    return json.dumps(o, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
 
@dataclass(frozen=True)
class ToolSegments:
    core: tuple   # immutable for the session; breakpoint goes on its last element
    tail: tuple   # free to mutate
 
    @property
    def core_fingerprint(self) -> str:
        h = hashlib.sha256()
        for t in self.core:
            h.update(canon(t).encode())
        return h.hexdigest()[:16]
 
    def to_request_tools(self) -> list:
        # ship the canonicalized dicts; hashing them alone accomplishes nothing
        tools = [json.loads(canon(t)) for t in self.core]
        tools[-1]["cache_control"] = {"type": "ephemeral"}   # breakpoint at core tail
        tools += [json.loads(canon(t)) for t in self.tail]
        return tools

Exactly one element in the returned array carries cache_control: the last tool of the core. That is the segment boundary. However many tools hang off the tail, nothing before that boundary moves.

A guard that refuses core edits

Discipline that can be broken will be broken, so I made the core hard to touch.

class ToolSetSession:
    """Tool configuration for one session, with the core fingerprint pinned."""
 
    def __init__(self, segments: ToolSegments):
        self._pinned = segments.core_fingerprint
        self.segments = segments
 
    def mutate_tail(self, new_tail) -> ToolSegments:
        nxt = ToolSegments(core=self.segments.core, tail=tuple(new_tail))
        if nxt.core_fingerprint != self._pinned:
            raise ToolPrefixViolation(
                f"core changed: pinned={self._pinned} got={nxt.core_fingerprint}"
            )
        self.segments = nxt
        return nxt

Running it:

initial core fingerprint: daf5a27a5273fe97
cache_control position in request tools: [2]
core fingerprint after tail swap: daf5a27a5273fe97 -> match
core edit detected: core changed: pinned=daf5a27a5273fe97 got=8ac9bd8f8b1072d2

frozen=True and tuples are there so that segments.core.append(...) simply does not work. Left unlocked, a future version of me forgets the rule.

Raising is deliberate. Failing loudly is cheaper than degrading silently, because a cache miss is functionally correct behavior — tests will never catch it. You find out from the invoice.

Comparing across a 40-turn session

To check whether the split actually pays, I generated 40 turns of random tool operations and replayed the identical operation sequence against two strategies.

A: single segment. Each turn, collect the active tools, sort them by name, and rebuild the array — a very common pattern
B: six-tool stable core held fixed, mutations confined to the tail

The operation mix was 10 adds, 12 drops, 9 swaps, and 9 no-ops.

Strategy	Core fingerprint matches	Retention	Wasted core resend
A single segment	1/40 turns	2%	39 turns = 38,610 bytes
B stable core + volatile tail	40/40 turns	100%	0 bytes

The six-tool core serialized to 990 bytes. Strategy A resent all of it 39 times.

What drives the gap is not the split so much as A's tidy-looking "sort by name and rebuild" step. Add or drop a single tool and the sorted positions shift, moving the core. My intuition said sorting is deterministic and therefore stable. Determinism and stability under mutation are not the same property.

One caveat on the numbers: I am measuring serialized bytes of the tool block, not billed tokens. What you actually save depends on how much conversation history sits behind the core. The longer the session, the more those 990 bytes drag down with them.

Moving the tail takes the history with it

For a while after the split went in, I felt safe. The core fingerprint matched for all forty turns and the guard never fired once.

The bill did not come down the way I expected.

The reason looks elementary in hindsight. The prefix is linear. tools comes first, then system, then the accumulated conversation. A breakpoint at the end of the core protects everything up to that breakpoint — which is the core's 981 bytes and nothing more. The moment I appended one tool to the tail, the system block and every turn of history behind it stopped matching.

The segment I had been guarding was the one least worth guarding.

I re-measured across a forty-turn session, mutating the tail every third turn. Each turn adds one user and one assistant message; by turn forty the history serializes to 46,262 bytes.

Component	Bytes
Core preserved by B (981 bytes × 13 tail mutations)	12,753
`system` plus history lost by B (same 13 turns)	333,325
Share of the moved bytes that the core split actually saved	3.69%

The split rescued less than four percent of what moved. The other ninety-six percent was re-sent every time I touched the tail.

So I compared it against a setup with no tail at all (C): the tool block never changes during the conversation, unused tools stay in the array, and narrowing happens through tool_choice and through the tools themselves replying that they are not available yet.

Approach	`system` + history lost on mutation turns	Extra tool definitions carried
B — stable core plus volatile tail, mutated every 3rd turn	333,325 bytes	0
C — tool block frozen, gated with `tool_choice`	0 bytes	1,072 bytes × 40 turns = 42,880 bytes

Carrying eight unused tools to the end of the session costs 12.9% of what B was throwing away. Leaving things you do not need in the array turns out to be cheaper than removing them — which runs directly against the instinct that got me here. I resisted it for a while.

What decides it is how often you touch the tail. Same forty turns, varying only the interval:

Tail mutated every…	`system` + history lost	Against C's 42,880 bytes
2 turns	512,812 bytes	C wins by a wide margin
3 turns	333,325 bytes	C wins
5 turns	219,004 bytes	C wins
10 turns	121,068 bytes	C wins
15 turns	54,749 bytes	Roughly even
40 turns (once, effectively)	47,622 bytes	B is fine

They break even somewhere around one mutation per fifteen turns. Touch the tail more often than that and freezing the array is cheaper. Swap your toolkit once at the midpoint of a session and the split earns its keep.

Only move the things that are not carrying anything behind them. The tool block carries the entire conversation. The split limits how badly a change hurts; it is not a way to make changes free, and I had been reading it as the latter.

These are serialized byte counts, not billed tokens — a tokenizer will shift the ratios. What it will not shift is the two-order-of-magnitude gap between the segment you protect and the segment you lose, which only widens as the history grows.

What I watch in production

Running scheduled jobs as an indie developer means failures often stay invisible until the invoice arrives, and I could not reconstruct how long mine had been broken. A few habits have helped.

Have a rule for what enters the core. Anything that might be called even once belongs in the core; "probably unused" goes in the tail. When in doubt, tail. A smaller core breaks less often.
Do not assemble tool definitions inline. Keep declarations as module constants and let conditionals choose among them, never build them. The moment a branch composes the body, ordering drifts.
Log cache_read_input_tokens per turn alongside the tool fingerprint. When it drops to zero, diffing two adjacent fingerprints identifies the cause in minutes. Without it, you cannot even tell which turn broke.
Recycle the session when the tail grows. Past a certain size, rebuilding a fresh core and folding the conversation is cheaper. Tail tools are still tokens.
Prefer disabling over removing. Withholding a tool via tool_choice, or having the tool itself return "not available at this stage," is often safer than pulling it out of the array.

The third one has already earned its keep. Put the fingerprint from the broken turn next to the one before it, and the difference is obvious at a glance.

Here is how I decide per setup.

Setup	Recommendation	Why
Five tools or fewer, short conversations	Stay single-segment	Little history to protect; the bookkeeping costs more than it saves
Six or more tools, growing history, toolkit swapped once or twice per session	Stable core plus volatile tail	Worth it when mutations are further apart than roughly fifteen turns
Six or more tools, tail touched every few turns	Freeze the tool block, gate with `tool_choice`	Carrying unused definitions costs an order of magnitude less than re-sending history
Tools vary by permission	Put every candidate in the core, gate with `tool_choice`	Narrows the effective surface without moving the array
Tools generated dynamically	Concatenate generated tools into the tail in fixed order	Isolates generation-order drift from the core

Where to start

Serialize your current agent's tool array once, and hash it both with and without sort_keys=True. If those two differ, you may be losing the cache on turns where nothing changed at all.

If they match, look at breakpoint placement next — and at how many bytes sit behind it. Print which entries carry cache_control, then log the serialized size of system and the history on that same turn. Those two numbers side by side show you what you are protecting and what you are re-paying for.

Trimming tools to save context can cost more than it saves. What made it click was writing the preserved bytes and the lost bytes on the same line — until then I kept trusting the instinct over the measurement. If you have been stuck at the same spot, I hope this saves you the evening I lost.

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.