When Prompt Caching Was On but the Bill Didn't Move — Field Notes on Instrumenting cache_read to Close the Hit-Rate Gap

A low hit rate usually breaks into three causes

After running the probe for a few days, the misses clearly came in three flavors. Viewed together they're a fog; separated, the fix for each gets concrete.

1. Non-deterministic prefix — I had a timestamp up front

This was my culprit. I was injecting the current time at the very start of the system prompt, which made the prefix different on every request. Change one character and the cache treats it as a new prefix.

# What I was doing wrong — the head changes every call
system = [
    {
        "type": "text",
        "text": f"The current time is {datetime.now().isoformat()}. You are a summarizer...",
        "cache_control": {"type": "ephemeral"},
    }
]
 
# Fixed — move the variable part outside the cache boundary
system = [
    {
        "type": "text",
        "text": "You are a summarization assistant. ...(stable instructions and knowledge)...",
        "cache_control": {"type": "ephemeral"},   # identical every time up to here
    },
    {
        "type": "text",
        "text": f"The current time is {datetime.now().isoformat()}.",  # outside = not cached
    },
]

The easy-to-miss part isn't the obvious variables like timestamps. If you embed a dict or set via json.dumps, key ordering can wobble between runs and shift the prefix. When you embed, pass sort_keys=True to pin the order before placing it inside the boundary. That alone cleared out a good share of the "same content, still misses."

2. TTL expiry — the cache quietly disappears after five minutes

The standard lifetime of an ephemeral cache is about five minutes. Five minutes after the last access, the prefix is evicted, and the next request starts over with a write (a miss). For a sparse batch, this is where the cost hides.

The split is measurable. If you also record the seconds since the previous request on any call where cache_creation fired, expiry misses cluster on the "long gap" calls.

def classify_miss(gap_seconds: float, read: int, write: int) -> str:
    """Bucket a miss into a cause so you can pick a fix."""
    if read == 0 and write == 0:
        return "under_threshold"          # 3. sub-threshold
    if write > 0 and gap_seconds > 300:
        return "ttl_expired"              # 2. likely TTL expiry
    if write > 0:
        return "prefix_changed"           # 1. likely non-deterministic prefix
    return "hit"

Once expiry was the main driver, I had two options: tighten the call spacing to keep the cache warm, or switch to the one-hour extended cache (write is ~2x but it lives longer). I reshaped the batch to submit in 5-minute clusters instead of sprinkling calls out. Rewrite rate fell from 61.2% to 18.4%.

3. Sub-threshold — short prompts simply aren't cached

If the tokens up to the breakpoint don't meet the minimum, no cache is created at all. If read and creation both sit at 0 forever, suspect this first.

I had cache_control on a short instruction block — exactly this. Pulling the knowledge base inside the boundary so the prefix clears the minimum fixed it. The work here isn't trimming; it's gathering stable content inside the boundary.

When cache_creation won't die down, walk the boundary back one block at a time

If writes still flicker after clearing all three causes, some wobble remains somewhere in the prefix. I moved the boundary like a binary search to corner the source.

The procedure: narrow the breakpoint to just the most stable leading block and measure the rewrite rate. Once it settles, pull the next block inside the boundary and re-measure. The block that spikes the rewrite rate is the source of the wobble.

def build_system(blocks: list[str], cache_upto: int) -> list[dict]:
    """Bundle the first `cache_upto` blocks into one stable prefix."""
    out = []
    for i, text in enumerate(blocks):
        part = {"type": "text", "text": text}
        if i == cache_upto - 1:              # one boundary on the last bundled block
            part["cache_control"] = {"type": "ephemeral"}
        out.append(part)
    return out
 
# Increase cache_upto 1 -> 2 -> 3 while watching rewrite_rate;
# stop just before it jumps. That edge is the end of your stable prefix.

The key thing here is that more breakpoints is not better. You get up to four, but in my measurements, bundling the stable head into one or two produced a lower rewrite rate than slicing it finely. Every extra boundary adds another "must match exactly up to here" condition, and if any one shifts, everything after it collapses together. Order least-changing content first: system (stable instructions) → tools → messages (old history) → newest message settled out best.

Keep the hit rate on a dashboard

Even after a fix, the cache drifts again, quietly. So I stopped treating the probe as a one-off and started streaming rewrite rate and hit rate into monitoring. With a series of numbers, degradation shows up before the invoice does.

def export_prometheus(probe) -> str:
    """Emit CacheProbe state in Prometheus format."""
    return (
        "# TYPE claude_cache_hit_rate gauge\n"
        f"claude_cache_hit_rate {probe.hit_rate:.2f}\n"
        "# TYPE claude_cache_rewrite_rate gauge\n"
        f"claude_cache_rewrite_rate {probe.rewrite_rate:.2f}\n"
        "# TYPE claude_cache_read_tokens counter\n"
        f"claude_cache_read_tokens {probe.read_tokens}\n"
    )

If I set only one alert, I put a threshold on the rewrite rate. It stirs before the hit rate falls. On the summarization batch I alert above 15%, and it caught a regression the day after I touched the prompt.

How far it dropped, and where I draw the line

The final numbers on the batch: hit rate 34.8% → 88.6%, rewrite rate 61.2% → 6.1%, effective input-token unit cost down to about 41%. That falls short of the "up to 90% off" headline. I think falling short is normal. A workload where every request is a clean cache hit rarely exists in practice.

Where I landed: caching isn't a setting you flip on or off — it's an operational metric you keep watching, called the hit rate. When the default model rolls over, when you add one line to a prompt, when you change call spacing, the prefix can quietly drift. So the real star of optimization wasn't clever placement; it was the boring habit of glancing at cache_read every time.

When the rewrite rate starts creeping up, that's the signal a prefix has begun to wobble somewhere. With this harness in place, you catch it without waiting for the invoice. For my part, the next time I swap the default model, I plan to watch these numbers for a few days before leaning on it in production.

Measure, name it, move one boundary. That loop is what carried caching from "I thought it was on" to "it's actually working." If it helps someone else's hand pause a little sooner over the same invoice, I'll be glad. Thank you for reading.