⬡ API & SDK/2026-07-19Advanced

When Prompt Caching Was On but the Bill Didn't Move — Field Notes on Instrumenting cache_read to Close the Hit-Rate Gap

You added cache_control but your Claude API bill barely changed. Before guessing at fixes, record cache_read and cache_creation on every request and split the hit-rate gap into three causes: non-deterministic prefixes, TTL expiry, and sub-threshold blocks.

prompt-caching¹⁵cache-hit-ratecost-optimization³¹ observability²¹ api³⁹

✦ Premium Article

I opened last month's invoice and paused. I had added cache_control to my system prompt three weeks earlier. I never checked the hit rate — I just assumed "that's a 90% discount now." But the input-token charges had barely moved.

That was the moment it finally landed: caching isn't a binary of on or off. It was on. It just wasn't being read. And that gap only shows up in the invoice — you cannot see it by staring at the code.

These are my notes from raising the cache hit rate on a summarization batch I run as an indie developer, from the low 30s to nearly 90%. This isn't a tour of clever optimization tricks. Measure first, name the leak, then move one breakpoint at a time. It's a record of that unglamorous back-and-forth.

"Enabled" and "working" are two different states

Prompt caching stores the prefix — the leading portion of system, tools, and messages — on the server. When a later request reuses the same prefix, that portion is billed at roughly a tenth of the normal input rate.

The catch is that writing cache_control guarantees nothing at the moment you write it. Whether the cache was actually read only appears in the response usage. Not looking there was the whole shape of my three-week blind spot.

usage field	Meaning	Billing (relative to normal input)
cache_creation_input_tokens	Tokens written to the cache (this call missed)	~1.25x (5-minute cache)
cache_read_input_tokens	Tokens read from the cache (this call hit)	~0.1x
input_tokens	Normal, non-cached input	1x

Reading it is simple. If cache_read_input_tokens comes back non-trivial on most calls, caching is working. If cache_creation_input_tokens keeps firing instead, the cache is being built and thrown away over and over. "Added but not working" is almost always this state.

Add one thin measurement layer first

Before I started guessing at causes, I inserted a thin layer that records usage on every request. Ahead of any optimization, I wanted to know where I actually stood, in numbers.

import logging
from dataclasses import dataclass, field
 
logger = logging.getLogger("cache")
 
@dataclass
class CacheProbe:
    """Minimal harness that logs cache_read / cache_creation per request."""
    requests: int = 0
    hits: int = 0            # count of calls where cache_read > 0
    rewrites: int = 0        # count of calls where cache_creation > 0
    read_tokens: int = 0
    write_tokens: int = 0
    last_prefix_fingerprint: str | None = field(default=None)
 
    def record(self, usage) -> None:
        self.requests += 1
        read = getattr(usage, "cache_read_input_tokens", 0) or 0
        write = getattr(usage, "cache_creation_input_tokens", 0) or 0
        self.read_tokens += read
        self.write_tokens += write
        if read > 0:
            self.hits += 1
        if write > 0:
            self.rewrites += 1
 
    @property
    def hit_rate(self) -> float:
        return (self.hits / self.requests * 100) if self.requests else 0.0
 
    @property
    def rewrite_rate(self) -> float:
        return (self.rewrites / self.requests * 100) if self.requests else 0.0
 
    def snapshot(self) -> str:
        return (
            f"req={self.requests} hit={self.hit_rate:.1f}% "
            f"rewrite={self.rewrite_rate:.1f}% "
            f"read_tok={self.read_tokens:,} write_tok={self.write_tokens:,}"
        )

It's a layer that just calls record(). I wanted to see one thing: is the rewrite rate stuck high? Measured on the summarization batch, hit rate was 34.8% and rewrite rate was 61.2%. More than half of all requests were rebuilding the cache. That put a number on why the bill wasn't dropping.

Why make rewrite rate the lead metric instead of hit rate? Because hit rate alone hides the cause. It tells you a call missed, but not whether the prefix shifted or whether the cache timed out. Seeing how often — and at what spacing — writes happen makes that split fall out quickly.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦A minimal harness that pulls cache_read and cache_creation from usage and logs hit rate and rewrite rate on every request

✦A measurement-first way to split a low hit rate into three distinct causes: non-deterministic prefix, TTL expiry, and sub-threshold blocks

✦How to walk one cache breakpoint backward at a time to locate the block that keeps re-triggering cache_creation

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

A low hit rate usually breaks into three causes

After running the probe for a few days, the misses clearly came in three flavors. Viewed together they're a fog; separated, the fix for each gets concrete.

Cause	Signal in the numbers	Root
1. Non-deterministic prefix	cache_creation fires every call, unrelated to spacing	A changing string leaks into the prefix
2. TTL expiry	cache_creation only on calls after a long gap	The 5-minute lifetime was crossed
3. Sub-threshold	Both read and creation are always 0	Prefix is below the minimum token count

1. Non-deterministic prefix — I had a timestamp up front

This was my culprit. I was injecting the current time at the very start of the system prompt, which made the prefix different on every request. Change one character and the cache treats it as a new prefix.

# What I was doing wrong — the head changes every call
system = [
    {
        "type": "text",
        "text": f"The current time is {datetime.now().isoformat()}. You are a summarizer...",
        "cache_control": {"type": "ephemeral"},
    }
]
 
# Fixed — move the variable part outside the cache boundary
system = [
    {
        "type": "text",
        "text": "You are a summarization assistant. ...(stable instructions and knowledge)...",
        "cache_control": {"type": "ephemeral"},   # identical every time up to here
    },
    {
        "type": "text",
        "text": f"The current time is {datetime.now().isoformat()}.",  # outside = not cached
    },
]

The easy-to-miss part isn't the obvious variables like timestamps. If you embed a dict or set via json.dumps, key ordering can wobble between runs and shift the prefix. When you embed, pass sort_keys=True to pin the order before placing it inside the boundary. That alone cleared out a good share of the "same content, still misses."

2. TTL expiry — the cache quietly disappears after five minutes

The standard lifetime of an ephemeral cache is about five minutes. Five minutes after the last access, the prefix is evicted, and the next request starts over with a write (a miss). For a sparse batch, this is where the cost hides.

The split is measurable. If you also record the seconds since the previous request on any call where cache_creation fired, expiry misses cluster on the "long gap" calls.

def classify_miss(gap_seconds: float, read: int, write: int) -> str:
    """Bucket a miss into a cause so you can pick a fix."""
    if read == 0 and write == 0:
        return "under_threshold"          # 3. sub-threshold
    if write > 0 and gap_seconds > 300:
        return "ttl_expired"              # 2. likely TTL expiry
    if write > 0:
        return "prefix_changed"           # 1. likely non-deterministic prefix
    return "hit"

Once expiry was the main driver, I had two options: tighten the call spacing to keep the cache warm, or switch to the one-hour extended cache (write is ~2x but it lives longer). I reshaped the batch to submit in 5-minute clusters instead of sprinkling calls out. Rewrite rate fell from 61.2% to 18.4%.

3. Sub-threshold — short prompts simply aren't cached

If the tokens up to the breakpoint don't meet the minimum, no cache is created at all. If read and creation both sit at 0 forever, suspect this first.

Model tier	Minimum tokens to form a cache
Flagship / standard tier (Opus / Sonnet family)	~1,024 tokens
Lightweight tier (Haiku family)	~2,048 tokens

I had cache_control on a short instruction block — exactly this. Pulling the knowledge base inside the boundary so the prefix clears the minimum fixed it. The work here isn't trimming; it's gathering stable content inside the boundary.

When cache_creation won't die down, walk the boundary back one block at a time

If writes still flicker after clearing all three causes, some wobble remains somewhere in the prefix. I moved the boundary like a binary search to corner the source.

The procedure: narrow the breakpoint to just the most stable leading block and measure the rewrite rate. Once it settles, pull the next block inside the boundary and re-measure. The block that spikes the rewrite rate is the source of the wobble.

def build_system(blocks: list[str], cache_upto: int) -> list[dict]:
    """Bundle the first `cache_upto` blocks into one stable prefix."""
    out = []
    for i, text in enumerate(blocks):
        part = {"type": "text", "text": text}
        if i == cache_upto - 1:              # one boundary on the last bundled block
            part["cache_control"] = {"type": "ephemeral"}
        out.append(part)
    return out
 
# Increase cache_upto 1 -> 2 -> 3 while watching rewrite_rate;
# stop just before it jumps. That edge is the end of your stable prefix.

The key thing here is that more breakpoints is not better. You get up to four, but in my measurements, bundling the stable head into one or two produced a lower rewrite rate than slicing it finely. Every extra boundary adds another "must match exactly up to here" condition, and if any one shifts, everything after it collapses together. Order least-changing content first: system (stable instructions) → tools → messages (old history) → newest message settled out best.

Keep the hit rate on a dashboard

Even after a fix, the cache drifts again, quietly. So I stopped treating the probe as a one-off and started streaming rewrite rate and hit rate into monitoring. With a series of numbers, degradation shows up before the invoice does.

def export_prometheus(probe) -> str:
    """Emit CacheProbe state in Prometheus format."""
    return (
        "# TYPE claude_cache_hit_rate gauge\n"
        f"claude_cache_hit_rate {probe.hit_rate:.2f}\n"
        "# TYPE claude_cache_rewrite_rate gauge\n"
        f"claude_cache_rewrite_rate {probe.rewrite_rate:.2f}\n"
        "# TYPE claude_cache_read_tokens counter\n"
        f"claude_cache_read_tokens {probe.read_tokens}\n"
    )

If I set only one alert, I put a threshold on the rewrite rate. It stirs before the hit rate falls. On the summarization batch I alert above 15%, and it caught a regression the day after I touched the prompt.

How far it dropped, and where I draw the line

The final numbers on the batch: hit rate 34.8% → 88.6%, rewrite rate 61.2% → 6.1%, effective input-token unit cost down to about 41%. That falls short of the "up to 90% off" headline. I think falling short is normal. A workload where every request is a clean cache hit rarely exists in practice.

Where I landed: caching isn't a setting you flip on or off — it's an operational metric you keep watching, called the hit rate. When the default model rolls over, when you add one line to a prompt, when you change call spacing, the prefix can quietly drift. So the real star of optimization wasn't clever placement; it was the boring habit of glancing at cache_read every time.

When the rewrite rate starts creeping up, that's the signal a prefix has begun to wobble somewhere. With this harness in place, you catch it without waiting for the invoice. For my part, the next time I swap the default model, I plan to watch these numbers for a few days before leaning on it in production.

Measure, name it, move one boundary. That loop is what carried caching from "I thought it was on" to "it's actually working." If it helps someone else's hand pause a little sooner over the same invoice, I'll be glad. Thank you for reading.

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

✦Copy-paste ready implementation code
✦New advanced guides published daily
✦$5/mo or $10 for lifetime access

View Membership →