Splitting Claude API prompt cache into 5m and 1h tiers — separate TTLs cut cost and stabilize ops

The two-tier layout

This is the hierarchy I run today:

Implementation:

import anthropic
 
client = anthropic.Anthropic()
 
def build_two_tier_request(
    system_prompt: str,
    tools: list[dict],
    fewshot_block: str,
    user_text: str,
):
    """Build a Messages API request with a 1h+5m two-tier cache.
 
    L1/L2 are 1h; L3 is 5m. If L3 lapses overnight, L1/L2 still
    survive the next morning.
    """
    system = [
        {
            "type": "text",
            "text": system_prompt,
            "cache_control": {"type": "ephemeral", "ttl": "1h"},  # L1
        }
    ]
 
    # L2: 1h breakpoint at the end of the tools array
    tools_with_cache = [dict(t) for t in tools]
    tools_with_cache[-1] = {
        **tools_with_cache[-1],
        "cache_control": {"type": "ephemeral", "ttl": "1h"},
    }
 
    messages = [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": fewshot_block,
                    "cache_control": {"type": "ephemeral"},  # L3 (5m)
                },
                {"type": "text", "text": user_text},  # L4: never cached
            ],
        }
    ]
    return system, tools_with_cache, messages
 
 
system, tools, messages = build_two_tier_request(
    system_prompt=SYSTEM_PROMPT,
    tools=TOOLS,
    fewshot_block=current_fewshot(),
    user_text=user_input,
)
 
resp = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=2048,
    system=system,
    tools=tools,
    messages=messages,
)

Three pieces matter here. L1 and L2 must live on separate breakpoints, so swapping tools does not invalidate the system prompt. L3 sits on the user content array, not on system; placing it on system chains invalidations as the variable content grows. L4 must never carry cache_control; user input is unique every time, so caching it only inflates write cost.

Metrics — let numbers decide if your TTLs are right

I do not trust intuition on whether caching "is working." I look at three numbers from the usage block of every response:

def cache_efficiency_metrics(usage: dict) -> dict:
    """Per-request cache efficiency, suitable for daily aggregation."""
    cache_read = usage.get("cache_read_input_tokens", 0)
    cache_create = usage.get("cache_creation_input_tokens", 0)
    plain_input = usage.get("input_tokens", 0)
 
    total_input = cache_read + cache_create + plain_input
    if total_input == 0:
        return {}
 
    return {
        # (1) hit ratio: share of input that came from cache
        "hit_ratio": cache_read / total_input,
        # (2) write ratio: writes per read
        "write_ratio": cache_create / max(cache_read, 1),
        # (3) plain ratio: input that bypassed the cache entirely
        "plain_ratio": plain_input / total_input,
    }

I read those three numbers like this:

1. If hit_ratio < 0.6, either the TTL or the breakpoint position is wrong. My chat-style workload sits at 0.72–0.85 once tuned.
2. If write_ratio > 0.3, you are writing too often. A 1h tier that does not lower write_ratio is a layer whose access pattern does not survive an hour. Put it back on 5m.
3. If plain_ratio > 0.5, you are not exploiting the cache at all. The most common cause I see is putting variable few-shot inside system instead of in the user message.

Aggregating those daily, the change to a two-tier layout moved my mean hit_ratio from 0.58 to 0.78 and dropped my monthly input-token bill by about 42%. Doubling the write cost on the 1h layer is a real number, but the read savings dwarf it.

Three cases where 1h TTL hurts

I have seen 1h fail in three specific shapes. All three came down to me misreading the access pattern.

The first is a service whose traffic goes to zero at night. If you regularly leave the cache idle for more than an hour, you pay the higher write rate without recovering it on the next call. I have one art-related companion app that quiets down between 03:00 and 05:00 JST; that path stays on 5m.

The second is a system prompt under heavy A/B testing. When a prompt registry bumps from v17 to v18, the cache key changes and the write fires. Keep 5m while you are iterating; switch to 1h after the prompt has been stable for a week.

The third is trying to cache the entire messages history. Putting 1h breakpoints inside long conversation history invalidates every layer the moment a user sends a new turn. I do not place explicit breakpoints inside conversation history at all; I let it fall off naturally.

A safe rollout order

I would not flip every layer to 1h on day one. The order I used was:

1. Add measurement first. Ship the three metrics above to whatever you use for dashboards. Watch for a week.
2. Put L3 (few-shot) on 5m first. Smallest blast radius. Easy to roll back.
3. Move L1 (system) to 1h. cache_creation_input_tokens will spike for the first few hours. If hit_ratio rises after 24 hours, you are good.
4. Move L2 (tools) to 1h. Big payoff if you have five or more tools defined.
5. Add a "force 5m" kill switch. Useful for A/B phases and bad days.

A 24–48 hour observation window between each step makes regressions easy to attribute.

Watch out — only four breakpoints per request

cache_control allows at most four breakpoints per request. The mistake I caught myself making was putting a breakpoint on every tool definition. The fifth tool I added silently disabled all my caching.

# Anti-pattern: a breakpoint per tool
for tool in tools:
    tool["cache_control"] = {"type": "ephemeral", "ttl": "1h"}

The correct pattern is to put a single breakpoint on the last tool definition; the cache key covers everything up to that point. Breakpoints partition the prefix; they do not multiply caches.

Ship the rollback before the change

Before I push a TTL change to production, I add an environment-variable kill switch that flattens everything back to 5m:

import os
 
def cache_ttl_for(layer: str) -> dict:
    """Env-var rollback to 5m for emergencies."""
    if os.getenv("PROMPT_CACHE_FORCE_5M") == "1":
        return {"type": "ephemeral"}  # 5m
    if layer in ("system", "tools"):
        return {"type": "ephemeral", "ttl": "1h"}
    return {"type": "ephemeral"}  # 5m

This means a bad evening for hit_ratio is a one-variable rollback, not a deploy. From years of running AdMob since 2014, the design choices that pay back the most are not new features but the ones that save you 30 seconds of judgment at 2 AM. This is one of those.

What I learned after two months on the two-tier layout

Two things shifted in how I think about prompt design after running this for a couple of months.

First, the cache layout is the seam in the system design. Drawing the line between L1 and L3 forced me to be explicit about which content is shared across all users and which is per-request. That clarity helped the code, not just the bill.

Second, TTL is a third operational lever, alongside "edit the prompt" and "swap the model." Being able to dial TTL based on access pattern changed how I respond to cost incidents — sometimes the right answer is not a prompt rewrite but a TTL change.

Twelve years of indie development has trained me to care about server cost in a slightly anxious way. Prompt caching reframes that anxiety into a design problem with crisp controls, which I appreciate.

If you are about to try this

Start with the measurement step. Resist the urge to flip 1h on first; ship the three metrics and watch them for a week. Once you know your current hit_ratio and the time bands where it dips, the order of breakpoints becomes obvious.

In my own case the first week of measurement is what surfaced the dead overnight window I would have missed. A small investment in observability up front saves a lot of "is it working?" debate later. Thank you for reading — I hope this layout is useful in your own workload.