I have been running prompt caching in production for about six months now. I was skeptical at first, but on workloads where every request carries a long system prompt, my bill genuinely landed at less than half of what it used to be. The catch is that the wrong design produces almost no savings at all — it is one of those features with a sharp gap between "configured properly" and "configured anyway."
This article is a write-up of what I learned introducing prompt caching across several of my own apps, watching the monthly bill move, and rewriting the prompt structure until it actually paid off. I have included the behaviors I picked up from real traffic that the official docs do not really spell out.
Why I started taking prompt caching seriously
The honest reason: my API bill came in at roughly three times what I expected. Chat-style services tend to grow very long system prompts — context, persona, output rules — and mine had drifted to about 6,000 to 10,000 tokens. I was sending the whole thing on every turn.
Twenty turns of conversation means I was paying for north of 120,000 tokens of system prompt alone. It was hidden in the noise of context handling on each request, but once I broke down the bill it was painful to look at.
Prompt caching attacks exactly this "send the same preamble every time" cost. A cache hit charges input tokens at one tenth of the standard rate (cache writes cost 1.25x, but only on first write).
That said, whether you actually get savings depends heavily on where the static portions of your prompt live and how your traffic accesses the cache.
What kind of prompt structure actually caches
Anthropic's cache_control parameter lets you mark a portion of the prompt as "cache up to here." In the Python SDK it looks like this.
from anthropic import Anthropic
client = Anthropic()
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
system=[
{
"type": "text",
"text": LONG_STATIC_SYSTEM_PROMPT, # several thousand tokens
"cache_control": {"type": "ephemeral"}
}
],
messages=[
{"role": "user", "content": user_input}
]
)Everything up to and including the marked element gets cached. The crucial point: a cache only hits when the prompt matches exactly from the very beginning up to the marked position. If anything user-specific lives before the cache marker, you will miss every time.
This was my first stumble. I had been embedding usernames and session metadata at the top of my system prompt. The cache existed; I just never hit it.
The right ordering is: static content first, dynamic content after.
system=[
{
"type": "text",
"text": STATIC_PROMPT, # persona, rules, examples — never changes
"cache_control": {"type": "ephemeral"}
},
{
"type": "text",
"text": f"User context: {user_id}, time: {now}" # dynamic
}
]Start by measuring hit rate
The very first thing to do after wiring caching up is measure your hit rate. The response usage object contains the fields you need.
print(response.usage)
# Usage(
# input_tokens=120,
# cache_creation_input_tokens=0,
# cache_read_input_tokens=8500,
# output_tokens=300
# )cache_read_input_tokens is what hit the cache. cache_creation_input_tokens is what got freshly written. input_tokens is regular billed input.
I log at minimum these metrics on every request.
def log_cache_metrics(response, request_id):
usage = response.usage
total_input = (
usage.input_tokens
+ usage.cache_creation_input_tokens
+ usage.cache_read_input_tokens
)
hit_rate = usage.cache_read_input_tokens / total_input if total_input else 0
logger.info({
"request_id": request_id,
"cache_hit_rate": hit_rate,
"cache_read": usage.cache_read_input_tokens,
"cache_write": usage.cache_creation_input_tokens,
"input_uncached": usage.input_tokens,
"output": usage.output_tokens,
})If the rolling hit rate stays above 80 percent your design is in roughly the right shape. If it drops under 50 percent, almost always something dynamic has crept in front of the cache marker — go re-read the prompt assembly code.
Picking a TTL
The default TTL for an ephemeral cache entry is five minutes since last access. For continuous chat that is fine, but consider an internal tool that only sees traffic during weekday lunch breaks — every break, the cache evaporates and gets rewritten.
For longer TTL (one hour) you pass ttl inside cache_control.
"cache_control": {"type": "ephemeral", "ttl": "1h"}The catch: writes against a one-hour TTL cost twice as much as the standard write. If your hit rate is high it is a clear win; if your hit rate is low, you are now paying more on writes too.
My rule of thumb: if a typical user touches the system at least ten times a day, the one-hour TTL pays for itself. For overnight batch jobs that have no follow-up traffic, the default five minutes is right. Read your access logs before flipping the switch.
Mind the token floor
There is a minimum cacheable size: 1,024 tokens for the Sonnet family, 2,048 for Haiku. Below that, cache_control does nothing.
A subtle trap is dynamically assembled system prompts. If you grow or shrink rules based on user role, you can land at 2,000 tokens one day and 900 tokens the next, and the second day silently bills you full price.
In production I padded my system prompt template so that it always lands above 1,024 tokens. I did this honestly — by enriching the examples section — and pushed all per-user content past the cache boundary. Hit rate became stable.
The four-block ceiling
You can mark at most four blocks per request with cache_control. If you carve up a long RAG document carelessly, you will hit the ceiling without realizing it.
The pattern that has worked best for me: split along logical boundaries, four or fewer blocks total.
system=[
{"type": "text", "text": COMPANY_RULES, "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": OUTPUT_FORMAT_SPEC, "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": EXAMPLES, "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": USER_RAG_CONTEXT, "cache_control": {"type": "ephemeral"}},
]Each block forms an independent "cache layer." If only USER_RAG_CONTEXT changes, the three blocks before it still hit. This is enormously useful for bots that hold long-lived background knowledge.
A working RAG pattern
Here is the layout I actually use in a small support bot of mine — a basic RAG that loads a FAQ corpus and answers from it.
def build_messages(user_query: str, retrieved_docs: list[str]) -> dict:
return {
"system": [
{
"type": "text",
"text": SYSTEM_PROMPT_BASE, # persona + output rules
"cache_control": {"type": "ephemeral"}
},
{
"type": "text",
"text": FAQ_KNOWLEDGE_BASE, # ~50KB fixed FAQ
"cache_control": {"type": "ephemeral"}
},
{
"type": "text",
"text": "\n".join(retrieved_docs) # changes per query
}
],
"messages": [
{"role": "user", "content": user_query}
]
}With this layout the FAQ block holds a hit rate above 95 percent. The retrieved docs are uncached by definition, but that is unavoidable — they do change per query.
In aggregate, my monthly cost dropped to 38 percent of the pre-caching baseline. Closer to a third than a half.
Five things to check when caching is not hitting
This is the checklist I have written for myself more than once.
First: is anything dynamic appearing before your cache_control marker? This is where I tripped initially.
Second: are you above the minimum token threshold? Sonnet will not cache below 1,024 tokens.
Third: has the TTL expired? Five minutes of idle time and the entry is gone.
Fourth: is the model the same? claude-sonnet-4-6 and claude-sonnet-4-5 keep separate caches, so a model bump effectively resets you.
Fifth: is the prompt byte-identical? A single different whitespace breaks the match. If your prompt is templated, suspect the template function.
A real cost trajectory
Numbers from my support bot (about 12,000 requests/month) for context:
Before introduction: $187/month, no caching, ~8,000-token system prompt, 20-turn average.
One week in: $142/month. Cache markers placed wrong, hit rate around 30 percent.
One month in: $79/month. Dynamic content moved past the boundary, TTL bumped to one hour, hit rate at 92 percent.
Three months in: $71/month. RAG documents added as a cached block.
Roughly 62 percent off the baseline. The longer your prompt, the bigger the savings — services that lean on long system prompts have the most to gain.
Where to start
If reading this you suspect your service might benefit, pull a day of request logs and look at two things: the size of your system prompt, and how often it repeats. If you are above 1,024 tokens and shipping the same context repeatedly, the win is almost guaranteed.
The implementation is essentially "make system a list and add cache_control," which you can finish in an afternoon. The hard part is the design — where to draw the boundary, and how to measure. Keep this checklist nearby and walk the boundary back and forth while watching hit rate.
If you can land at 80 percent hit rate within the first few days, the cost reduction is not far behind.