Production Semantic Cache for Claude API — Similarity Thresholds, Pollution Defense, and What to Track

Step 1 — choose embeddings and tune the threshold

This is the single most important section of the article. Set the threshold too low and you serve wrong answers. Too high and the cache barely fires. The right value is empirical, not philosophical.

My rule of thumb: cosine 0.90–0.94 for mixed-language support, 0.88–0.92 for English technical Q&A, and 0.93–0.96 for highly structured domains like billing or compliance where small wording differences carry meaning. Use a multilingual model like voyage-3-large or cohere-embed-multilingual-v3 for the embedding step. Claude itself does not return embeddings, so this is a deliberate vendor pairing — not a workaround.

A few more considerations on embedding choice. Voyage tends to perform well on English plus East Asian languages with relatively short text. Cohere multilingual handles a wider language set out of the box. Open-source options like intfloat/multilingual-e5-large or BAAI/bge-m3 are viable if you self-host and want to control cost — expect a roughly 1–3 point precision drop versus the hosted commercial models in my benchmarks, but with predictable latency.

Do not pick the threshold by intuition. Label 100+ historical query pairs (300+ is better) and use a precision-recall curve to find the operating point.

# threshold_tuning.py — pick the threshold from labeled data
import json
import numpy as np
from sklearn.metrics import precision_recall_curve
 
# format: {"query_a": "...", "query_b": "...", "should_match": true/false}
with open("labeled_pairs.jsonl") as f:
    pairs = [json.loads(line) for line in f]
 
similarities = []
labels = []
for p in pairs:
    a = embed_fn(p["query_a"])
    b = embed_fn(p["query_b"])
    cos = float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
    similarities.append(cos)
    labels.append(1 if p["should_match"] else 0)
 
precision, recall, thresholds = precision_recall_curve(labels, similarities)
# Take the smallest threshold that holds precision >= 0.98 (≤ 2% false-positive rate)
acceptable = [(t, p, r) for t, p, r in zip(thresholds, precision[:-1], recall[:-1]) if p >= 0.98]
if acceptable:
    t, p, r = min(acceptable, key=lambda x: x[0])
    print(f"recommended threshold: {t:.4f} (precision={p:.3f}, recall={r:.3f})")
else:
    print("no threshold satisfies precision >= 0.98 — consider switching embedding models")

Why optimize Precision over Recall? A miss costs you API dollars, which is annoying but bounded. A wrong-answer hit costs you user trust, which is unbounded. The right asymmetry is to fix Precision at 0.98 or higher and let Recall float. If you cannot reach Precision 0.98 with your current embedding model, that is a strong signal to try a different model rather than to lower your standard.

How to label pairs. I keep a small CLI script that pulls 5 random query pairs per day from production logs (filtered for cacheable endpoints), shows them side by side, and asks "should the same answer satisfy both?" with y/n/skip. Twenty minutes per week is enough to maintain a labeled set of several hundred pairs and re-tune the threshold quarterly.

Step 2 — cache keys and multi-tenant isolation

For multi-tenant SaaS, include tenant_id in both the vector payload and the search filter. Once, in a previous role, I put it only in the payload and forgot to add the filter at read time. Tenant A started seeing Tenant B's responses; we caught it during an internal audit days later. It is a clean way to lose customer trust, and depending on your jurisdiction and data category, it can be a reportable incident.

I also separate by feature_id. The right threshold and TTL for an FAQ bot are not the right values for a code-review feature. Either use separate collections, or filter strictly on both keys.

For regulated domains (healthcare, finance, legal), you may decide that semantic caching itself is cacheable: false — the leakage cost outweighs the latency win. Always weigh "cost of not caching" against "cost of leakage" before defaulting to on. In my consulting work the balance often tips this way:

• Marketing-facing FAQ: cacheable, low risk, high win
• Internal employee Q&A: cacheable with tenant_id = company_id, medium win
• Customer-specific advice on financial products: NOT cacheable
• Personal medical questions: NOT cacheable
• Generic public-knowledge queries: cacheable across tenants if the answer truly does not depend on tenant context

The key heuristic: if the answer would change based on who is asking, do not share the cache entry across users.

Step 3 — guard the cache against poisoning

The bigger semantic-cache risk is poisoning: a malicious or low-quality response gets saved and then served to every near-match user that follows. I run three guards before any write.

# cache_guard.py — pre-write checks
import re
 
INJECTION_PATTERNS = [
    r"ignore (previous|all) instructions",
    r"system prompt",
    r"you are now",
    r"disregard the above",
]
 
def is_safe_to_cache(query: str, response, anthropic: Anthropic) -> bool:
    # Guard 1: drop obvious prompt-injection attempts
    if any(re.search(p, query, re.IGNORECASE) for p in INJECTION_PATTERNS):
        return False
 
    # Guard 2: never cache truncated or safety-stopped responses
    if response.stop_reason not in ("end_turn",):
        return False
 
    # Guard 3: cheap LLM-as-a-judge for content safety (Haiku, ~0.0003 USD)
    judge = anthropic.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=10,
        messages=[{"role": "user", "content":
            f"Reply YES only if the following is safe (no hate, no illegal advice, no PII). Otherwise NO.\n\n{response.content[0].text[:2000]}"}],
    )
    return judge.content[0].text.strip().upper().startswith("YES")

Guard 2 is the one most teams skip and regret. If you cache a response that hit max_tokens mid-thought, every user with a near-match keeps getting the same half-finished answer forever. Filter on stop_reason == "end_turn" only. Other stop reasons — max_tokens, tool_use, stop_sequence, refusal-style stops — usually represent incomplete or context-specific responses that should not be cached.

Guard 3 is cheap because it only runs on writes — typically 5–20% of traffic — and Haiku 4.5 is fast and inexpensive enough that the overhead is invisible in your P95. If you want to make this even cheaper, you can run the judge asynchronously after writing to Tier 1 (so the user sees the response immediately), and only persist the entry to Tier 2 once the judge approves.

A fourth guard worth considering: PII redaction. If your domain produces responses that may contain user-supplied PII (names, addresses, internal IDs), run a redaction pass before storing in the cache. The pattern is to replace likely PII with placeholders before the embedding step, so two queries that differ only in a customer name still collapse to the same cache entry.

Step 4 — fallbacks and graceful degradation

When the vector DB (Qdrant, Pinecone, pgvector, etc.) goes down, your service should not. Semantic cache is an optimization layer, not a hard dependency.

# graceful_fallback.py — fall back to Claude API if Qdrant is unhealthy
def safe_semantic_lookup(qdrant, query_vector, **kwargs):
    try:
        return qdrant.search(query_vector=query_vector, **kwargs)
    except (qdrant_client.http.exceptions.UnexpectedResponse, ConnectionError) as e:
        log_metric("semantic_cache.lookup_error", 1, tags={"err": type(e).__name__})
        return None
 
hits = safe_semantic_lookup(qdrant, embedding, collection_name="claude_cache",
                            query_filter=tenant_filter, limit=1, score_threshold=0.92)
if hits is None:
    # Qdrant degraded — go straight to Claude API
    response = anthropic.messages.create(...)

Pair the try/except with a hard timeout (asyncio.wait_for(timeout=0.2) is what I have used in async code, or a circuit breaker in sync code) so that a slow vector DB cannot stretch your end-to-end latency. If the lookup does not come back in 200 ms, give up and call Claude. The cost of a slow cache lookup followed by an API call is far worse than skipping the lookup entirely.

A related pattern: circuit breaking on consecutive failures. If the vector DB has been failing for the past N requests, stop trying for the next M seconds. Libraries like pybreaker or your favorite resilience framework handle this cleanly. The win is that during a long Qdrant outage you do not pay 200 ms per request just to discover the cache is still down — you skip it entirely until the breaker resets.

For exact-match cache (Redis), the same principle applies. A failing Redis should not break the request path; treat its absence as "cache miss" and continue to the next tier. Most Redis clients support short connection timeouts and automatic retries; configure them aggressively.

Cache invalidation — the famously hard problem

You will eventually need to invalidate cache entries. Common triggers include:

• Source data changed (a price updated, a policy changed)
• A bad response was served and reported by a user
• The underlying Claude model was upgraded (cached answers may be stale)
• A tenant was deleted or churned
• The system prompt for a feature changed

For source-data triggers, the cleanest approach is event-driven invalidation. Whenever the upstream data changes, publish an event with the affected tenant_id and feature_id, then run a delete query against the vector DB and a DEL against Redis. Pinecone, Qdrant, and pgvector all support filtered deletes — use them.

For model upgrades, I recommend versioning the cache namespace itself. Add a cache_version field to your Redis key prefix and your vector DB payload. When you swap models, increment the version, and old entries become unreachable without a destructive delete. They expire naturally via TTL.

For user-reported bad answers, add a "report this answer" button on cacheable surfaces. Reports go to a queue, a human reviews them daily, and confirmed-bad entries are deleted from both tiers. This is also where you find your false-positive rate empirically — if reports cluster around a particular feature_id, you have evidence to tighten its threshold.

Observability — the metrics that prove it works

Five metrics tell the whole story:

• Hit rate: (exact_hits + semantic_hits) / total_requests — target 30–60%
• False-positive rate: estimated from user feedback or A/B comparisons — keep at ≤ 2%
• Response latency P50/P95/P99: exact ≤ 5 ms, semantic ≤ 80 ms is reasonable
• Cost reduction: (cache_hits × avg_token_cost) / total_spend
• Staleness incidents: complaints from old answers (a TTL-design signal)

Track each by tenant_id and feature_id. Aggregate hit rates can hide a tenant whose cache is broken; per-tenant cuts reveal it. The simplest path is OpenTelemetry: tag each span with the layer attribute and the rest is dashboards in Datadog, Honeycomb, or Grafana. The full pattern is covered in Claude API + OpenTelemetry — Establishing Observability for AI Applications.

A useful weekly ritual: sample 20 random semantic hits and review them with a human. Are the matched query pairs actually equivalent? This is your ongoing false-positive sanity check. If the team is small, rotate the duty so no single person bears the load. The labeled pairs collected this way also feed back into the threshold-tuning dataset from Step 1, closing the loop.

For cost reduction specifically, separate "spend avoided" from "spend reduced." Spend avoided is the dollar value of API calls you did not make because the cache hit. Spend reduced is the change in your actual Anthropic invoice. They diverge when traffic grows — cache hit rate of 50% while traffic doubles still leaves your bill flat or slightly higher. Telling that story clearly to a CFO is part of why this metric matters.

Pitfalls I've stepped on, four times each

Pitfall 1: picking the threshold by gut feel. I once shipped 0.85 because "it sounds reasonable." False positives spiked. Use the labeled PR curve from Step 1 — minimum 100 pairs, ideally 300. Treat threshold tuning as an ongoing process, not a one-time setup.

Pitfall 2: writing without checking stop_reason. Truncated responses live forever in the cache, infuriating users. The stop_reason == "end_turn" check is non-negotiable. If you forget this and ship, the symptom is users complaining that "Claude keeps cutting off at the same place every time" — which sounds like a Claude bug but is actually a cache bug.

Pitfall 3: tenant_id in payload only, not filter. Cross-tenant leakage is silent until an audit. Put the tenant condition in both places. I now write a one-line unit test that issues a query as Tenant B and asserts that no cached entry from Tenant A is returnable, and run it on every CI build.

Pitfall 4: a single TTL across all data. I once used 24 hours globally — pricing-page answers stayed wrong for a day. Anything tied to fast-moving data (price, inventory, contract terms, real-time status) needs ≤ 1 hour TTL or should be excluded from caching entirely. Static educational content can safely live for days. Treat TTL as a per-feature decision and document the reasoning.

A fifth, subtler pitfall worth mentioning: evaluating the cache only on hit-rate. A 60% hit rate sounds great until you realize that 5% of those hits are wrong answers that nobody complained about loudly enough to escalate. Always pair hit rate with a false-positive estimate from human review.

Production checklist and the next step

Run through this list before flipping the flag in prod:

• The endpoint has cacheable: true set explicitly
• Threshold was chosen from a PR curve with Precision ≥ 0.98
• tenant_id lives in both payload and filter
• Only stop_reason == "end_turn" responses are written
• LLM-as-a-judge runs on writes
• Vector DB outage path is tested (simulate a 5-minute outage in staging)
• Hit rate, false-positive rate, P95 latency on a dashboard, segmented by tenant and feature
• TTLs differ by feature; volatile data has TTL ≤ 1 hour or is excluded
• Cache version is in the key prefix so model upgrades can be invalidated cleanly
• A human reviews 20 random semantic hits per week to estimate false positives

Start with one cacheable endpoint, A/B test it with your real traffic, and only then expand. In my own rollout, the FAQ bot alone hit 47% cache hit rate and dropped monthly API spend 38% — a clean signal to widen scope without rushing. The temptation to enable caching across all surfaces at once is strong, especially when the early metrics look good. Resist it for at least two weeks per new endpoint.

Semantic caching is one of the few optimizations where "fear it correctly, ship it correctly" produces real, measurable wins. Keep this checklist nearby and the pattern will pay for itself within weeks — both in the bill and in the latency budget your users will feel without ever knowing why.

When semantic caching is the wrong tool

Not every LLM workload should sit behind a semantic cache. I have seen teams force-fit it into use cases where it actively hurt user experience. A few patterns to watch for:

• Long-form generation with creative requirements. If users expect a fresh take every time (poems, marketing copy, code variations), even a perfect semantic match feels broken. The reuse itself violates the promise of the product.
• Highly personalized advice. Anything that reasons about a specific user's account, history, or preferences is not safe to share across users, even within the same tenant. The safest implementation here is to scope the cache by user_id, but at that point the hit rate often collapses to single digits and the engineering cost is hard to justify.
• Tool-calling with side effects. If the response triggers writes, payments, or external API calls, caching the language layer while skipping the side effect creates ghost responses where the user thinks something happened but nothing did. Cache only the explanation, not the action.
• Stateful conversations. Multi-turn conversations carry context that mutates the meaning of nearly identical surface queries. "Tell me more" embeds nearly identically across all conversations, but the right answer depends on the prior turn. Cache at the turn level, not the message level — and even then, only after careful design.

A useful internal heuristic at my last project: if I cannot explain to a non-technical stakeholder why it is safe for User A's answer to be served to User B, the answer is not cacheable. The explanation has to be more concrete than "the embeddings are close." It has to be "the question is about a public fact that does not change based on who is asking, and the response is general-purpose."

Cost modeling — what to expect, and what to promise

I have been asked enough times "what hit rate should we plan for?" that I want to set realistic expectations. The numbers below are from production deployments I have personally tracked, not benchmarks.

• Public-knowledge FAQ (e.g., "what is your refund policy"): 60–75% hit rate, 40–60% cost reduction in steady state.
• Internal helpdesk (e.g., "how do I request VPN access"): 50–65% hit rate, 35–50% cost reduction.
• Product-discovery search (e.g., "show me running shoes for flat feet"): 30–45% hit rate, 20–30% cost reduction. Lower because of long-tail queries.
• Customer-specific support (e.g., "why was my order delayed"): 5–15% hit rate. Often not worth the engineering cost on its own.
• Code-related Q&A (e.g., "how do I fix this Python error"): 20–35% hit rate. Highly dependent on whether you canonicalize variable names and stack traces before embedding.

Promise less than you expect to deliver. A common pitfall is to promise the executive team "50% cost reduction" based on a one-day measurement, then watch the number erode over a quarter as users find new ways to phrase things. Promise 25–30% and over-deliver, and budget conversations stay productive.

A word on testing the cache itself

This is the section nobody writes, so let me write it. Your semantic cache deserves the same rigor as any other production component, and that means deterministic tests in CI.

I recommend three categories of tests. First, unit tests that mock the embedding function and the vector DB, verifying that the threshold logic, tenant filter, and stop_reason guard fire correctly. These are fast and run on every commit.

Second, integration tests that hit a real vector DB (often a Docker-Compose Qdrant or pgvector instance) with a fixed corpus of seed embeddings. These verify that filtered queries actually return only matching tenant data — the kind of cross-tenant leakage we discussed earlier is best caught here.

Third, evaluation tests that periodically (daily or weekly) re-run the labeled query-pair set from Step 1 against the current threshold and embedding model, alerting if Precision drops below 0.98. These are slow and expensive (one embedding call per pair) but catch silent regressions when you upgrade the embedding model or refresh the corpus.

The cost of building these test layers is substantial — easily a sprint of work — but the cost of not having them, when the eventual cache regression slips into production, is much higher. I treat this as part of the launch checklist, not a "nice to have" follow-up.