When Your Claude API Response Cache Returns Stale Answers and Near-Miss Wrong Ones — Field Notes on Freshness and False-Hit Suppression

Freshness: volatility-class TTLs and tag-based invalidation

A single global TTL is always wrong somewhere. Give 24 hours to something that should live 30 minutes and stale hits climb; give 5 minutes to something that barely changes and you throw away hit rate. I class answers by volatility and set TTL per class.

Set TTL by "how stale is tolerable at worst," and don't rely on TTL alone. Pair it with tag-based invalidation that actively deletes cached answers the moment a source document changes. The trick is to keep a reverse index in a Redis Set — which cached keys depend on which document.

import Redis from "ioredis";
const redis = new Redis(process.env.REDIS_URL as string);
 
// On store: register the document IDs this answer depends on as tags.
async function setWithTags(
  key: string,
  value: string,
  ttlSeconds: number,
  docTags: string[]
): Promise<void> {
  const pipe = redis.pipeline();
  pipe.set(key, value, "EX", ttlSeconds);
  for (const tag of docTags) {
    pipe.sadd(`tag:${tag}`, key);
    pipe.expire(`tag:${tag}`, ttlSeconds + 60);
  }
  await pipe.exec();
}
 
// When a document changes, delete every cache entry that depends on it.
async function invalidateByDoc(docId: string): Promise<number> {
  const keys = await redis.smembers(`tag:${docId}`);
  if (keys.length === 0) return 0;
  const pipe = redis.pipeline();
  pipe.del(...keys);
  pipe.del(`tag:${docId}`);
  await pipe.exec();
  return keys.length;
}

With this in place, a CMS or product-master update hook just calls invalidateByDoc(docId) and retires exactly the answers tied to that document. You stop waiting for natural TTL expiry, which shrinks the stale-hit window from "until the next TTL" to "the moment of the update."

A verification layer to suppress semantic false hits

Semantic caching is powerful, but deciding acceptance on a similarity threshold alone always lets false hits through. In embedding space, "Can I get a refund?" and "I was told I can't get a refund and I'm not satisfied" sit close together. The wording is near; the desired conclusion is opposite.

Raising the threshold reduces false hits but also drops hit rate. My conclusion is not to crank the threshold but to insert a cheap check after a neighbor is found. Just comparing negation and key entities catches a large share of conclusion-flipping false hits.

// Decide acceptance after a semantic neighbor is found.
// Even at high similarity, reject if negation or entities diverge.
interface VerifyInput {
  query: string;
  candidateQuery: string;  // the original question stored in the cache
  similarity: number;      // 0..1
}
 
const NEGATIONS = ["not", "can't", "cannot", "no ", "denied", "refused", "unsupported"];
 
function extractEntities(text: string): Set<string> {
  // Swap in real NER in production; here we cheaply grab alphanumeric tokens.
  const tokens = text.toLowerCase().match(/[a-z0-9_]+/g) ?? [];
  return new Set(tokens);
}
 
export function acceptSemanticHit(input: VerifyInput): boolean {
  const { query, candidateQuery, similarity } = input;
  if (similarity < 0.86) return false; // coarse first filter
 
  // Negation guard: if only one side is negated, the conclusion likely flipped.
  const qNeg = NEGATIONS.some((n) => query.toLowerCase().includes(n));
  const cNeg = NEGATIONS.some((n) => candidateQuery.toLowerCase().includes(n));
  if (qNeg !== cNeg) return false;
 
  // Entity guard: weak overlap of key entities means a different question.
  const qe = extractEntities(query);
  const ce = extractEntities(candidateQuery);
  const overlap = [...qe].filter((e) => ce.has(e)).length;
  const denom = Math.max(1, Math.min(qe.size, ce.size));
  if (overlap / denom < 0.5) return false;
 
  return true;
}

This check is orders of magnitude cheaper than the embedding lookup and adds no API calls. When acceptSemanticHit returns false, ignore the cache and call Claude normally. The point is to stop trying to settle everything with one threshold: keep the threshold as a coarse filter, and kill conclusion-flipping factors (negation, entity drift) with dedicated guards.

Measure the poisoned-hit rate — never ship a cache you can't observe

This is the part I learned the hard way as an indie developer. The technical blogs I run under Dolice Labs are served through a Cloudflare Workers edge cache, and once an empty article page that briefly failed to render — and even an error HTML page — got pinned at the edge for hours and was served all the way to search crawlers. The cache itself was working perfectly, hit rate was high, and what it served was broken. The lesson was blunt: a cache with no gauge for poisoned hits is fast and untrustworthy.

Average hit rate is not a health check. What you actually want is (1) the true hit rate, (2) the poisoned-hit rate (stale plus false hits), and (3) the share the verification layer rejected. Estimate the poisoned rate with a shadow comparison — on a small sample, fetch fresh alongside the cached answer and diff them.

// On a few percent of traffic, fetch fresh in parallel with the cache hit
// and record the diff. Cost scales with sampleRate.
async function shadowCompare(
  key: string,
  cached: string,
  freshFetch: () => Promise<string>,
  sampleRate = 0.02
): Promise<void> {
  if (Math.random() > sampleRate) return;
  const fresh = await freshFetch();
  const drifted = normalize(cached) !== normalize(fresh);
  await redis.hincrby("cache:audit", drifted ? "poisoned" : "clean", 1);
  if (drifted) {
    await redis.del(key); // retire the drifted key without waiting for TTL
    await redis.lpush("cache:drift_samples", JSON.stringify({ key, cached, fresh }));
    await redis.ltrim("cache:drift_samples", 0, 199);
  }
}
 
function normalize(s: string): string {
  return s.replace(/\s+/g, " ").trim();
}

Adding observation to a cache you already run takes three steps at minimum.

1. Add clean / poisoned counters to cache:audit and record true hit rate separately from poisoned-hit rate.
2. Run shadowCompare at 2% sampling and retire any key whose diff drifts.
3. Make a one-line rule: if the poisoned-hit rate crosses 1%, revisit the threshold or the TTL.

Running this shadow comparison at just 2% sampling is enough to catch a rising poisoned-hit rate early. My operating rule is simple: if the poisoned-hit rate crosses 1%, revisit the threshold or the TTL. With a number in hand, the choice between an aggressive and a defensive cache rests on evidence rather than instinct.

Knowing when not to cache

The highest-leverage optimization is the one most often skipped: choosing not to cache. A cache is not universal; questions clearly split into ones it suits and ones it doesn't.

It suits questions that repeat often, have stable answers, and aren't catastrophic when wrong — product FAQs, glossary entries, the skeleton of a standard report. It doesn't suit strongly personalized answers, answers that depend on time or remaining balance, or domains where a single wrong answer badly erodes trust (medical or legal). For those, a fast but stale or off-target answer inverts its own value the instant it's served.

Lately, when I add a cache to a new feature, I think first about "what breaks when it misses," not "what a hit saves." If a miss does serious damage, I don't cache it even when duplication is high. If the damage is small and duplication is high, I push — longer TTL, lighter verification. Drawing that line up front keeps you out of the trap of chasing hit rate and breeding false hits.

As a next step, add a cache:audit counter and a shadow comparison to whatever cache you already run. Just giving a hit-rate-only cache a poisoned-hit gauge changes how you weigh offense against defense. I hope it helps anyone working through the same problem.