Coalescing Concurrent Claude API Calls: Single-Flight Against Duplicate Inference and Cache Stampede

Folding Across a Distributed Runtime: Multi-Worker and Edge

In an environment like Cloudflare Workers, where callers are spread across many instances, an in-process Map is not shared. Here we "elect one representative" with a short-lived lock. Only the representative computes upstream; the others wait briefly for the result under a short TTL.

// distributed-singleflight.ts (backed by Cloudflare KV)
interface Env { CACHE: KVNamespace }
 
const RESULT_TTL = 60;  // how long a shared result lives (seconds)
const LOCK_TTL   = 30;  // lock that marks "computation in progress" (seconds)
const NEG_TTL    = 10;  // short negative cache that remembers a failure (seconds)
 
export async function coalesced(
  env: Env, key: string, compute: () => Promise<string>,
): Promise<string> {
  const resKey = `res:${key}`;
  const lockKey = `lock:${key}`;
 
  // 1) If someone already finished computing, return immediately.
  const hit = await env.CACHE.get(resKey);
  if (hit !== null) return hit;
 
  // 2) Try to take the lock. If you can't, a representative is computing; wait and re-read.
  const gotLock = await tryAcquire(env, lockKey, LOCK_TTL);
  if (!gotLock) return await waitForResult(env, resKey);
 
  // 3) As the representative, compute -> share the result; remember failures briefly.
  try {
    const value = await compute();
    await env.CACHE.put(resKey, value, { expirationTtl: withJitter(RESULT_TTL) });
    return value;
  } catch (e) {
    await env.CACHE.put(`neg:${key}`, "1", { expirationTtl: NEG_TTL });
    throw e;
  } finally {
    await env.CACHE.delete(lockKey);
  }
}
 
async function tryAcquire(env: Env, lockKey: string, ttl: number): Promise<boolean> {
  if ((await env.CACHE.get(lockKey)) !== null) return false; // approximate mutual exclusion
  await env.CACHE.put(lockKey, "1", { expirationTtl: ttl });
  return true;
}
 
async function waitForResult(env: Env, resKey: string): Promise<string> {
  for (let i = 0; i < 15; i++) {
    await sleep(200 + Math.random() * 200); // jitter the polling too
    const v = await env.CACHE.get(resKey);
    if (v !== null) return v;
  }
  throw new Error("coalesced wait timeout"); // insurance if the representative dies
}
 
function withJitter(ttl: number) { return ttl + Math.floor(Math.random() * ttl * 0.2); }
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

The part that really earns its keep here is withJitter. If every entry expires on the same 60-second boundary, a fresh stampede fires the instant they expire. Adding a plus/minus 20% wobble to the TTL scatters the expiries and stops the herd from reassembling. KV is eventually consistent, so this is not a strict lock — but for uses like summarization, where "a little duplication is fine but a lot is painful," it is more than enough. If you truly need exactness, a Durable Object can give you a single serialization point.

Scattering the Retry Herd: Backoff with Jitter

The other herd is born from retries after failure. If several callers that hit a 429 at the same time all decide to "retry in one second," they all rush again one second later. The standard fix is to always mix jitter into exponential backoff.

async function withRetry<T>(fn: () => Promise<T>, max = 5): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn();
    } catch (e: any) {
      const status = e?.status ?? e?.response?.status;
      const retryable = status === 429 || status === 503 || status === 529;
      if (!retryable || attempt >= max) throw e;
 
      // Respect Retry-After when present; otherwise use full jitter (uniform 0..cap).
      const header = Number(e?.headers?.["retry-after"]);
      const base = Number.isFinite(header) ? header * 1000 : Math.min(1000 * 2 ** attempt, 20_000);
      const delay = Math.random() * base; // full jitter
      await new Promise((r) => setTimeout(r, delay));
    }
  }
}

The Math.random() * base full-jitter form spreads the wait uniformly across 0 to the cap. It is known to converge faster than waiting for exactly the cap. For the fundamentals of handling 429s, "Handling Claude API Rate Limit (429) Errors" is a good companion.

Pitfalls the Official Docs Don't Cover

Here are the spots that quietly cause incidents after the code "works." These are ones I actually stepped on.

"Riding along on streaming" is especially easy to miss. Single-flight shines for uses that return one settled response — summarization, classification, extraction. If you want to re-broadcast incremental tokens to multiple waiters, either hand out the finished string after it settles, or decide not to coalesce at all.

Recommendations by Situation

Rollout Checklist

1. Build the key from model, system, normalized user text, temperature, and max_tokens — all of them.
2. In-process, always delete from the Map in finally (success or failure).
3. In a distributed runtime, elect a representative with a short-TTL lock; have waiters poll with jitter.
4. Separate the result TTL from the negative-cache TTL (keep the negative one short, around 10 seconds).
5. Add jitter to every TTL to break expiry synchronization.
6. Use full jitter for retries, and respect Retry-After when you can.
7. For streaming, decide whether coalescing is even appropriate before you fold.
8. Keep measuring duplicate rate, upstream call count, and waiter timeout rate.

Here are the numbers after rollout. The top-of-the-hour duplicate rate went from 76% to 3%, and upstream calls converged from an average of 4.2 to 1.1 per minute. Input tokens fell by roughly 60% per month. This is not a flashy optimization. It is the quiet accumulation of one obvious principle: never buy the same answer twice.

The thinner the headroom, the more this kind of trimming pays off. If this small way of folding requests brings a little more calm to your nightly runs, I would be glad. Thank you for reading.