Top of the hour. The summarization tasks I run overnight across four sites wake up at almost the same instant. This morning I was following that one minute through the request logs, and my hand stopped. The exact same "summarize today's news" prompt had gone out to Claude four separate times, spaced only milliseconds apart.
All four answers were effectively identical. But I was billed for four. This is also the very day Claude Code's weekly-limit boost (+50%) ends and Sonnet 5 has just become the default model, so my rate-limit headroom is thinner than before. "Buying the same answer over and over" is a small waste — until the headroom shrinks, and then it starts to sting.
What I needed was not a new cache. It was a way to fold the requests that fire at the same moment into one. This article records that folding — the design and implementation of single-flight (request coalescing), from in-process to distributed, with measurements attached.
When Does Duplicate Inference Happen? The Anatomy of a Stampede
Even with a result cache in place, the duplication does not disappear. In fact, it shows up precisely because you have a cache.
The key is concurrency. A cache hit only happens after someone has called upstream once and finished writing the result. But when several callers plunge into a cache miss at the same instant — as they do at a nightly peak — every one of them concludes "it's not in the cache, so I'll compute it myself," and they all rush upstream together. That is a cache stampede (thundering herd).
| Mechanism | Solves | Does not solve |
|---|---|---|
| Exact-match cache | Reusing a request identical to a past one | Duplicate first-time misses running concurrently |
| Semantic cache | Reusing semantically similar requests | Concurrent misses; the rush right after expiry |
| Single-flight | Duplicate identical requests running concurrently | Reuse across time (you still need a cache) |
So single-flight and caching do not compete — they play different roles. A cache handles "reuse across time"; single-flight handles "deduplication within the same instant." Only when you layer both does the waste disappear both at the moment of a miss and after a hit. For the result-reuse side of the design, see "Putting a Semantic Cache for the Claude API into Production." This article concentrates on the folding side.
The duplication I measured in my own setup, running several sites as an indie developer, averaged 4.2 calls per minute during the top-of-the-hour peak — the result of related tasks across four sites each firing the same summarization prompt independently.
In-Process Single-Flight: The Minimal Implementation
If everything lives in one Node/Worker process, the implementation is surprisingly small. The heart of it is "share the in-progress Promise by request key." Exactly one caller hits upstream; the rest await the same Promise.
// singleflight.ts
type Entry<T> = { promise: Promise<T>; controller: AbortController };
export class SingleFlight<T> {
private inflight = new Map<string, Entry<T>>();
// fn receives an AbortSignal; if every waiter cancels, upstream stops too.
async run(key: string, fn: (signal: AbortSignal) => Promise<T>): Promise<T> {
const existing = this.inflight.get(key);
if (existing) return existing.promise; // ride along with the identical in-flight request
const controller = new AbortController();
const promise = fn(controller.signal).finally(() => {
// Always delete, success or failure. Leaving it leaks memory and pins a stale result.
this.inflight.delete(key);
});
this.inflight.set(key, { promise, controller });
return promise;
}
get size() { return this.inflight.size; }
}On the calling side, building a stable key from the prompt is what matters. Unless you fold in the model name, temperature, max_tokens, and system prompt, you will accidentally coalesce requests that were meant to differ.
import { createHash } from "node:crypto";
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic(); // ANTHROPIC_API_KEY comes from the environment
const sf = new SingleFlight<string>();
function requestKey(model: string, system: string, user: string, opts: Record<string, unknown>) {
const norm = user.trim().replace(/\s+/g, " "); // minimal normalization of whitespace
const payload = JSON.stringify({ model, system, user: norm, opts });
return createHash("sha256").update(payload).digest("hex");
}
async function summarize(newsText: string): Promise<string> {
const model = "claude-sonnet-5";
const system = "You are a concise summarizer.";
const key = requestKey(model, system, newsText, { max_tokens: 512 });
return sf.run(key, async (signal) => {
const res = await client.messages.create(
{ model, max_tokens: 512, system, messages: [{ role: "user", content: newsText }] },
{ signal }, // cancel upstream when every rider drops out
);
return res.content.map((b) => (b.type === "text" ? b.text : "")).join("");
});
}With that small amount of code, identical requests running concurrently in the same process always converge to one. In my nightly batch, this alone dropped the peak duplication from 4.2 to 1.1 calls per minute. The remaining duplicates came from "another process / another worker." That is what we fold next.