Tail Latency in Scheduled Claude API Workloads: A Three-Layer Guardrail Against Retry Storms

Three patterns that blow up the tail

Sorting the p99-plus samples from those 14 days separated cleanly into three repeatable patterns. These are not specific to my environment — anyone driving scheduled Claude API workloads should expect to see at least two of them.

1. Model-side jitter during global peak windows. Concentrated on JST 7-9 in the morning and 18-23 in the evening, which overlaps with global peaks. Responses still come back, but the spread of latency widens to three or four times the off-peak baseline.
2. Client-side retry collisions. Multiple scheduled tasks land on the same endpoint within a narrow window. A single 429/529 nudges all of them into retry at once, and the next slot starts in resonance. In my case this happened around 04:00 when 02:00, 02:45, 04:00, and 04:45 were all stacking.
3. Long system prompt plus large input on first call. When the prompt cache is cold and input crosses roughly 12,000 tokens, the tail curve bends upward. The model is fine; the request is just expensive.

Guardrail 1: jittered exponential backoff with hard caps

The first move is to put deliberate jitter into the retry interval and to cap the maximum attempts hard. This is the implementation I am running today.

// retry.ts
type RetryConfig = {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
  jitterRatio: number;
};
 
const DEFAULT: RetryConfig = {
  maxAttempts: 3,
  baseDelayMs: 1_000,
  maxDelayMs: 30_000,
  jitterRatio: 0.5,
};
 
export async function retryWithJitter<T>(
  fn: (attempt: number) => Promise<T>,
  cfg: Partial<RetryConfig> = {},
): Promise<T> {
  const c = { ...DEFAULT, ...cfg };
  let lastErr: unknown;
  for (let attempt = 1; attempt <= c.maxAttempts; attempt++) {
    try {
      return await fn(attempt);
    } catch (err) {
      lastErr = err;
      if (attempt === c.maxAttempts) break;
      if (!isRetryable(err)) break;
      const expo = Math.min(
        c.baseDelayMs * Math.pow(2, attempt - 1),
        c.maxDelayMs,
      );
      const jitter = expo * c.jitterRatio * Math.random();
      const wait = expo + jitter - (expo * c.jitterRatio) / 2;
      await sleep(wait);
    }
  }
  throw lastErr;
}
 
function isRetryable(err: unknown): boolean {
  if (!err || typeof err !== "object") return false;
  const status = (err as { status?: number }).status;
  return status === 429 || status === 529 || (status !== undefined && status >= 500);
}
 
const sleep = (ms: number) =>
  new Promise((resolve) => setTimeout(resolve, ms));

Three details matter. First, the cap is three attempts. The previous 4.6 average came from a chain of "just five for safety" and "let me try ten more" settings layered on top of each other; pulling the ceiling down was more effective than I expected. Second, the jitter is a symmetric wobble around the expected value rather than full jitter — full jitter is fine, but when several calls run sequentially in one process the aggregate tends to compress short. Third, the retryable set is tight: 429, 529, and 5xx. The old client was retrying 4xx-request errors, which is wasted load.

Guardrail 2: a per-minute budget so I stop hammering myself

Capping retries in one call still allows other tasks to flood the same endpoint. Six parallel sites can build a storm even if each one is well behaved. A second layer watches the total call rate and refuses to dispatch when the window fills.

// budget.ts
type BudgetState = {
  windowStartMs: number;
  windowSizeMs: number;
  callCount: number;
  maxCalls: number;
};
 
const STATE: Map<string, BudgetState> = new Map();
 
export function tryConsume(
  key: string,
  maxCalls: number,
  windowSizeMs = 60_000,
): boolean {
  const now = Date.now();
  const s = STATE.get(key) ?? {
    windowStartMs: now,
    windowSizeMs,
    callCount: 0,
    maxCalls,
  };
  if (now - s.windowStartMs > s.windowSizeMs) {
    s.windowStartMs = now;
    s.callCount = 0;
  }
  if (s.callCount >= s.maxCalls) {
    STATE.set(key, s);
    return false;
  }
  s.callCount += 1;
  STATE.set(key, s);
  return true;
}
 
export async function withBudget<T>(
  key: string,
  maxCalls: number,
  fn: () => Promise<T>,
): Promise<T | null> {
  if (!tryConsume(key, maxCalls)) return null;
  return fn();
}

A 30-call-per-minute ceiling, returning null when the budget is gone. For an article generation pipeline this is the right shape of failure: the task that hits the ceiling just gives up gracefully, and the later slot picks up the work. Once this layer was in place, the cascade where a bad ten minutes would knock out the whole night cleared up.

Guardrail 3: a thin circuit breaker for the upstream

Jitter and budget keep me from running myself over. When the upstream itself goes wobbly, I want a third layer that stops requesting entirely until it stabilizes. This is the smallest circuit breaker I could justify in the codebase.

// breaker.ts
type BreakerState = {
  failures: number;
  successes: number;
  openedAt: number | null;
  cooldownMs: number;
  failureRatio: number;
  minSamples: number;
};
 
const BREAKERS: Map<string, BreakerState> = new Map();
 
export function shouldAttempt(key: string): boolean {
  const b = BREAKERS.get(key);
  if (!b || b.openedAt === null) return true;
  if (Date.now() - b.openedAt > b.cooldownMs) {
    b.openedAt = null;
    b.failures = 0;
    b.successes = 0;
    return true;
  }
  return false;
}
 
export function recordResult(key: string, ok: boolean) {
  const b = BREAKERS.get(key) ?? {
    failures: 0,
    successes: 0,
    openedAt: null,
    cooldownMs: 60_000,
    failureRatio: 0.5,
    minSamples: 10,
  };
  if (ok) b.successes += 1;
  else b.failures += 1;
  const total = b.successes + b.failures;
  if (total >= b.minSamples && b.failures / total >= b.failureRatio) {
    b.openedAt = Date.now();
  }
  BREAKERS.set(key, b);
}

The first version opened after five samples and burned me when a flurry of 5xx kept the breaker stuck for half an hour, taking out the rest of the schedule with it. Ten samples plus a 60-second cooldown turned out to be the sweet spot for my volume. Tune this against your own traffic; the wrong values are worse than no breaker.

Stop creating retry collisions in the scheduler itself

Three guardrails on the client side still cannot prevent a scheduler from launching everything at once. For Cowork scheduled tasks I follow three rules that have held up well.

• Put minutes of space between adjacent tasks. My rotation runs at 02:00, 02:45, 04:00, 04:45 — about 45-minute gaps where it can be afforded.
• Avoid the prime windows: JST 7-9 and 18-23. Both global peak and personal-attention peak.
• Keep at least 15 minutes between tasks that share an API key, regardless of which site they belong to.

Off-peak scheduling and minute-level offsets are easy to skip when you set up a system, and they are usually the first improvement that pays back. The full daily plan for the four Dolice Labs sites lives in the operations notes, but the principle generalizes to any solo Claude API workload.

What changed after the guardrails went in

Here are the same metrics measured across the 14 days following the rollout (through 2026-05-26).

p50 barely moved. That is the result I wanted: the guardrails cut tail without slowing the happy path. The p99 drop tracks the 59% reduction in wasted retries almost perfectly.

The 17 yielded calls split into 8 budget refusals and 9 breaker openings, all of which were picked up cleanly by the next slot. For a content-generation workload, this is the correct failure mode. Give up now, save the next task.

What to keep and what to leave out as a solo operator

The whole point of writing this down was not to ship a perfect retry framework. It was to share the smallest set of guardrails that holds up when one person is running six production sites. I have leaned on the habit my mentor passed on to me at 17 — that the work is to be made open and reachable — and that has meant choosing tools that are simple enough for me to maintain alone at 4am.

What I kept: jittered exponential backoff, a per-minute budget, a thin circuit breaker, and a scheduler that respects off-peak windows. What I deliberately did not implement: external metric collectors, ML anomaly detection, a globally consistent token bucket. The latter group works at scale; the cost of operating them does not pencil out below a certain headcount.

If you are running Claude API on a scheduled workload as an indie developer or small team, start by graphing p95 and p99 rather than the mean. From there, layer in whichever of the three guardrails best matches the pattern you see first. Thank you for reading — I will be back to watch the 02:00 run tonight and see what else is left to trim.