When I first started building on the Claude API, I assumed Anthropic's side was stable enough that I didn't need to bother with my own redundancy. Honestly, more than 99.9% of calls do come back clean. The problem is that the remaining 0.1% has a habit of clustering — a brief regional degradation, a rate limit that only hits one model, a timeout that only appears on long prompts — and of all landing together on the same evening peak. When they do, a single-model integration goes down with them.
This article is the production playbook I actually use for putting Sonnet, Opus, and Haiku into a fallback chain that degrades gracefully instead of failing. By the end, you should have the code to migrate from "total outage when the primary model blinks" to "keep serving with slightly lower quality and keep the user happy." I'll share the code, the numbers I use for thresholds, and — most importantly — the pitfalls I've hit that aren't covered anywhere else.
Three moments that break a single-model design
Before the code, a short tour of why redundancy is worth the trouble. These are three patterns I hit in the field and spent real days recovering from.
- Rate limits only hitting one model. Running out of Sonnet requests-per-minute while Haiku on the same account still has plenty of headroom is routine. Without a chain, that 429 becomes a red error screen for every user, even though you had plenty of quota sitting unused on a sister model.
- Timeouts triggered only by long prompts. The same endpoint returns a 10k-token prompt in 30 seconds and times out on a 100k-token RAG payload. This is a localized failure caused by your prompt shape, not by the model. If you only have one model configured, every long-context request becomes an error — even though a slightly different model (or a chunked retry) would have served it.
- Quality variance right after a model rollout. Newly rolled-out models occasionally run 2–3× slower than usual, especially with Extended Thinking, for a few hours. Being able to route that traffic to the previous Sonnet line, or to Haiku when strict accuracy isn't needed, keeps the perceived quality steady while the new model stabilizes.
Each has a different root cause, but they share one mitigation: keep a chain of models so that when the primary stumbles, the request falls through to a working alternative. It's not a perfect fix, but it's vastly better than nothing, and it's cheap to build once you've done it a second time.
Three axes for designing the chain
Which models you pick, and in what order, depends on the product. Here are the three axes I always check during a design review. Getting these wrong tends to produce chains that look reasonable on paper but behave badly under real load.
- Quality tolerance. Is this a flow that genuinely requires Sonnet 4 (complex reasoning, coding), or is Haiku fine (FAQs, summaries, simple classification)? Wider tolerance gives you more fallback choices. Narrow tolerance means you might need to send an error rather than a degraded answer, and that's okay.
- Cost ceiling. Putting Opus everywhere will blow up the monthly bill. If Opus sits on the fallback path, an incident translates directly into a spike in spend. Decide the ceiling up front — I usually set a daily spend cap per model and let the fallback skip models that have passed the cap that day.
- Latency SLO. For products that want sub-2-second responses, putting Extended-Thinking Opus on the fallback path breaks the SLO worse than a failed request. Speed-first stacks should move Haiku up the chain, even as the primary in some cases.
Three standard chains I keep on the shelf:
- Quality-first:
claude-sonnet-4-6→claude-opus-4-6→claude-haiku-4-5-20251001 - Speed-first:
claude-sonnet-4-6→claude-haiku-4-5-20251001→ error (Opus excluded) - Cost-first:
claude-haiku-4-5-20251001→claude-sonnet-4-6→ error (try Haiku first)
Most real products end up using different chains for different features — speed-first for dashboard summaries, quality-first for the user-facing answer generation, cost-first for internal batch jobs. Mixing is fine. What's not fine is using one chain for everything and pretending the features have the same requirements.
Baseline: a serial fallback client in TypeScript
We'll start without a circuit breaker. Just getting this into production already avoids 80% of outages, and it's the cheapest change you can ship this week.
What this code solves: on a single Claude API request, iterate through the chain and only fall through on 429 (rate-limited), 529 (overloaded), 5xx (server errors), and timeouts. Other errors surface immediately so you don't waste latency on retries that will all return the same failure.
// claude-fallback-client.ts
import Anthropic from "@anthropic-ai/sdk";
type FallbackChain = readonly string[];
interface FallbackConfig {
readonly models: FallbackChain;
readonly perCallTimeoutMs: number;
readonly onFallback?: (fromModel: string, toModel: string, reason: string) => void;
}
// default quality-first chain
const DEFAULT_CHAIN: FallbackChain = [
"claude-sonnet-4-6",
"claude-opus-4-6",
"claude-haiku-4-5-20251001",
] as const;
export class ClaudeFallbackClient {
private readonly client: Anthropic;
constructor(apiKey: string) {
this.client = new Anthropic({ apiKey });
}
async createMessage(
params: Omit<Anthropic.MessageCreateParamsNonStreaming, "model">,
config: Partial<FallbackConfig> = {},
): Promise<{ message: Anthropic.Message; modelUsed: string }> {
const chain = config.models ?? DEFAULT_CHAIN;
const timeoutMs = config.perCallTimeoutMs ?? 30_000;
const errors: { model: string; error: unknown }[] = [];
for (let i = 0; i < chain.length; i++) {
const model = chain[i];
try {
const message = await this.callWithTimeout(
() => this.client.messages.create({ ...params, model }),
timeoutMs,
);
return { message, modelUsed: model };
} catch (err) {
errors.push({ model, error: err });
if (!this.shouldFallback(err) || i === chain.length - 1) {
// 400, 401 etc. should not fall through; last model failing is terminal
throw new AggregateError(
errors.map((e) => e.error),
`All models in chain failed: ${errors.map((e) => e.model).join(" → ")}`,
);
}
const next = chain[i + 1];
config.onFallback?.(model, next, this.describeError(err));
}
}
throw new Error("unreachable");
}
private shouldFallback(err: unknown): boolean {
if (err instanceof Anthropic.APIError) {
if (err.status === 429 || err.status === 529) return true;
if (err.status !== undefined && err.status >= 500) return true;
return false;
}
// timeouts and network errors are eligible for fallback
if (err instanceof Error) {
if (err.name === "AbortError") return true;
if (err.message.includes("ECONNRESET") || err.message.includes("ETIMEDOUT")) return true;
}
return false;
}
private describeError(err: unknown): string {
if (err instanceof Anthropic.APIError) return `${err.status} ${err.name}`;
if (err instanceof Error) return err.name;
return "unknown";
}
private async callWithTimeout<T>(fn: () => Promise<T>, ms: number): Promise<T> {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), ms);
try {
return await fn();
} finally {
clearTimeout(timer);
}
}
}
// example
const client = new ClaudeFallbackClient(process.env.ANTHROPIC_API_KEY!);
const { message, modelUsed } = await client.createMessage(
{
max_tokens: 1024,
messages: [{ role: "user", content: "Summarize quantum tunneling in 3 sentences." }],
},
{
onFallback: (from, to, reason) => {
console.warn(`[fallback] ${from} → ${to} (${reason})`);
},
},
);
console.log(`[ok] model=${modelUsed}`);
console.log(message.content[0].type === "text" ? message.content[0].text : "");The important thing here is that shouldFallback is explicit about which errors deserve a retry with the next model. Falling through on 400 or 401 just burns latency because every model will return the same error. I've reviewed a lot of fallback code that blurred this line, and every one of those codebases turned incident logs into mush — when the real problem was, say, a malformed prompt, every log line showed "Sonnet failed, Opus failed, Haiku failed" without the one error message that actually mattered.
Another subtle point: the per-call timeout wraps each model attempt individually. If you use a global timeout on the whole chain, a slow primary starves the fallback. Individual per-call timeouts give every model a fair shot.
Adding a circuit breaker so the primary stops costing you latency
A serial chain alone has a hidden cost: when the primary is fully down, every single request spends a timeout on the primary before hitting the alternative. Under load this compounds into global latency regression — requests pile up waiting for a primary that will never answer, connections leak, and the error rate climbs even on the fallback model because the request queue has backed up behind doomed primary attempts. A circuit breaker fixes this by short-circuiting the primary while it is known bad.
What this code solves: when the primary fails repeatedly in a short window, mark it "open" and skip it in subsequent calls until a cool-off period passes. Then probe with one request before fully re-enabling. The state machine is deliberately simple — three states — because I've never seen a four-state breaker be worth the complexity.
// circuit-breaker.ts
type CircuitState = "closed" | "open" | "half-open";
interface BreakerOptions {
readonly failureThreshold: number;
readonly openDurationMs: number;
readonly halfOpenTrialLimit: number;
}
const DEFAULTS: BreakerOptions = {
failureThreshold: 5,
openDurationMs: 30_000,
halfOpenTrialLimit: 1,
};
export class CircuitBreaker {
private state: CircuitState = "closed";
private failureCount = 0;
private openedAt = 0;
private halfOpenTrials = 0;
constructor(private readonly name: string, private readonly opts: BreakerOptions = DEFAULTS) {}
canExecute(): boolean {
if (this.state === "closed") return true;
if (this.state === "open") {
if (Date.now() - this.openedAt >= this.opts.openDurationMs) {
this.state = "half-open";
this.halfOpenTrials = 0;
return true;
}
return false;
}
return this.halfOpenTrials < this.opts.halfOpenTrialLimit;
}
onSuccess(): void {
this.failureCount = 0;
if (this.state !== "closed") this.state = "closed";
}
onFailure(): void {
this.failureCount += 1;
if (this.state === "half-open") {
this.halfOpenTrials += 1;
this.state = "open";
this.openedAt = Date.now();
return;
}
if (this.failureCount >= this.opts.failureThreshold) {
this.state = "open";
this.openedAt = Date.now();
}
}
describe(): string {
return `${this.name}[${this.state} failures=${this.failureCount}]`;
}
}
// wire it into ClaudeFallbackClient:
const breakers = new Map<string, CircuitBreaker>();
function getBreaker(model: string): CircuitBreaker {
let breaker = breakers.get(model);
if (!breaker) {
breaker = new CircuitBreaker(model);
breakers.set(model, breaker);
}
return breaker;
}
// inside the for-loop in createMessage:
// const breaker = getBreaker(model);
// if (!breaker.canExecute()) continue; // skip if open
// try {
// const message = await this.callWithTimeout(...);
// breaker.onSuccess();
// return { message, modelUsed: model };
// } catch (err) {
// if (this.shouldFallback(err)) breaker.onFailure();
// // ...
// }The thresholds I've settled on from production are failureThreshold: 5, openDurationMs: 30s, halfOpenTrialLimit: 1. Smaller thresholds make the breaker too twitchy — it keeps the primary offline after it has actually recovered, costing you quality opportunities. Larger ones keep sending to a known-dead model and blow the SLO. Expect to retune these every couple of weeks as your traffic profile shifts. If you use a request coordinator like Redis, you can also have the breaker state shared across instances, which matters once you scale horizontally (see pitfall 2 below).
One pattern I'll flag here because it trips people up: don't increment failureCount on non-fallback errors. If a request returns 400 because the prompt was malformed, that's not the model's fault, and it shouldn't count toward the breaker. Only count the error types that shouldFallback returns true for.
Streaming fallback — where the simple approach breaks down
Unary fallback is straightforward. Streaming is harder, because once tokens have started flowing, "just retry with the next model" is not a good user experience. The user has seen half a sentence; you can't un-show it. Any design that pretends otherwise will produce visibly broken output.
In production I split streaming fallback into two cases:
- Errors before the stream has started (a 429 or 5xx on the initial HTTP response): treat as if no stream opened; fall through to the next model. The user has seen nothing yet, so no experience is broken. This case handles most real-world streaming failures, because most incidents — rate limiting, overload, server-side errors — surface at connection establishment.
- Errors after tokens have started flowing: do not silently retry. Tag the stream with a visible marker like "connection interrupted, re-sending" and let the caller explicitly request a new generation. Mixing tokens from two models mid-stream creates incoherent text because of tokenizer differences, even when the models are from the same family.
What this code solves: differentiates start-time errors (fall through) from mid-stream errors (surface visibly), and avoids the trap of trying to stitch two models' outputs together.
// stream-fallback.ts
import Anthropic from "@anthropic-ai/sdk";
export async function streamWithFallback(
client: Anthropic,
params: Omit<Anthropic.MessageStreamParams, "model">,
chain: readonly string[],
onDelta: (text: string) => void,
): Promise<{ modelUsed: string; completedCleanly: boolean }> {
for (let i = 0; i < chain.length; i++) {
const model = chain[i];
try {
const stream = client.messages.stream({ ...params, model });
let streamStarted = false;
for await (const event of stream) {
if (!streamStarted && event.type === "message_start") {
streamStarted = true;
}
if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
onDelta(event.delta.text);
}
}
return { modelUsed: model, completedCleanly: true };
} catch (err) {
if (!isStreamStartError(err) || i === chain.length - 1) {
// after-start failure: surface it and let the caller resend
onDelta("\n\n*[the connection dropped mid-response; we'll re-send]*");
return { modelUsed: model, completedCleanly: false };
}
// start-time failure: continue to the next model silently
}
}
throw new Error("unreachable");
}
function isStreamStartError(err: unknown): boolean {
if (err instanceof Anthropic.APIError) {
return err.status === 429 || err.status === 529 || (err.status ?? 0) >= 500;
}
return false;
}My blunt lesson from running this in production: do not try to stitch mid-stream output from two different models. People ask about it a lot and it seems elegant on paper, but the seams are always visible to users. A clean cut plus an explicit re-send message yields a higher satisfaction score than any stitching approach I've tried. The user experience of "we had a hiccup, generating again" is familiar from every chat product on the market — it's far less jarring than receiving a paragraph that suddenly changes voice mid-sentence.
Backoff strategy — because retrying immediately is often worse than failing
A point that tutorials almost always skip: between chain hops, you may or may not want to back off. The default serial chain above retries the next model immediately, which is correct for connection-independent failures (a 5xx on Sonnet has nothing to do with Opus). But when the issue is account-wide — a concurrent-request cap, a shared billing rail overload — immediately hammering the next model produces the same error.
The heuristic I use: on 429 specifically, read the retry-after header if present and respect it before the next chain hop. Other errors can skip straight to the next model. Here's the pattern:
function extractRetryAfterMs(err: unknown): number | null {
if (err instanceof Anthropic.APIError && err.status === 429) {
const retryAfter = err.headers?.["retry-after"];
if (retryAfter) {
const seconds = Number(retryAfter);
if (!isNaN(seconds)) return seconds * 1000;
}
}
return null;
}
// inside the for-loop, before continuing:
const waitMs = extractRetryAfterMs(err);
if (waitMs !== null && waitMs < 10_000) {
await new Promise((r) => setTimeout(r, waitMs));
}
// a hard cap of 10s prevents a pathological server from making you wait minutesThe hard cap matters — a poorly behaved upstream can return an absurd retry-after value, and blindly honoring it stalls your whole request. Keeping the cap at 10 seconds is a pragmatic compromise for most real-time user-facing flows.
Pitfalls that are not obvious until they hit you
Even with everything above, these are the traps I've personally stepped on more than once.
- Pitfall 1: Falling through on every 4xx. If you fall back on 400 or 401, you just pay 3× the latency while every model returns the same error. Restrict fallback to 429 / 529 / 5xx / timeouts. Treat other 4xx as user-caused or config-caused; log and return them.
- Pitfall 2: Per-process breaker state in a multi-instance deployment. If you run on multiple servers but each process keeps the breaker in memory, you end up with instances disagreeing on whether Sonnet is dead. At scale, store breaker state in Redis (or a similar shared store) so the fleet agrees. Until you do, you'll notice weird "some requests succeed, some fail" patterns during a partial incident.
- Pitfall 3: Putting Opus at the end of the chain as a "safety net". It's tempting — "fall back to the best model". In practice, when the primary fails, all traffic shifts to Opus, which is slower and more expensive. That breaks latency and burns budget simultaneously. Use Opus where you need its quality on the first hop, not as a catch-all tail.
- Pitfall 4: Hiding the fallback from the user entirely. The instinct is to make it invisible. That's fine when output quality is genuinely similar. But when it isn't — especially in B2B SaaS — users notice and file confused support tickets. A small "currently in lightweight mode" label increases trust more than hiding it.
- Pitfall 5: Logging fallback events synchronously into a database. I've seen implementations where
onFallbackdoes a synchronous DB write. When the DB is under pressure (often during the same incident), fallback itself starts blocking. Always push events through an async queue, or at minimum fire-and-forget.
Pitfalls 1 and 3 are my own scars — cases where "we added fallback but made the outage worse." Both are invisible in design review and catastrophic in production. Put them on a checklist for any PR that touches the fallback client.
Operational signals worth watching
Adding fallback is not the finish line. The fact that fallback fires at all is itself a signal worth watching. These are the five metrics I always put on the on-call dashboard:
- Per-model success rate. 200 OK rate per model. If the primary drops below 99.5%, investigate — it's usually not a single broken request, it's the start of a wave.
- Fallback rate. Percentage of requests where a fallback fired. Normal is well under 0.1%; incidents spike it. A slow creep upward is often the earliest signal of a regression.
- Chain-end reached rate. Requests where even the last model failed. Anything above 0.01% is an incident worth paging on.
- Per-model cost. A fallback storm into Opus inflates cost sharply. Set a daily threshold alert that fires when cost exceeds the expected daily average by more than, say, 2×.
- Circuit breaker transitions. closed → open → half-open → closed. Useful for noticing when a model has actually recovered and for sanity-checking that your thresholds aren't too trigger-happy.
Cloudflare AI Gateway or OpenTelemetry make all of this dramatically easier. Related reading: Claude API OpenTelemetry observability guide and Cloudflare AI Gateway production monitoring.
Testing the fallback path — a rehearsal that actually runs
The worst thing about fallback code is that you only find out it's broken when you need it. I've seen fallback clients where the chain was written correctly, the circuit breaker was correct, but an off-by-one in error classification made fallback never fire — and nobody noticed until production burned. The only defense is to run a rehearsal.
What this code solves: a simulated primary failure in a staging or test environment, where you can verify that the fallback chain actually kicks in and emits the metrics you expect. Ship this into your test suite and run it nightly.
// fallback.test.ts
import { describe, it, expect, vi } from "vitest";
import { ClaudeFallbackClient } from "./claude-fallback-client";
describe("ClaudeFallbackClient", () => {
it("falls through 429 to the next model", async () => {
const fallbackEvents: string[] = [];
const client = new ClaudeFallbackClient("test-key");
// Stub the underlying SDK: primary throws 429, secondary returns a Message
vi.spyOn((client as any).client.messages, "create")
.mockImplementationOnce(async () => {
const err: any = new Error("rate_limit");
err.status = 429;
err.name = "RateLimitError";
Object.setPrototypeOf(err, (await import("@anthropic-ai/sdk")).default.APIError.prototype);
throw err;
})
.mockImplementationOnce(async () => ({
id: "msg_2",
content: [{ type: "text", text: "ok" }],
role: "assistant",
}));
const { modelUsed } = await client.createMessage(
{ max_tokens: 16, messages: [{ role: "user", content: "hi" }] },
{
models: ["claude-sonnet-4-6", "claude-haiku-4-5-20251001"],
onFallback: (from, to) => fallbackEvents.push(`${from}->${to}`),
},
);
expect(modelUsed).toBe("claude-haiku-4-5-20251001");
expect(fallbackEvents).toEqual(["claude-sonnet-4-6->claude-haiku-4-5-20251001"]);
});
it("does not fall through on 400", async () => {
const client = new ClaudeFallbackClient("test-key");
vi.spyOn((client as any).client.messages, "create").mockImplementation(async () => {
const err: any = new Error("bad_request");
err.status = 400;
err.name = "BadRequestError";
Object.setPrototypeOf(err, (await import("@anthropic-ai/sdk")).default.APIError.prototype);
throw err;
});
await expect(
client.createMessage(
{ max_tokens: 16, messages: [{ role: "user", content: "hi" }] },
{ models: ["claude-sonnet-4-6", "claude-haiku-4-5-20251001"] },
),
).rejects.toThrow();
});
});The second test — "does not fall through on 400" — is the one that catches the most common regression I've seen in fallback clients: somebody loosens the error classification during a refactor, and suddenly every malformed prompt does three round-trips before erroring. If you only test the happy path, you won't catch it.
For stagi a more realistic rehearsal, I periodically set up a synthetic incident window in staging where the primary model is forced to return 5xx via an API gateway rule. The fallback client should degrade gracefully during that window and recover automatically when the rule is removed. Running this drill once a quarter is enough to keep the fallback path from rotting.
Deployment pattern — how the chain changes as your product grows
One thing I wish I'd known earlier: the right chain composition changes as your product matures, and you should plan for that explicitly rather than treating the chain as a one-time decision.
- Early stage (single-digit QPS): use the speed-first chain (Sonnet → Haiku). Your traffic is small enough that rate limits are effectively impossible, so the chain is mostly defending against transient network errors. Don't bother with breakers yet.
- Growth stage (tens to hundreds of QPS): move to the quality-first chain and add circuit breakers with Redis-shared state. Rate limits become a real thing at this scale, and per-process breakers start lying to you.
- Scale stage (thousands of QPS): split the chain by feature. Introduce per-feature cost ceilings. Add regional redundancy if you're running in multiple data centers. This is also the stage where dedicated model endpoints (if available) start to pay off.
Treating the chain as a static config, even a well-designed one, produces a slow-burning liability. The shape of your traffic will change, and the chain needs to evolve with it. Code review on the fallback client should be a recurring calendar event, not a one-off task.
Closing — first step is an honest audit of your single points of failure
If after reading this you realize "we're running on Sonnet alone," give yourself one hour this week to drop in the baseline ClaudeFallbackClient. Skip the breaker and the monitoring for now. Set the chain to [sonnet, haiku], log onFallback events, and wait a few days to see your real fallback rate. Once you have that number, you'll know whether to prioritize the circuit breaker next or whether to think about Opus placement. That sequence is the most cost-efficient path to high availability — you build what your actual traffic tells you to build, not what you assumed was worth building. In my experience, teams that try to boil the ocean on resilience upfront usually end up with a beautifully architected system that still has a basic gap somewhere critical, because attention was everywhere instead of on the failure modes that actually fire. Iterate, measure, and fix the real leak.
High availability is a thing you strengthen in stages, not something you ship all at once. Today's one hour is probably worth a future 3 AM pager, and that trade is almost always worth taking. And once the first version is running, treat every incident as data — each one tells you which parameter to tune, which metric to add, or which chain to split. Over a few iterations, your fallback client goes from "the thing that kept the site up once" to "a load-bearing piece of the product" that you trust without thinking.