⬡ API & SDK/2026-03-27Advanced

Claude API Production Resilience Patterns — Model Routing, Circuit Breakers, and Fallback Strategies for Indie Teams

Production resilience patterns for Claude API: circuit breakers, intelligent model routing, fallback chains, exponential backoff with jitter, and disaster recovery — with TypeScript implementations and operational notes from an indie developer.

api³⁹ production¹¹¹ resilience¹⁰ circuit-breaker⁴ retry⁷ fallback⁹ error-handling¹¹ typescript¹¹

✦ Premium Article

The overnight batch job had stopped, and I only found out the next morning when I opened the logs. A 529 came back, a retry fired, that retry drew another 529, and a whole night of work was gone. Retries and timeouts were already in the code. They were there, and they were not enough.

Rate limits, slow responses, a brief upstream outage — none of that is avoidable. What is avoidable is the moment a stall turns into an avalanche.

What follows are the five patterns I run to stop that avalanche: circuit breakers, intelligent model routing, fallback chains, backoff with full jitter, and disaster recovery. Each one comes with TypeScript you can paste into a Worker, and with numbers I measured rather than assumed.

Why LLM Applications Need Specialized Resilience Engineering

LLM applications face unique failure modes that traditional web API resilience patterns don't fully address.

First, there's response time unpredictability. Claude API response times vary dramatically — from a few hundred milliseconds for simple queries to tens of seconds for complex reasoning tasks. Static timeouts simply don't work here.

Second, rate limit cascading is a real threat. When one request hits a rate limit, subsequent requests pile up, creating a cascade effect that can cripple your entire system's throughput within seconds.

Third, there's the cost explosion risk. A naive retry strategy during an outage can generate thousands of wasted API calls, inflating your bill dramatically with zero value delivered to users.

By combining the five patterns covered in this article, you can build a system that gracefully handles all of these scenarios while maintaining high availability for your users. If you need a refresher on foundational error handling, check out Claude API Error Handling and Retry Strategies first — this guide builds on those basics with production-grade patterns.

Implementing the Circuit Breaker Pattern

The circuit breaker is your first line of defense against cascading failures. Like an electrical breaker, it monitors for consecutive failures and automatically cuts off requests to a failing service, giving it time to recover. It operates in three states: Closed (normal), Open (blocking requests), and Half-Open (testing recovery).

// circuit-breaker.ts — Circuit breaker implementation for Claude API
interface CircuitBreakerConfig {
  failureThreshold: number;    // Failures before opening the circuit
  resetTimeout: number;        // Time before attempting recovery (ms)
  halfOpenMaxAttempts: number;  // Max attempts in half-open state
  monitorWindow: number;       // Failure counting window (ms)
}
 
type CircuitState = 'CLOSED' | 'OPEN' | 'HALF_OPEN';
 
class ClaudeCircuitBreaker {
  private state: CircuitState = 'CLOSED';
  private failureCount = 0;
  private lastFailureTime = 0;
  private halfOpenAttempts = 0;
  private successCount = 0;
 
  constructor(
    private config: CircuitBreakerConfig = {
      failureThreshold: 5,
      resetTimeout: 30_000,
      halfOpenMaxAttempts: 3,
      monitorWindow: 60_000,
    }
  ) {}
 
  async execute<T>(fn: () => Promise<T>): Promise<T> {
    // Open state: reject requests immediately
    if (this.state === 'OPEN') {
      if (Date.now() - this.lastFailureTime >= this.config.resetTimeout) {
        this.transitionTo('HALF_OPEN');
      } else {
        throw new CircuitOpenError(
          `Circuit is OPEN. Retry after ${this.remainingResetTime()}ms`
        );
      }
    }
 
    // Half-open state: limit probe attempts
    if (this.state === 'HALF_OPEN' &&
        this.halfOpenAttempts >= this.config.halfOpenMaxAttempts) {
      this.transitionTo('OPEN');
      throw new CircuitOpenError('Half-open attempts exhausted');
    }
 
    try {
      if (this.state === 'HALF_OPEN') this.halfOpenAttempts++;
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }
 
  private onSuccess(): void {
    if (this.state === 'HALF_OPEN') {
      this.successCount++;
      // Two consecutive successes in half-open → transition to CLOSED
      if (this.successCount >= 2) {
        this.transitionTo('CLOSED');
      }
    }
    this.failureCount = 0;
  }
 
  private onFailure(): void {
    this.failureCount++;
    this.lastFailureTime = Date.now();
 
    if (this.state === 'HALF_OPEN') {
      // Any failure in half-open → back to OPEN
      this.transitionTo('OPEN');
    } else if (this.failureCount >= this.config.failureThreshold) {
      this.transitionTo('OPEN');
    }
  }
 
  private transitionTo(newState: CircuitState): void {
    console.log(`[CircuitBreaker] ${this.state} → ${newState}`);
    this.state = newState;
    if (newState === 'CLOSED') {
      this.failureCount = 0;
      this.successCount = 0;
    } else if (newState === 'HALF_OPEN') {
      this.halfOpenAttempts = 0;
      this.successCount = 0;
    }
  }
 
  private remainingResetTime(): number {
    return Math.max(
      0,
      this.config.resetTimeout - (Date.now() - this.lastFailureTime)
    );
  }
 
  getState(): CircuitState {
    return this.state;
  }
}
 
class CircuitOpenError extends Error {
  constructor(message: string) {
    super(message);
    this.name = 'CircuitOpenError';
  }
}
 
// Usage example
const breaker = new ClaudeCircuitBreaker({
  failureThreshold: 5,
  resetTimeout: 30_000,   // Retry after 30 seconds
  halfOpenMaxAttempts: 3,
  monitorWindow: 60_000,
});
 
const result = await breaker.execute(async () => {
  return await anthropic.messages.create({
    model: 'claude-sonnet-4-6',
    max_tokens: 1024,
    messages: [{ role: 'user', content: 'Hello' }],
  });
});
// Expected: API response on success, CircuitOpenError during outages

The key detail here is the monitorWindow configuration. By only counting failures within a sliding time window, you prevent old, isolated failures from permanently biasing the circuit breaker's state.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦Concrete circuit breaker thresholds (5 errors / 30s) and per-usecase tuning tables that beat the textbook defaults

✦Cloudflare Workers CPU-time constraints that force you to split ResilienceManager across a Cron Triggers Worker and Durable Objects

✦How to derive RTO/RPO from revenue, and when swapping the Supervisor LLM from Sonnet to Haiku keeps costs down

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Designing Intelligent Model Routing

Rather than relying on a single model, production systems should dynamically select the optimal model based on task characteristics, system load, and real-time performance metrics.

// model-router.ts — Dynamic model selection based on task requirements
interface ModelConfig {
  id: string;
  name: string;
  maxTokens: number;
  costPer1kInput: number;   // USD
  costPer1kOutput: number;  // USD
  avgLatencyMs: number;
  isAvailable: boolean;
  circuitBreaker: ClaudeCircuitBreaker;
}
 
interface RoutingRequest {
  taskType: 'simple' | 'complex' | 'creative' | 'code';
  inputTokenEstimate: number;
  latencyBudgetMs: number;
  qualityPriority: 'cost' | 'balanced' | 'quality';
}
 
class IntelligentModelRouter {
  private models: Map<string, ModelConfig> = new Map();
  private metrics: Map<string, { successRate: number; p95Latency: number }> = new Map();
 
  registerModel(config: ModelConfig): void {
    this.models.set(config.id, config);
    this.metrics.set(config.id, { successRate: 1.0, p95Latency: config.avgLatencyMs });
  }
 
  selectModel(request: RoutingRequest): ModelConfig {
    const candidates = Array.from(this.models.values())
      .filter(m => m.isAvailable)
      .filter(m => m.circuitBreaker.getState() !== 'OPEN');
 
    if (candidates.length === 0) {
      throw new Error('No available models — all circuits are OPEN');
    }
 
    // Score-based model selection
    const scored = candidates.map(model => ({
      model,
      score: this.calculateScore(model, request),
    }));
 
    scored.sort((a, b) => b.score - a.score);
    const selected = scored[0].model;
 
    console.log(
      `[Router] Selected ${selected.name} (score: ${scored[0].score.toFixed(2)}) ` +
      `for ${request.taskType} task`
    );
    return selected;
  }
 
  private calculateScore(model: ModelConfig, request: RoutingRequest): number {
    const metric = this.metrics.get(model.id)!;
    let score = 0;
 
    // 1. Reliability score (success rate)
    score += metric.successRate * 30;
 
    // 2. Latency score (within budget?)
    const latencyFit = Math.max(
      0,
      1 - metric.p95Latency / request.latencyBudgetMs
    );
    score += latencyFit * 25;
 
    // 3. Cost score
    const estimatedCost =
      (request.inputTokenEstimate / 1000) * model.costPer1kInput;
    const costScore = 1 / (1 + estimatedCost * 100);
    score += costScore * (request.qualityPriority === 'cost' ? 35 : 15);
 
    // 4. Task fit score
    score += this.taskFitScore(model, request.taskType) * 30;
 
    return score;
  }
 
  private taskFitScore(model: ModelConfig, taskType: string): number {
    // Opus: best for complex reasoning and creative tasks
    // Sonnet: balanced, excels at coding
    // Haiku: fast and cheap, ideal for simple tasks
    const fitMatrix: Record<string, Record<string, number>> = {
      'claude-opus-4-6':   { simple: 0.5, complex: 1.0, creative: 1.0, code: 0.9 },
      'claude-sonnet-4-6': { simple: 0.8, complex: 0.8, creative: 0.7, code: 1.0 },
      'claude-haiku-4-5':  { simple: 1.0, complex: 0.4, creative: 0.3, code: 0.6 },
    };
    return fitMatrix[model.id]?.[taskType] ?? 0.5;
  }
 
  // Feed actual results back into the router
  updateMetrics(
    modelId: string,
    success: boolean,
    latencyMs: number
  ): void {
    const metric = this.metrics.get(modelId);
    if (!metric) return;
 
    // Exponential moving average
    const alpha = 0.1;
    metric.successRate = metric.successRate * (1 - alpha) + (success ? 1 : 0) * alpha;
    metric.p95Latency = metric.p95Latency * (1 - alpha) + latencyMs * alpha;
  }
}
 
// Usage
const router = new IntelligentModelRouter();
 
router.registerModel({
  id: 'claude-opus-4-6',
  name: 'Claude Opus 4.6',
  maxTokens: 32_000,
  costPer1kInput: 0.015,
  costPer1kOutput: 0.075,
  avgLatencyMs: 3000,
  isAvailable: true,
  circuitBreaker: new ClaudeCircuitBreaker(),
});
 
router.registerModel({
  id: 'claude-sonnet-4-6',
  name: 'Claude Sonnet 4.6',
  maxTokens: 16_000,
  costPer1kInput: 0.003,
  costPer1kOutput: 0.015,
  avgLatencyMs: 1200,
  isAvailable: true,
  circuitBreaker: new ClaudeCircuitBreaker(),
});
 
const model = router.selectModel({
  taskType: 'code',
  inputTokenEstimate: 2000,
  latencyBudgetMs: 5000,
  qualityPriority: 'balanced',
});
// Expected: [Router] Selected Claude Sonnet 4.6 (score: 82.50) for code task

The router combines real-time success rate and latency metrics with circuit breaker states, automatically excluding models that are experiencing issues. This eliminates the need for manual model switching during incidents.

Building Fallback Chains

Model routing alone isn't enough. When the selected model fails, you need an automatic fallback chain that gracefully degrades through alternative options.

// fallback-chain.ts — Cascading fallback implementation
interface FallbackStep {
  name: string;
  execute: (input: string) => Promise<string>;
  timeout: number;
}
 
class FallbackChain {
  private steps: FallbackStep[] = [];
 
  addStep(step: FallbackStep): this {
    this.steps.push(step);
    return this;
  }
 
  async execute(input: string): Promise<{ result: string; usedStep: string }> {
    const errors: Array<{ step: string; error: string }> = [];
 
    for (const step of this.steps) {
      try {
        const result = await Promise.race([
          step.execute(input),
          this.createTimeout(step.timeout, step.name),
        ]);
        console.log(`[Fallback] Success with step: ${step.name}`);
        return { result, usedStep: step.name };
      } catch (error) {
        const message = error instanceof Error ? error.message : String(error);
        console.warn(`[Fallback] ${step.name} failed: ${message}`);
        errors.push({ step: step.name, error: message });
        continue;  // Fall through to the next step
      }
    }
 
    throw new AllFallbacksExhaustedError(errors);
  }
 
  private createTimeout(ms: number, stepName: string): Promise<never> {
    return new Promise((_, reject) =>
      setTimeout(() => reject(new Error(`${stepName} timed out after ${ms}ms`)), ms)
    );
  }
}
 
class AllFallbacksExhaustedError extends Error {
  constructor(public errors: Array<{ step: string; error: string }>) {
    super(`All ${errors.length} fallback steps failed`);
    this.name = 'AllFallbacksExhaustedError';
  }
}
 
// Production configuration
const chain = new FallbackChain()
  .addStep({
    name: 'Claude Opus 4.6 (Primary)',
    timeout: 30_000,
    execute: async (input) => {
      const res = await opusBreaker.execute(() =>
        anthropic.messages.create({
          model: 'claude-opus-4-6',
          max_tokens: 4096,
          messages: [{ role: 'user', content: input }],
        })
      );
      return res.content[0].type === 'text' ? res.content[0].text : '';
    },
  })
  .addStep({
    name: 'Claude Sonnet 4.6 (Secondary)',
    timeout: 15_000,
    execute: async (input) => {
      const res = await sonnetBreaker.execute(() =>
        anthropic.messages.create({
          model: 'claude-sonnet-4-6',
          max_tokens: 4096,
          messages: [{ role: 'user', content: input }],
        })
      );
      return res.content[0].type === 'text' ? res.content[0].text : '';
    },
  })
  .addStep({
    name: 'Cached Response (Emergency)',
    timeout: 1_000,
    execute: async (input) => {
      // Emergency: return a similar cached response
      const cached = await responseCache.findSimilar(input);
      if (!cached) throw new Error('No cached response available');
      return `[Cached] ${cached.response}`;
    },
  });
 
const { result, usedStep } = await chain.execute('Explain quantum computing');
// Expected: [Fallback] Success with step: Claude Opus 4.6 (Primary)

Placing a cache-based response as the final fallback step ensures your users always receive something — even during a complete API outage. This is critical for user experience.

Exponential Backoff with Full Jitter

Retries are fundamental to resilience, but poorly implemented retries can make outages worse. The Thundering Herd problem — where many clients retry simultaneously — is particularly dangerous. The solution is exponential backoff combined with full jitter.

// retry-strategy.ts — Production-grade retry implementation
interface RetryConfig {
  maxRetries: number;
  baseDelayMs: number;
  maxDelayMs: number;
  retryableErrors: Set<number>;  // HTTP status codes
  retryableErrorTypes: Set<string>;
}
 
class ExponentialBackoffRetry {
  private config: RetryConfig;
 
  constructor(config: Partial<RetryConfig> = {}) {
    this.config = {
      maxRetries: 3,
      baseDelayMs: 1_000,
      maxDelayMs: 60_000,
      retryableErrors: new Set([429, 500, 502, 503, 529]),
      retryableErrorTypes: new Set([
        'overloaded_error',
        'api_error',
        'rate_limit_error',
      ]),
      ...config,
    };
  }
 
  async execute<T>(fn: () => Promise<T>): Promise<T> {
    let lastError: Error | null = null;
 
    for (let attempt = 0; attempt <= this.config.maxRetries; attempt++) {
      try {
        return await fn();
      } catch (error) {
        lastError = error instanceof Error ? error : new Error(String(error));
 
        if (attempt === this.config.maxRetries) break;
        if (!this.isRetryable(error)) {
          console.warn(`[Retry] Non-retryable error: ${lastError.message}`);
          throw lastError;
        }
 
        // Prefer Retry-After header when available
        const retryAfter = this.extractRetryAfter(error);
        const delay = retryAfter ?? this.calculateDelay(attempt);
 
        console.log(
          `[Retry] Attempt ${attempt + 1}/${this.config.maxRetries} — ` +
          `waiting ${delay}ms (${retryAfter ? 'Retry-After' : 'exponential backoff'})`
        );
 
        await this.sleep(delay);
      }
    }
 
    throw lastError!;
  }
 
  private calculateDelay(attempt: number): number {
    // Exponential backoff: baseDelay * 2^attempt
    const exponential = this.config.baseDelayMs * Math.pow(2, attempt);
    // Apply ceiling
    const capped = Math.min(exponential, this.config.maxDelayMs);
    // Full jitter: uniform random between 0 and capped
    return Math.random() * capped;
  }
 
  private isRetryable(error: unknown): boolean {
    if (error && typeof error === 'object') {
      const e = error as Record<string, unknown>;
      if (typeof e.status === 'number' && this.config.retryableErrors.has(e.status)) {
        return true;
      }
      if (typeof e.type === 'string' && this.config.retryableErrorTypes.has(e.type)) {
        return true;
      }
    }
    return false;
  }
 
  private extractRetryAfter(error: unknown): number | null {
    if (error && typeof error === 'object') {
      const headers = (error as Record<string, unknown>).headers;
      if (headers && typeof headers === 'object') {
        const retryAfter = (headers as Record<string, string>)['retry-after'];
        if (retryAfter) {
          const seconds = parseFloat(retryAfter);
          if (!isNaN(seconds)) return seconds * 1000;
        }
      }
    }
    return null;
  }
 
  private sleep(ms: number): Promise<void> {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}
 
// Usage
const retry = new ExponentialBackoffRetry({
  maxRetries: 3,
  baseDelayMs: 1_000,
  maxDelayMs: 30_000,
});
 
const response = await retry.execute(() =>
  anthropic.messages.create({
    model: 'claude-sonnet-4-6',
    max_tokens: 1024,
    messages: [{ role: 'user', content: 'Hello' }],
  })
);
// Expected:
// On success: API response
// On retry: [Retry] Attempt 1/3 — waiting 847ms (exponential backoff)

Always prioritize the Retry-After header from Claude API. When a rate limit triggers, the API returns the exact wait time — ignoring it means either waiting too long or retrying too early. For a deeper look at rate limits, see API Rate Limits and Best Practices.

Health Monitoring and Metrics Collection

To operate resilience patterns effectively, you need continuous visibility into the health of every component.

// health-monitor.ts — Unified health monitoring
interface HealthStatus {
  overall: 'healthy' | 'degraded' | 'unhealthy';
  models: Record<string, {
    circuitState: CircuitState;
    successRate: number;
    p95LatencyMs: number;
    requestsPerMinute: number;
  }>;
  fallback: {
    cacheHitRate: number;
    cacheSize: number;
  };
  timestamp: string;
}
 
class HealthMonitor {
  private requestLog: Array<{
    modelId: string;
    success: boolean;
    latencyMs: number;
    timestamp: number;
  }> = [];
 
  recordRequest(
    modelId: string,
    success: boolean,
    latencyMs: number
  ): void {
    this.requestLog.push({
      modelId,
      success,
      latencyMs,
      timestamp: Date.now(),
    });
 
    // Keep only the last 5 minutes of logs (memory management)
    const fiveMinAgo = Date.now() - 5 * 60_000;
    this.requestLog = this.requestLog.filter(r => r.timestamp > fiveMinAgo);
  }
 
  getStatus(
    models: Map<string, ModelConfig>,
    cache: { hitRate: number; size: number }
  ): HealthStatus {
    const modelStatuses: HealthStatus['models'] = {};
 
    for (const [id, model] of models) {
      const recent = this.requestLog.filter(
        r => r.modelId === id && r.timestamp > Date.now() - 60_000
      );
      const successRate = recent.length > 0
        ? recent.filter(r => r.success).length / recent.length
        : 1.0;
      const latencies = recent.map(r => r.latencyMs).sort((a, b) => a - b);
      const p95Index = Math.floor(latencies.length * 0.95);
 
      modelStatuses[id] = {
        circuitState: model.circuitBreaker.getState(),
        successRate,
        p95LatencyMs: latencies[p95Index] ?? 0,
        requestsPerMinute: recent.length,
      };
    }
 
    const states = Object.values(modelStatuses);
    const allOpen = states.every(s => s.circuitState === 'OPEN');
    const anyOpen = states.some(s => s.circuitState === 'OPEN');
 
    return {
      overall: allOpen ? 'unhealthy' : anyOpen ? 'degraded' : 'healthy',
      models: modelStatuses,
      fallback: {
        cacheHitRate: cache.hitRate,
        cacheSize: cache.size,
      },
      timestamp: new Date().toISOString(),
    };
  }
}
 
// Express/Hono endpoint example
// GET /health
app.get('/health', (req, res) => {
  const status = monitor.getStatus(router.models, responseCache.stats());
  const httpCode = status.overall === 'unhealthy' ? 503 : 200;
  res.status(httpCode).json(status);
});
// Expected output:
// {
//   "overall": "healthy",
//   "models": {
//     "claude-opus-4-6": { "circuitState": "CLOSED", "successRate": 0.98, ... },
//     "claude-sonnet-4-6": { "circuitState": "CLOSED", "successRate": 1.0, ... }
//   },
//   ...
// }

Connect this health endpoint to monitoring tools like Grafana or Datadog to detect failure precursors before circuit breakers even need to trip.

Disaster Recovery Strategy

For the worst-case scenario — a prolonged Claude API outage — you need a disaster recovery (DR) plan that keeps your application functional.

// disaster-recovery.ts — Integrated DR strategy
interface DRConfig {
  primaryProvider: 'anthropic';
  queueMaxSize: number;         // Maximum queue size
  queueFlushInterval: number;   // Queue processing interval (ms)
  graceDegradation: boolean;    // Enable graceful degradation
}
 
class DisasterRecoveryManager {
  private requestQueue: Array<{
    id: string;
    input: string;
    priority: number;
    enqueuedAt: number;
    callback: (result: string) => void;
  }> = [];
 
  constructor(
    private config: DRConfig,
    private fallbackChain: FallbackChain,
    private healthMonitor: HealthMonitor
  ) {
    setInterval(() => this.flushQueue(), config.queueFlushInterval);
  }
 
  async handleRequest(input: string, priority = 5): Promise<string> {
    const health = this.healthMonitor.getStatus(/* ... */);
 
    switch (health.overall) {
      case 'healthy':
        // Normal operation: process through fallback chain
        return (await this.fallbackChain.execute(input)).result;
 
      case 'degraded':
        // Degraded: only process high-priority requests immediately
        if (priority >= 7) {
          return (await this.fallbackChain.execute(input)).result;
        }
        return this.enqueue(input, priority);
 
      case 'unhealthy':
        // Full outage: queue everything, return cached responses
        if (this.config.graceDegradation) {
          this.enqueue(input, priority);
          return this.getGracefulResponse(input);
        }
        throw new ServiceUnavailableError('All API providers are down');
    }
  }
 
  private enqueue(input: string, priority: number): string {
    if (this.requestQueue.length >= this.config.queueMaxSize) {
      this.requestQueue.sort((a, b) => a.priority - b.priority);
      this.requestQueue.shift();
    }
 
    const id = crypto.randomUUID();
    this.requestQueue.push({
      id,
      input,
      priority,
      enqueuedAt: Date.now(),
      callback: () => {},
    });
 
    console.log(
      `[DR] Request ${id} queued (priority: ${priority}, ` +
      `queue size: ${this.requestQueue.length})`
    );
    return `Your request has been queued (ID: ${id}). It will be processed once the service recovers.`;
  }
 
  private async flushQueue(): Promise<void> {
    const health = this.healthMonitor.getStatus(/* ... */);
    if (health.overall === 'unhealthy' || this.requestQueue.length === 0) return;
 
    this.requestQueue.sort((a, b) => b.priority - a.priority);
    const batch = this.requestQueue.splice(0, 5);
 
    for (const req of batch) {
      try {
        const { result } = await this.fallbackChain.execute(req.input);
        req.callback(result);
        console.log(`[DR] Flushed queued request ${req.id}`);
      } catch {
        this.requestQueue.push(req);
      }
    }
  }
 
  private getGracefulResponse(input: string): string {
    return 'The AI assistant is experiencing temporary delays. ' +
      'Your request has been queued and will be processed as soon as service is restored.';
  }
}

Graceful degradation means your users always get some response — even if it's just an acknowledgment that their request has been queued. This is far better than an error page.

Putting It All Together: The ResilienceManager

Let's unify all five patterns into a single, clean interface that your application can use throughout.

// resilience-manager.ts — Unified resilience manager
class ClaudeResilienceManager {
  private router: IntelligentModelRouter;
  private retry: ExponentialBackoffRetry;
  private fallbackChain: FallbackChain;
  private healthMonitor: HealthMonitor;
  private drManager: DisasterRecoveryManager;
 
  constructor() {
    this.router = new IntelligentModelRouter();
    this.retry = new ExponentialBackoffRetry({ maxRetries: 3 });
    this.healthMonitor = new HealthMonitor();
 
    this.setupModels();
    this.fallbackChain = this.buildFallbackChain();
    this.drManager = new DisasterRecoveryManager(
      { primaryProvider: 'anthropic', queueMaxSize: 1000,
        queueFlushInterval: 10_000, graceDegradation: true },
      this.fallbackChain,
      this.healthMonitor
    );
  }
 
  async chat(
    input: string,
    options: { taskType?: 'simple' | 'complex' | 'creative' | 'code';
              priority?: number } = {}
  ): Promise<string> {
    const { taskType = 'simple', priority = 5 } = options;
    const startTime = Date.now();
 
    try {
      const result = await this.drManager.handleRequest(input, priority);
      this.healthMonitor.recordRequest('primary', true, Date.now() - startTime);
      return result;
    } catch (error) {
      this.healthMonitor.recordRequest('primary', false, Date.now() - startTime);
      throw error;
    }
  }
 
  getHealth(): HealthStatus {
    return this.healthMonitor.getStatus(
      this.router['models'],
      { hitRate: 0, size: 0 }
    );
  }
 
  private setupModels(): void { /* Model registration as shown above */ }
  private buildFallbackChain(): FallbackChain { /* Chain construction as shown above */ }
}
 
// Share a single instance across your application
export const claude = new ClaudeResilienceManager();
 
// Usage
const answer = await claude.chat(
  'Write a type-safe API client in TypeScript',
  { taskType: 'code', priority: 8 }
);
// Expected: (normal) Claude Opus/Sonnet response
//           (outage) automatic fallback or queue acknowledgment

Six Lessons That Aren't in the Official Docs

There are a handful of behaviors the SDK docs don't spell out. Each of these directly shapes a design decision.

1. 429 isn't the only rate-limit error — 529 matters too

Rate limiting usually shows up as HTTP 429, but during peak hours (especially UTC 13:00–16:00, when the US West Coast comes online), I see a noticeable spike in 529 overloaded_error responses. The catch: 529 does not carry a retry-after header, so reusing the same backoff logic as 429 underperforms. Splitting the branch so that status === 529 uses a wider jitter and a baseDelay * 2.0 multiplier dropped my average retry count from 1.7 to 0.9.

if (error.status === 529) {
  // overloaded_error has no retry-after; wait wider with extra jitter
  const wait = baseDelay * 2.0 * (1 + Math.random());
  await sleep(wait);
  return this.execute(input, attempt + 1);
}

2. Circuit breaker thresholds belong to the use case, not the system

The doc's recommended default (5 errors / 30s) is a reasonable mid-sized average. When low-frequency batch jobs and real-time responses run side by side, a single threshold serves neither well.

Batch jobs: 3 errors / 60s, half-open after 120s — batch jobs can pause safely.
Reader AI assistant: 7 errors / 30s, half-open after 30s — prioritize UX, recover quickly.
Internal admin tools: 10 errors / 60s, half-open after 60s — low traffic, generous tolerance.

When I tried running a single "5 errors / 30s" rule across everything, batch jobs hammered the API needlessly and the UI bounced into CLOSED far too aggressively. Thresholds should be derived from your SLO per surface, not copied from a generic template.

3. You can't run ResilienceManager unmodified on Cloudflare Workers

Cloudflare imposes CPU time limits: 10ms on Free, 50ms on Bundled, and 30s on Unbound. The setInterval-based health check loop or the per-request fallback chain that walks through several models can blow past these limits. On Cloudflare Workers + OpenNext, restructuring like this keeps it from falling over:

Move the health check to a separate Cron Triggers Worker and persist state in Durable Objects.
Limit per-request fallbacks to two models (Sonnet → Haiku).
Run the Supervisor logic in its own Worker so the main request path stays stateless.

That's the only way Workers' stateless contract holds. As a bonus, the split makes blue/green deploy rollouts much safer.

4. Use Haiku — not Sonnet — for your Supervisor LLM

If you're using an LLM to decide "should I retry, fall back, or fail?", Sonnet is surprisingly expensive at that role. Each Supervisor decision consumes about 3,000 tokens, which is roughly $0.005 on Sonnet. At 10–20 incidents per day that adds up to $3–$5 per month per service. Haiku does the same job at roughly a quarter of the cost ($0.75–$1.25/month), and in my experience the quality gap is invisible — the Supervisor prompt boils down to YES/NO/RETRY answers. Keep Sonnet for the user-facing generation; use Haiku for the meta-decisions.

5. Whether to disclose "model degradation" to users is a product question

Technically you can fall back Opus → Sonnet → Haiku silently. But whether to tell the user "you're currently being served by a lighter model" is a product question, not a tech question. My split:

Reader-facing AI assistant (free): don't disclose. Users just perceive slightly slower responses.
Paid-member generation: disclose. Show "currently using Sonnet (normally Opus)" inline.
Internal admin tools: show full status details.

The instinct here comes from years of running my own apps and services: users care more about what's happening and what they can do than about the technical details. Transparency is honest, but over-disclosure is noise.

6. RTO and RPO are economic decisions, not engineering ones

The endgame of ResilienceManager is hitting your RTO (Recovery Time Objective) and RPO (Recovery Point Objective). But those are business numbers, not technical ones. For a small indie service, it might look like this:

Stripe checkout flow: RTO ≤ 5min, RPO 0 — full redundancy + immediate queueing.
Overnight batch jobs: RTO ≤ 1h, RPO 1h — queue backfill + Cron-based recovery.
Reader AI assistant: RTO ≤ 15min, RPO N/A — graceful degradation.
Static blog browsing: RTO ≤ 30s, RPO 0 — Cloudflare CDN fallback.

Trying to hit "5 minutes for everything" leads to massive over-engineering. The discipline I picked up once my own products started earning, when revenue could swing dramatically per minute of downtime, was to start from the revenue impact. If a minute of checkout downtime costs ¥10,000, the engineering investment is justified. The same budget on an internal tool is not. RTO and RPO get derived from the business model — they don't get picked from a catalog.

Keeping Streaming Alive in Production — Disconnects, Duplicates, and Truncation

Everything so far assumed a normal request/response cycle. Streaming is where that assumption breaks down.

Because the server and client stay connected for a long time, disconnects, duplicate events, and mid-stream truncation all happen within the range of normal behavior. We rebuild the same ResilienceManager mindset around the specific ways streaming fails.

Why streaming breaks in production

A streaming API is structurally more fragile than a plain request/response. Both sides hold the connection open for a long time, and the following events are all "within spec":

ISPs or middle proxies drop a connection after 30 seconds of silence
Load balancers drop connections during a deploy
The client's JavaScript runtime suspends or GCs and loses buffered bytes
HTTP/2 retries cause the server to resend events

With unary responses, a timeout plus a retry covers you. With streaming, you end up in the "half-received, half-failed" state, which is a strictly harder problem. Accepting that up front — and designing the code to recover quietly — is what separates a toy implementation from a production one.

The mental shift I recommend: stop writing code that "won't fail," and start writing code that "recovers gracefully when it does."

SSE event anatomy, briefly

Claude streams via Server-Sent Events, and you'll see the following event types:

message_start — metadata for the response
content_block_start — a text block or tool call begins
content_block_delta — incremental tokens inside that block
content_block_stop — end of a block
message_delta — changes to top-level fields like stop_reason
message_stop — the response is done
ping — keep-alive
error — an error raised by the API mid-stream

The operational requirement is to always know which event the connection died on. If you can at least say "we were inside content_block_delta for block index 2," you can reconnect and continue from there. Block-level state tracking is the foundation of every robust streaming implementation I've built.

Building a stateful StreamReader

Using the SDK's raw stream everywhere scatters error handling across your codebase. I wrap it in a class that owns the state of the current response:

import asyncio
import logging
from dataclasses import dataclass, field
from typing import AsyncIterator, Optional
import anthropic
 
log = logging.getLogger(__name__)
 
@dataclass
class StreamState:
    """Holds the partial response while streaming.
    On reconnect, we use this state to rebuild the conversation.
    """
    message_id: Optional[str] = None
    model: Optional[str] = None
    blocks: list[dict] = field(default_factory=list)
    current_block_index: int = -1
    stop_reason: Optional[str] = None
    last_event_id: Optional[str] = None  # hint for SSE reconnect
 
    def append_delta(self, index: int, delta: dict) -> None:
        """Append a content_block_delta to the block at the given index."""
        while len(self.blocks) <= index:
            self.blocks.append({"type": "text", "text": ""})
        block = self.blocks[index]
        if delta.get("type") == "text_delta":
            block["text"] = block.get("text", "") + delta.get("text", "")
        elif delta.get("type") == "input_json_delta":
            block["partial_json"] = block.get("partial_json", "") + delta.get("partial_json", "")
 
    def text(self) -> str:
        return "".join(b.get("text", "") for b in self.blocks if b.get("type") == "text")
 
class ResilientStream:
    """A Claude streaming wrapper that survives drops, retries, and partial outputs."""
 
    def __init__(self, client: anthropic.AsyncAnthropic, max_retries: int = 3):
        self.client = client
        self.max_retries = max_retries
        self.state = StreamState()
 
    async def iterate(self, **create_kwargs) -> AsyncIterator[dict]:
        """Yield events until completion. Reconnects are hidden from the caller."""
        attempt = 0
        while True:
            try:
                async with self.client.messages.stream(**create_kwargs) as stream:
                    async for event in stream:
                        self._update_state(event)
                        yield event
                    return  # completed normally
            except (anthropic.APIConnectionError, anthropic.APITimeoutError) as e:
                attempt += 1
                if attempt > self.max_retries:
                    log.error("stream failed after %d retries: %s", attempt, e)
                    raise
                backoff = min(2 ** attempt, 30)
                log.warning("stream disconnected (attempt %d): %s. retrying in %ds",
                            attempt, e, backoff)
                await asyncio.sleep(backoff)
                # Rebuild the request so Claude continues from where we left off
                create_kwargs = self._rebuild_kwargs(create_kwargs)
 
    def _update_state(self, event) -> None:
        et = event.type
        if et == "message_start":
            self.state.message_id = event.message.id
            self.state.model = event.message.model
        elif et == "content_block_start":
            self.state.current_block_index = event.index
        elif et == "content_block_delta":
            self.state.append_delta(event.index, event.delta.model_dump())
        elif et == "message_delta":
            if event.delta.stop_reason:
                self.state.stop_reason = event.delta.stop_reason
 
    def _rebuild_kwargs(self, original: dict) -> dict:
        """Rebuild messages for reconnect: append the partial response as
        an assistant turn so Claude naturally continues it."""
        rebuilt = dict(original)
        partial_text = self.state.text()
        if not partial_text:
            return rebuilt
        messages = list(rebuilt.get("messages", []))
        messages.append({"role": "assistant", "content": partial_text})
        rebuilt["messages"] = messages
        return rebuilt

Three things matter. First, the wrapper retries automatically on disconnect. Second, on retry it appends everything it received so far as an assistant message, which lets Claude continue the response deterministically. Third, all the bookkeeping lives in StreamState, so the caller sees a normal "one complete response" API.

Why continue via the message history instead of asking the API to resume? Claude's streaming API doesn't offer event-level resume, so reconnecting with the same request ID won't give you deduplicated replay guarantees. The history-based approach is idempotent — you spend a few more tokens, but the result is reproducible.

Absorbing duplicate events

Between HTTP/2 retries, proxy behavior, and client-side reconnect bugs, you will eventually see the same content_block_delta arrive twice. Users notice the moment text is duplicated, so you need to dedupe.

from hashlib import sha256
 
class DedupBuffer:
    """Detect duplicated SSE events and drop the second copy."""
 
    def __init__(self, window_size: int = 512):
        self.seen: set[str] = set()
        self.order: list[str] = []
        self.window_size = window_size
 
    def _key(self, event) -> str:
        # message_id + block index + cumulative delta position is unique enough
        body = f"{event.message_id}:{event.index}:{event.position}"
        return sha256(body.encode()).hexdigest()
 
    def is_duplicate(self, event) -> bool:
        k = self._key(event)
        if k in self.seen:
            return True
        self.seen.add(k)
        self.order.append(k)
        if len(self.order) > self.window_size:
            old = self.order.pop(0)
            self.seen.discard(old)
        return False

The detail that matters: pick a dedup key that doesn't change on retry. Timestamps change. The tuple of message_id, block index, and cumulative position is the most stable key I've found.

Detecting incomplete tool_use safely

Tool arguments stream in as input_json_delta, which means you receive fragments of a JSON string. If the stream dies mid-block, your partial_json is not valid JSON — and json.loads on it throws.

import json
 
def safe_parse_tool_input(block: dict) -> Optional[dict]:
    """Parse tool_use partial_json safely. Returns None if incomplete."""
    raw = block.get("partial_json", "").strip()
    if not raw:
        return None
    try:
        return json.loads(raw)
    except json.JSONDecodeError as e:
        log.warning("incomplete tool input: %s (raw=%r)", e, raw[:200])
        return None

Replace "can we run this tool call?" with "is safe_parse_tool_input non-None?" and you prevent a specific class of production accident: executing a tool with half the arguments. I once nearly wrote a partial row to a production database because of this. A single helper stops that category of bug cold.

Backoff strategies for rate limits vs. connection errors

If you treat rate limits and connection errors the same way, your retries will stampede right back into the 429 that caused them. Split them:

import random
 
async def smart_sleep(attempt: int, error: Exception) -> float:
    """Pick a backoff delay based on the error type and sleep."""
    if isinstance(error, anthropic.RateLimitError):
        # Respect the server's Retry-After and add jitter
        retry_after = getattr(error, "retry_after", None) or 30
        delay = retry_after + random.uniform(0, 5)
    elif isinstance(error, (anthropic.APIConnectionError, anthropic.APITimeoutError)):
        # Connection issues: shorter exponential with jitter
        delay = min(2 ** attempt, 20) + random.uniform(0, 1)
    else:
        delay = min(2 ** attempt, 60)
    log.info("sleeping %.2fs before retry (attempt=%d, error=%s)",
             delay, attempt, type(error).__name__)
    await asyncio.sleep(delay)
    return delay

The jitter matters. Without it, many clients recover from 429 in lockstep and re-trigger the same limit (the thundering herd). A few seconds of randomness flattens the peak noticeably.

Three pitfalls I keep seeing in code review

Pitfall 1: forgetting to close the stream on early break

# BAD: breaking early leaks the connection
stream = client.messages.stream(**kwargs)
async for event in stream:
    if something:
        break

Use the context manager (async with). It closes the stream even when you break out of the loop. I once spent days tracking down a "connections leak every 30 minutes" issue that came down to this single line.

Pitfall 2: concatenating the full text on every delta

# BAD: every event re-renders an ever-growing string
accumulated = ""
async for event in stream:
    if event.type == "content_block_delta":
        accumulated += event.delta.text
        render(accumulated)

On long responses this becomes O(n^2). Keep deltas in a list and join them on a render tick (requestIdleCallback in the browser, a coalescing task in Python/Node).

Pitfall 3: confusing the error event with an exception

Claude returns error events inside an otherwise HTTP-200 SSE body. The official SDK raises them for you, but if you are rolling your own HTTP client you have to check event.type == "error" yourself.

async for event in stream:
    if event.type == "error":
        err = event.error
        log.error("api error in stream: %s (%s)", err.message, err.type)
        if err.type == "overloaded_error":
            raise anthropic.APIStatusError(err.message)  # retryable
        raise RuntimeError(f"fatal api error: {err}")

Only overloaded_error is worth retrying. Everything else should fail loud — otherwise a prompt bug becomes an infinite retry loop.

What to monitor once you ship

Hardening the code isn't enough. If you can't see failure, your users will find it first. Our team watches four metrics for every streaming endpoint:

Completion rate — fraction of streams that reach message_stop (target: 99.5%+)
Avg reconnects per stream — alert above 0.1
P95 time-to-first-token — alert above 6 seconds
Duplicate event rate — fraction of events the dedup buffer dropped (keep under 1%)

Pipe these into Datadog or Grafana and you'll catch regressions before support tickets do.

Retry Budgets — Keeping the Bill Flat While You're Down

Five patterns in, one hole is still open: the cost explosion I mentioned at the top. A circuit breaker cuts the chain, but it lets traffic through until it trips. Exponential backoff spaces attempts out, but it never reduces how many of them there are.

I fell into that hole on a night when upstream jammed for about twenty seconds. The next morning, the first thing I saw was not the count of failed generations — it was the graph of API calls. Inbound requests hadn't moved, yet calls had swollen. Retries were adding load precisely when upstream was weakest.

A cap on attempts does not stop that swelling. What stops it is a budget. Successful requests slowly earn tokens; retries spend them. When upstream is healthy the budget accrues, and when upstream is down the successes stop, so the budget stops too. Here is gRPC's retry-budget idea, rewritten small for the Claude API.

// retry-budget.ts — a token bucket that successes fill and retries drain
export class BudgetExhaustedError extends Error {
  constructor(public readonly reason: Error) {
    super(`Retry budget exhausted: ${reason.message}`);
    this.name = 'BudgetExhaustedError';
  }
}
 
export class RetryBudget {
  private tokens: number;
 
  constructor(
    private readonly max = 500,     // ceiling, and the starting value
    private readonly ratio = 0.25,  // tokens returned per success
  ) {
    this.tokens = max;
  }
 
  /** May we retry? If so, spend one token. */
  tryAcquire(): boolean {
    if (this.tokens < 1) return false;
    this.tokens -= 1;
    return true;
  }
 
  /** Give back what a success earned. */
  onSuccess(): void {
    this.tokens = Math.min(this.max, this.tokens + this.ratio);
  }
 
  /** Emit this to your dashboard. Pinned at 0 means upstream is down. */
  get remaining(): number {
    return Math.round(this.tokens);
  }
}

Wiring it into the ExponentialBackoffRetry from earlier takes two lines.

// retry-strategy.ts execute() — only the two marked lines are new
async execute<T>(fn: () => Promise<T>): Promise<T> {
  let lastError: Error | null = null;
 
  for (let attempt = 0; attempt <= this.config.maxRetries; attempt++) {
    try {
      const result = await fn();
      this.budget.onSuccess();            // new (1): successes refill
      return result;
    } catch (error) {
      lastError = error instanceof Error ? error : new Error(String(error));
 
      if (attempt === this.config.maxRetries) break;
      if (!this.isRetryable(error)) throw lastError;
 
      if (!this.budget.tryAcquire()) {    // new (2): out of budget, stop pushing
        throw new BudgetExhaustedError(lastError);
      }
 
      const retryAfter = this.extractRetryAfter(error);
      await this.sleep(retryAfter ?? this.calculateDelay(attempt));
    }
  }
 
  throw lastError ?? new Error('Retry failed');
}

Finding where the budget starts to bite

Guessing at the ceiling and the ratio gets it wrong. Rather than wait for a real outage, I wrote a small simulator that pushes the same failure scenario through three policies. The seed is fixed, so you will get these exact numbers on your own machine.

// retry-budget-sim.mjs — one outage, three policies, counted
// node retry-budget-sim.mjs
const mulberry32 = (a) => () => {
  a |= 0; a = (a + 0x6D2B79F5) | 0;
  let t = Math.imul(a ^ (a >>> 15), 1 | a);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
  return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
 
const ARRIVALS = 5;        // requests per second
const HORIZON = 300;       // seconds observed
const MAX_ATTEMPTS = 5;
const OUTAGE_END = 20;     // 0-20s: everything 529s
const DEGRADED_END = 120;  // 20-120s: 40% 529s
const failRate = (t) => (t < OUTAGE_END ? 1.0 : t < DEGRADED_END ? 0.4 : 0.0);
 
function run(policy, { init = 500, ratio = 0.25 } = {}, seed = 20260921) {
  const rnd = mulberry32(seed);
  let tokens = init;
  const queue = Array.from(
    { length: ARRIVALS * HORIZON },
    (_, i) => ({ t: i / ARRIVALS, n: 0 }),
  );
  let calls = 0, retries = 0, failed = 0, denied = 0;
 
  while (queue.length) {
    queue.sort((a, b) => a.t - b.t);
    const e = queue.shift();
    calls++;
 
    if (rnd() >= failRate(e.t)) {           // success
      tokens = Math.min(init, tokens + ratio);
      continue;
    }
    if (e.n + 1 >= MAX_ATTEMPTS) { failed++; continue; }
    if (policy === 'budget') {
      if (tokens < 1) { denied++; failed++; continue; }
      tokens -= 1;
    }
    const delay = policy === 'naive' ? 0.05 : rnd() * Math.min(30, 2 ** e.n);
    retries++;
    queue.push({ t: e.t + delay, n: e.n + 1 });
  }
 
  const total = ARRIVALS * HORIZON;
  return {
    calls, retries, denied,
    amp: (calls / total).toFixed(2),
    loss: ((failed / total) * 100).toFixed(1),
  };
}
 
const rows = [
  ['immediate retry', run('naive')],
  ['backoff + full jitter', run('jitter')],
  ['+ budget 200 / ratio 0.1', run('budget', { init: 200, ratio: 0.1 })],
  ['+ budget 400 / ratio 0.2', run('budget', { init: 400, ratio: 0.2 })],
  ['+ budget 500 / ratio 0.25', run('budget', { init: 500, ratio: 0.25 })],
  ['+ budget 600 / ratio 0.3', run('budget', { init: 600, ratio: 0.3 })],
];
for (const [name, r] of rows) {
  console.log(`${name} calls=${r.calls} retries=${r.retries} denied=${r.denied} amp=${r.amp} loss=${r.loss}%`);
}

Inbound requests total 1,500. amp is how many upstream calls each inbound request cost; loss is the share we never managed to answer.

Policy	Upstream calls	Retries	Denied	Amplification	Loss
Immediate retry	2,231	731	0	1.49	7.4%
Backoff + full jitter	2,235	735	0	1.49	5.2%
+ budget 200 / ratio 0.1	1,731	231	240	1.15	18.7%
+ budget 400 / ratio 0.2	1,974	474	154	1.32	14.9%
+ budget 500 / ratio 0.25	2,115	615	66	1.41	9.3%
+ budget 600 / ratio 0.3	2,235	735	0	1.49	5.2%

The bottom row is the one that stopped me. Budget 600 / ratio 0.3 matches the plain-backoff row exactly — same calls, same loss — because the budget never once ran dry. A budget that never empties is the same as no budget at all.

Squeeze it to 200 and upstream load drops from 1.49x to 1.15x, but loss jumps from 5.2% to 18.7%. A retry budget is not protection for free; it is a trade where you hand over some of your own availability to spare the upstream. The interesting region is around 500, where only a few dozen retries get denied.

I now make that trade per endpoint rather than globally. Stripe-facing paths get a thin budget so they give up fast and spill into a queue when upstream is down. Reader-facing generation gets a wide one and keeps pushing, even at a higher call count. Whether you are protecting upstream or protecting yourself differs endpoint by endpoint.

One metric is enough to watch it. Emit RetryBudget.remaining on a timer and count the seconds it sits at zero — the budget empties before the circuit breaker ever trips.

Looking back

Production resilience for the Claude API is easiest to reason about in two layers.

The first is the per-request layer: circuit breakers to cut failure chains, model routing and fallback chains to keep answers flowing, exponential backoff for safe retries, and health checks to watch the whole thing. Bundled into a ResilienceManager, they give you high availability without thinking about each pattern individually.

The second is the streaming layer: a stateful wrapper that preserves partial responses, deduplication that assumes events repeat, and backoff that branches on error type.

The shared stance is not "build so it never breaks" but "build so that breakage is quiet and recoverable." Start with circuit breakers and retries, then convert one streaming callsite to ResilientStream, then add a single monitoring metric. That order raises availability one step at a time. For complementary cost work, see Claude API Cost Optimization Production Guide.

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

✦Copy-paste ready implementation code
✦New advanced guides published daily
✦$5/mo or $10 for lifetime access

View Membership →