The moment your local messages.create call works perfectly, and the moment that same code is serving real users, are separated by a much wider gap than most teams expect. I have personally been woken up at 3 a.m. by alerts that said "Claude is silent," "the bill just exploded," and "users are getting empty responses" — more times than I want to admit.
Each time, the post-mortem pointed to the same conclusion: the failure was not in Claude itself, but in the infrastructure I had not yet built around it. So this is a backwards walk through that pain — eight pieces of plumbing I now refuse to ship a Claude-powered feature without.
Why "It Works" and "It Holds Up" Are Different Animals
A messages.create call that succeeds on your laptop is quietly relying on four things being true:
- The network is stable
- No one else is hammering the same API key
- The response comes back in seconds
- A human is watching when something fails
In production, all four collapse at some point. AWS region flips, a teammate's batch script firing in parallel, dropped connections mid-stream, midnight outages — every team eventually meets all of them. Which is why the design work is less about the code itself and more about deciding, ahead of time, what should happen when each of those assumptions breaks.
1. Police Rate Limits on Your Side, Too
Anthropic's servers will return 429s, but waiting for them is not a strategy. Putting a small concurrency limiter on your own side keeps requests queued and predictable instead of hammering the upstream until it pushes back.
// lib/claude-limiter.ts
import Anthropic from "@anthropic-ai/sdk";
import pLimit from "p-limit"; // npm i p-limit
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY! });
const limit = pLimit(8); // cap to 8 concurrent requests
export async function safeCreate(params: Anthropic.MessageCreateParams) {
return limit(async () => {
const start = Date.now();
try {
const res = await client.messages.create(params);
console.log(JSON.stringify({ event: "claude_ok", ms: Date.now() - start }));
return res;
} catch (err: any) {
console.log(JSON.stringify({
event: "claude_err",
ms: Date.now() - start,
status: err.status,
type: err.error?.type,
}));
throw err;
}
});
}The expected output is a single JSON line per request that downstream tooling can aggregate. For a deeper treatment of rate-limit design, our rate limit best practices post goes into how to size the cap and combine it with token-aware throttling.
2. Layer a Real Timeout and Exponential Backoff
Claude responses can take tens of seconds. In practice, the failure mode is rarely "Claude never responded" — it is "your client gave up first." Don't trust the SDK's defaults; set them explicitly.
import { setTimeout as wait } from "node:timers/promises";
async function withRetry<T>(fn: () => Promise<T>, max = 3): Promise<T> {
for (let attempt = 0; ; attempt++) {
try {
return await fn();
} catch (err: any) {
const isRetriable = [429, 500, 502, 503, 504, 529].includes(err.status);
if (!isRetriable || attempt >= max) throw err;
const backoff = Math.min(2 ** attempt * 500 + Math.random() * 200, 8000);
await wait(backoff);
}
}
}529 is the code Anthropic uses to signal upstream overload. Forgetting to add it to the retriable list will, on busy days, surface a wall of errors directly to your users.
3. Turn Prompt Caching On Yesterday
If you keep sending the same long system prompt with every request, your bill at the end of the month will surprise you in a bad way. Properly applied prompt caching cuts the cost of repeated system prompts by up to 90%.
The mechanic is simple — add cache_control: { type: "ephemeral" } to the system block — but the gotchas (cache misses on tool changes, ordering rules) are real. I wrote them all up in the prompt-caching cost-halving playbook so you don't repeat my mistakes.
4. Have at Least One Fallback Route
Anthropic's status page goes red a few times a year. "Claude was down so we were down" is not a story you want to tell paying customers. Pick at least one fallback — Sonnet → Haiku, Bedrock-via-AWS, or Vertex AI on GCP — and wire it in early.
Perfect failover is not the goal. The goal is graceful degradation: when the primary path is unhealthy, users still receive something, even if quality drops. Our multi-model fallback guide walks through the routing logic, and if you already use Cloudflare, the AI Gateway production guide shows how to do it without writing the proxy yourself.
One mental model that helps: write down, for each user-facing feature, the minimum acceptable degraded experience. For a customer-support chat, that might be "fall back to a canned response with a link to the FAQ." For a search feature, "fall back to plain keyword search." Once you've articulated that, the engineering reduces to "make sure we always at least reach the degraded state," which is a much easier engineering problem than "achieve full Claude-quality at all times."
5. Structured Logs and Tracing — From Day One
The moment you find yourself thinking "I wish I had logged this," it is already too late. Ship structured logs from the very first day. The four fields I never skip:
- A request ID (your own UUID and Anthropic's
request_id) - Input and output token counts
- The model used
- A terminal status (success / how many retries / final error code)
If you have the budget for full OpenTelemetry tracing, even better — token-by-token latency distributions become observable. The OpenTelemetry observability guide shows the instrumentation in code.
A small but meaningful detail: log the prompt template name, not the prompt text itself. Logging full prompts gets you into compliance trouble fast (PII, customer data, GDPR) and quickly bloats your log retention bill. A short identifier like prompt: "summary.v3" is enough to debug 90% of "why did the answer change?" conversations, without ever leaving the boundary of what's safe to store.
6. Set Two Layers of Billing Defense
The Anthropic Console exposes both a Spend Alert and a Spend Limit. Use both. Alerts alone will not stop a runaway script in the middle of the night; limits alone will leave you blind for hours.
Add a third layer of your own — a dashboard that surfaces "tokens consumed in the last hour." If something starts misbehaving, you'll see the slope before the alert fires. A Slack webhook into a dedicated incident channel is the cheapest possible version of this.
7. Always Validate the Output
Forwarding raw LLM output to downstream systems is the moral equivalent of leaving a pipe disconnected. If you expect JSON, validate it with zod. If you generate HTML, run it through DOMPurify. If you build SQL, parameterize it — always. That is the floor.
import { z } from "zod";
const ClaudeOutputSchema = z.object({
intent: z.enum(["summarize", "translate", "search"]),
query: z.string().min(1).max(500),
language: z.string().length(2).optional(),
});
function parseClaude(text: string) {
const parsed = JSON.parse(text); // throw → retry
return ClaudeOutputSchema.parse(parsed); // throw → fallback
}Even with response_format: { type: "json_object" }, malformed JSON occasionally slips through. Decide ahead of time what happens when "the case that should never happen" finally happens.
The most expensive bug I've personally shipped came from skipping this step: a model occasionally returned a JSON string with smart quotes instead of straight quotes, the parser threw, the request retried three times, and the whole pipeline spent budget on the same broken response. A four-line zod check caught it the next day.
8. Keep a Path That Reaches a Human
The last item is operational, not technical. When something breaks in production, who learns about it, through which channel, within how many minutes? An infrastructure that hasn't answered that question won't hold up no matter how clean the code is. My personal floor is three things:
- Slack alerts with mentions when error rate crosses a threshold
- A user-facing error screen that offers both "retry" and "contact support"
- A rotation — even a few hours per person per month — of who is on call
You don't need a full SRE program on day one. You do need to ensure that the time spent silently broken is close to zero.
A small psychological trick that helps: write the runbook before the incident. Even one paragraph titled "what to do when Claude is unhealthy" makes the 3 a.m. version of you immensely more competent than the version that has to figure it out from a blank page.
A Realistic First Step Today
Trying to ship all eight at once is how teams burn out. Pick two for today: structured logs, and timeout + retry. Once those are in, even if everything else is unbuilt, you can at least reconstruct what happened during an incident. And what you can reconstruct, you can fix.
If you want to push further into hallucination defense and production-grade quality patterns, the companion piece in this series — a multi-layer hallucination defense architecture for Claude API — picks up where this one leaves off.