The first instinct when Claude API feels slow is usually to swap to a smaller model. In my own production work, that's almost never been the right first move. More often than half the latency lives outside the model: the network round-trip to Anthropic, the way the SDK opens connections, and how much input the model has to re-process every call.
This post walks through four infrastructure levers I reach for before considering a model downgrade. None of them require changing prompts or accuracy targets — they just let your existing model serve users faster.
Why "make the model faster" misses half the budget
A Claude API request spends time across roughly four segments:
- Client to the Anthropic gateway (network in)
- Auth, queueing, and routing inside the gateway
- Actual model inference
- Network on the way back
Most tuning advice fixates on the inference step. But if you're calling from Tokyo and routing through us-east-1, the two network legs alone consume 200–400 ms. With Sonnet's typical inference around 500 ms, you're spending close to half your budget on transport. Trimming the model alone leaves a lot of speed on the table.
In my own services, the four levers below moved the needle further than any model swap I tried.
1. Region choice — physical distance still matters
Claude API is reachable through Anthropic's direct endpoint, AWS Bedrock, and Google Vertex AI. Each lives in different regions, so picking one closer to your servers shrinks Time To First Byte (TTFB) immediately and visibly.
Measuring from an ECS cluster in Tokyo, I see roughly:
- Anthropic direct (routed via us-east-1): ~180 ms RTT
- AWS Bedrock in us-west-2: ~130 ms RTT
- AWS Bedrock in ap-northeast-1: ~25 ms RTT
That last jump — from a US region to Tokyo Bedrock — is an order of magnitude. Bedrock's Claude has reached near-feature parity with the direct API, so for latency-sensitive workloads it's worth a serious look.
# bedrock_client.py
# Calls Claude through Bedrock in ap-northeast-1 (Tokyo).
# Typically yields the lowest TTFB when your compute is in Japan.
import boto3
import json
bedrock = boto3.client(
"bedrock-runtime",
region_name="ap-northeast-1", # pick the region closest to your compute
)
def ask_claude(prompt: str) -> str:
body = json.dumps({
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 1024,
"messages": [{"role": "user", "content": prompt}],
})
resp = bedrock.invoke_model(
modelId="anthropic.claude-sonnet-4-6-v1:0",
body=body,
)
payload = json.loads(resp["body"].read())
return payload["content"][0]["text"]
# Expected: TTFB around 150 ms or below from Tokyo-resident Lambda/ECS workloadsA region switch isn't a free win — confirm the model you need is offered there and that pricing matches your budget. AWS publishes the per-region model availability table; check it before you migrate any production traffic.
2. HTTP connection pooling — stop paying for handshakes
Opening a fresh TCP connection plus TLS handshake on every request costs 100–200 ms each time. Default fetch in Node.js or requests in Python doesn't always reuse connections, especially across short-lived function invocations.
The Anthropic SDK pools internally, but if you're rolling a thin custom client over fetch, you'll want to wire up an HTTP/2 or keep-alive agent explicitly.
// claude-client.ts
// Reuses a pooled connection via undici's Agent.
// Most impactful for batch jobs that fire many sequential requests.
import Anthropic from "@anthropic-ai/sdk";
import { Agent } from "undici";
const dispatcher = new Agent({
keepAliveTimeout: 60_000, // hold each connection for up to 60s
keepAliveMaxTimeout: 300_000, // cap at 5 minutes
connections: 32, // pool up to 32 concurrent sockets
pipelining: 1,
});
const client = new Anthropic({
apiKey: process.env.ANTHROPIC_API_KEY!,
fetch: (url, init) =>
// @ts-expect-error: inject undici dispatcher into fetch
fetch(url, { ...init, dispatcher }),
});
export async function chat(prompt: string) {
return client.messages.create({
model: "claude-sonnet-4-6",
max_tokens: 1024,
messages: [{ role: "user", content: prompt }],
});
}
// Expected: average latency across 100 sequential requests drops by ~80 ms
// versus default fetch with no dispatcherSaving a single handshake feels small. Saving one per call across a long-running agent loop or a nightly batch is the kind of win that compounds quietly into your SLO budget.
3. Prompt caching — skip 90% of input token processing
If you ship a long system prompt or a chunky reference document on every call, prompt caching is the cheapest latency win on the menu. When the cache hits, the model skips re-tokenizing and re-embedding the cached prefix, so TTFB shrinks substantially.
The mental model I use is to split each request into "the part that doesn't change" and "the part that does." My typical layout:
- Top: system instructions (often several thousand tokens — cached)
- Middle: reference docs or RAG context (cached)
- Bottom: the user's actual question (not cached)
// cached_messages.ts
// Two cache breakpoints — system instructions and a long reference block.
// Subsequent calls within the cache TTL skip most input processing.
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY! });
export async function answer(question: string, longContext: string) {
return client.messages.create({
model: "claude-sonnet-4-6",
max_tokens: 1024,
system: [
{
type: "text",
text: "You are an assistant fluent in our internal docs. Always cite sources.",
cache_control: { type: "ephemeral" }, // cache up to here
},
{
type: "text",
text: longContext, // tens of thousands of tokens
cache_control: { type: "ephemeral" }, // and up to here
},
],
messages: [{ role: "user", content: question }],
});
}
// Expected: response.usage.cache_read_input_tokens dominates on warm calls,
// and TTFB drops noticeably for the second and later requestsCache TTL is around five minutes, so the goal is to keep that window busy. Short, isolated chat sessions don't benefit much. Knowledge-base lookups or internal Q&A bots that hit the same context repeatedly often see the highest payoff.
4. Streaming — get the first character on screen under a second
What users feel as "fast" is when the first character lands, not when the last token closes the response. For chat UIs, writing assistants, or anything with longer output, stream: true should be the default.
// streaming.ts
// Yields tokens to the client as Claude produces them.
// Drastically improves perceived latency without changing the model.
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY! });
export async function* streamAnswer(prompt: string) {
const stream = await client.messages.stream({
model: "claude-sonnet-4-6",
max_tokens: 1024,
messages: [{ role: "user", content: prompt }],
});
for await (const event of stream) {
if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
yield event.delta.text;
}
}
}
// Expected: time-to-first-character on the user's screen drops to less than
// half of the equivalent buffered (non-streaming) callStreaming pulls in some extra concerns — mid-flight disconnects, retry semantics, and timeouts. I worked through those failure modes in Diagnosing Claude API streaming cutoffs if you want to harden your handler before shipping.
You can't tune what you don't measure
Every lever above is genuinely useful, but rolling them in without measurement leaves you in a fog of "feels faster, I think." The three numbers I always track:
- TTFB (time until the first token surfaces)
- Total completion time
- Cache hit ratio on input tokens
These slot into Claude API spans as OpenTelemetry attributes — see Claude API observability with OpenTelemetry for a working setup. Recording before/after numbers per change is what separates real wins from confirmation bias.
Where I'd start tomorrow
Don't try all four levers in one PR. The order I follow:
Switch to Bedrock in a closer region first, and measure TTFB. Add a cache breakpoint to your system prompt next. Once those wins are clearly visible, layer in connection pooling and streaming. Going one at a time keeps each lever's contribution legible — and it's a lot easier to roll a single change back than to untangle four.
Swapping Sonnet for Haiku is a real option, but for me it's the last one to consider. Model substitution changes output quality, and the four levers above buy speed without that tradeoff. Spend the infrastructure budget first.