Point Claude at a set of tools, let it fire three or four of them off in a single turn, and the response feels twice as fast. That's the easy part, and it's where most articles stop. The interesting work starts the moment you ship it: what do you do when one of three parallel tools times out? What happens if two DB writes hit the same record? How do you keep the rest of the turn alive when one call dies?
Parallel tool use is one of the biggest UX wins in Claude agents, but every time you raise the concurrency, the combinatorial surface of failure modes grows with it. The official docs cover the happy path; production needs more. What follows is the set of patterns I've landed on after running parallel tools in real products, organized around three concerns: control, error design, and observability.
Why Parallel Tool Use Pays Off — Latency in Numbers
The concrete latency impact of parallel tool calls is dramatic. Picture an agent that checks inventory, fetches price, and calculates shipping through three separate APIs. Serially, you're looking at 200ms + 180ms + 150ms = 530ms. In parallel, it's max(200, 180, 150) = 200ms. Even after Claude's own generation overhead, user-perceived latency is often cut in half.
Anthropic has been explicit that Claude 3.5 Sonnet and later lean into parallel execution by default when tools are independent. Whether the model recognizes independence depends heavily on your prompt, though — a misleading prompt can push it back into serial mode and kill your speedup without warning.
When Parallelism Pays Off
- Search and comparison flows that hit multiple external APIs (product lookups, flight comparison, news aggregation)
- Batch operations with the same function over different inputs (scraping several URLs, summarizing multiple files)
- Independent evaluations (scoring along multiple axes, querying multiple models)
When Parallelism Backfires
- Chained flows where each tool depends on the previous result (Search → Read → Analyze)
- Writes or file operations that might touch the same resource
- Endpoints with tight rate limits where concurrency must be throttled
Two Axes of Control for Parallel Execution
Here's the distinction that most developers blur: there are two separate axes of control for parallel tool use — the model's decision to emit parallel tool_use blocks, and your application's decision to execute them concurrently. Getting them confused leads to "why isn't my parallelism actually parallel" debugging sessions.
Axis 1: Controlling Claude's Decision — tool_choice and disable_parallel_tool_use
Whether Claude emits multiple tool_use blocks in a single turn is governed by tool_choice.
# Explicitly forbid parallel tool use
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1024,
tools=[
{"name": "get_inventory", "description": "Get inventory", "input_schema": {"type": "object", "properties": {"sku": {"type": "string"}}}},
{"name": "get_price", "description": "Get price", "input_schema": {"type": "object", "properties": {"sku": {"type": "string"}}}},
],
tool_choice={
"type": "auto",
"disable_parallel_tool_use": True # Exactly one tool per turn
},
messages=[{"role": "user", "content": "What's the inventory and price for SKU-A123?"}]
)With disable_parallel_tool_use: True, Claude emits at most one tool_use block per turn. That's what you want when ordering matters or when you need to serialize side-effectful tools.
The default (False) lets Claude emit multiple tool_use blocks whenever it judges the tools independent. You can nudge the rate up noticeably by writing prompts like "fetch whatever information you need in parallel before answering" — the model is genuinely responsive to the wording.
Axis 2: Controlling Your Application — asyncio.gather and Semaphore
Here's where a lot of developers trip: even if Claude returns five tool_use blocks, your code still has to execute them concurrently. Sequential awaits in a loop won't do it.
import anthropic
import asyncio
import httpx
from typing import Any
client = anthropic.AsyncAnthropic()
# Each tool implementation must be async
async def get_inventory(sku: str) -> dict:
async with httpx.AsyncClient(timeout=5.0) as http:
r = await http.get(f"https://api.example.com/inventory/{sku}")
r.raise_for_status()
return r.json()
async def get_price(sku: str) -> dict:
async with httpx.AsyncClient(timeout=5.0) as http:
r = await http.get(f"https://api.example.com/price/{sku}")
r.raise_for_status()
return r.json()
async def get_shipping(sku: str, zip_code: str) -> dict:
async with httpx.AsyncClient(timeout=5.0) as http:
r = await http.get(f"https://api.example.com/shipping", params={"sku": sku, "zip": zip_code})
r.raise_for_status()
return r.json()
TOOL_MAP = {
"get_inventory": get_inventory,
"get_price": get_price,
"get_shipping": get_shipping,
}
# Cap the number of concurrent outbound calls
SEMAPHORE = asyncio.Semaphore(5)
async def run_tool(block: Any) -> dict:
"""Execute one tool_use block and return a tool_result."""
async with SEMAPHORE:
try:
fn = TOOL_MAP[block.name]
result = await fn(**block.input)
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": str(result),
"is_error": False,
}
except Exception as e:
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Error: {type(e).__name__}: {e}",
"is_error": True,
}
async def run_parallel_tools(response) -> list[dict]:
"""Execute every tool_use block in the response concurrently."""
tool_blocks = [b for b in response.content if b.type == "tool_use"]
results = await asyncio.gather(
*[run_tool(b) for b in tool_blocks],
return_exceptions=False # each run_tool already handles its own exceptions
)
return resultsThree things matter here.
First, use asyncio.gather. A for loop over tool_use blocks with await inside will serialize your execution no matter how many blocks Claude returned. Gather the coroutines into a single call.
Second, cap concurrency with a Semaphore. Every external API has rate limits, and Claude occasionally returns more parallel calls than you expect — ten is not unusual. Without a cap, one noisy turn can take the downstream API down. asyncio.Semaphore(5) is a reasonable starting point and you'll tune it against the tightest downstream limit.
Third, catch exceptions inside each tool function and convert them into is_error: True tool results. If you leave them to bubble out of asyncio.gather(return_exceptions=False), a single failing tool takes the whole batch down. You can use return_exceptions=True instead, but converting to error tool_results is cleaner because it feeds Claude enough context to reason about the failure and continue the turn with partial data.
Designing for Partial Failure — the Thing Nobody Warns You About
The production issue you'll hit most often with parallel tool use is partial failure. Five tools fire, three succeed, two fail — now what?
Pattern A: All-or-Nothing (Strict)
For finance, medical, or anything where acting on partial data is dangerous. If any tool fails, the whole turn fails and retries.
async def strict_multi_tool(messages, tools):
"""Strict mode — every tool must succeed."""
response = await client.messages.create(
model="claude-sonnet-4-5",
max_tokens=2048,
tools=tools,
messages=messages,
)
if response.stop_reason != "tool_use":
return response
results = await run_parallel_tools(response)
if any(r["is_error"] for r in results):
raise PartialFailureError(
failed_tools=[r["tool_use_id"] for r in results if r["is_error"]]
)
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": results})
return await strict_multi_tool(messages, tools)Pattern B: Pass Errors Back and Let Claude Decide (Lenient)
For search and information-gathering flows, letting Claude see the errors is surprisingly effective. The model picks up on is_error: True results and tends to answer with whatever information it did get — or asks the user for missing pieces.
async def lenient_multi_tool(messages, tools):
"""Lenient mode — tolerate partial failure and let Claude reason about it."""
response = await client.messages.create(
model="claude-sonnet-4-5",
max_tokens=2048,
tools=tools,
messages=messages,
)
if response.stop_reason != "tool_use":
return response
results = await run_parallel_tools(response)
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": results})
return await lenient_multi_tool(messages, tools)In my experience, being deliberate about which of these two you use per product is what separates flaky agents from robust ones. For payment amounts, stick with strict — missing data is never acceptable. For research and recommendation flows, lenient is better UX: Claude will often say "I couldn't reach the shipping service, but here's what I found on the others" instead of a hard failure.
Pattern C: Retry Individual Failures (Hybrid)
Use this when you've got transient errors like network blips that retry well.
async def run_tool_with_retry(block, max_retries=3):
"""Retry a single tool with exponential backoff."""
for attempt in range(max_retries):
async with SEMAPHORE:
try:
fn = TOOL_MAP[block.name]
result = await fn(**block.input)
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": str(result),
"is_error": False,
}
except (httpx.TimeoutException, httpx.ConnectError) as e:
if attempt < max_retries - 1:
await asyncio.sleep(2 ** attempt)
continue
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Error after {max_retries} retries: {e}",
"is_error": True,
}
except Exception as e:
# Non-retryable — fail fast
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Non-retryable error: {e}",
"is_error": True,
}Be precise about which exceptions you retry. A 401 won't heal with retries; a 400 is a parameter bug and retrying it just burns quota.
Idempotency — Can Your Tool Survive Running Twice?
The nastiest failure mode in parallel tool use is Claude calling the same tool with the same arguments twice — whether through its own planning, a retry, or a resubmitted turn. Reads are safe. Writes can be catastrophic.
Every Write Tool Needs an Idempotency Key
async def create_order(user_id: str, sku: str, quantity: int, idempotency_key: str) -> dict:
"""Create an order with deduplication via idempotency key."""
async with httpx.AsyncClient(timeout=10.0) as http:
r = await http.post(
"https://api.example.com/orders",
json={"user_id": user_id, "sku": sku, "quantity": quantity},
headers={"Idempotency-Key": idempotency_key}
)
r.raise_for_status()
return r.json()The cleanest approach is to let the application inject the idempotency key automatically from the tool_use_id. Claude guarantees each tool_use block has a unique id, which makes it a perfect idempotency key — you get retry safety for free.
async def run_tool_idempotent(block) -> dict:
"""Inject tool_use_id as the idempotency key."""
if block.name == "create_order":
input_with_key = {**block.input, "idempotency_key": block.id}
result = await create_order(**input_with_key)
else:
fn = TOOL_MAP[block.name]
result = await fn(**block.input)
return {"type": "tool_result", "tool_use_id": block.id, "content": str(result)}For a deeper dive into idempotency across your whole agent (not just parallel tools), see Designing Idempotency in Claude Agent SDK.
Streaming and Parallel Tool Use
With SSE streaming, tool_use blocks finish in arbitrary order within the stream. Here's the implementation that avoids the usual bugs.
import anthropic
import asyncio
client = anthropic.AsyncAnthropic()
async def stream_and_execute_parallel(messages, tools):
tool_blocks = []
assistant_content = []
async with client.messages.stream(
model="claude-sonnet-4-5",
max_tokens=2048,
tools=tools,
messages=messages,
) as stream:
async for event in stream:
# Handle events (text deltas, tool_use deltas, etc.)
pass
final = await stream.get_final_message()
tool_blocks = [b for b in final.content if b.type == "tool_use"]
assistant_content = final.content
# Once the stream finishes, run the tools concurrently
if tool_blocks:
results = await asyncio.gather(*[run_tool(b) for b in tool_blocks])
messages.append({"role": "assistant", "content": assistant_content})
messages.append({"role": "user", "content": results})
return await stream_and_execute_parallel(messages, tools)
return finalIt's tempting to kick off tool execution the moment a tool_use block completes mid-stream, but the Anthropic SDK finalizes tool_use blocks at stream.get_final_message(). Waiting for the stream to complete is the least buggy implementation.
For UX, stream the text blocks to the user immediately while tool execution runs in the background. That way the user sees Claude "thinking out loud" while the parallel calls finish.
Observability for Parallel Execution
If you're running parallel tools in production, instrument them. Without visibility into per-tool latency, success rate, and parallelism degree, you can't tell what's slow or what's failing.
import time
from prometheus_client import Histogram, Counter
TOOL_LATENCY = Histogram(
"claude_tool_duration_seconds",
"Tool execution duration",
["tool_name", "status"]
)
TOOL_CALLS = Counter(
"claude_tool_calls_total",
"Total tool calls",
["tool_name", "status"]
)
PARALLEL_DEGREE = Histogram(
"claude_parallel_tool_degree",
"Number of tools called in parallel per turn",
buckets=[1, 2, 3, 5, 8, 13, 21]
)
async def run_tool_observed(block) -> dict:
start = time.monotonic()
status = "success"
try:
async with SEMAPHORE:
fn = TOOL_MAP[block.name]
result = await fn(**block.input)
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": str(result),
"is_error": False,
}
except Exception as e:
status = "error"
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Error: {e}",
"is_error": True,
}
finally:
duration = time.monotonic() - start
TOOL_LATENCY.labels(tool_name=block.name, status=status).observe(duration)
TOOL_CALLS.labels(tool_name=block.name, status=status).inc()
async def run_parallel_tools_observed(response):
tool_blocks = [b for b in response.content if b.type == "tool_use"]
PARALLEL_DEGREE.observe(len(tool_blocks))
return await asyncio.gather(*[run_tool_observed(b) for b in tool_blocks])Tracking the distribution of parallel degree is particularly useful — it shows you how often Claude is actually going wide versus emitting single tool calls. Change a prompt, watch the distribution shift, and you can tune empirically.
For OpenTelemetry-based observability, see OpenTelemetry for Claude API Observability.
Five Pitfalls I've Hit in Production
Pitfall 1: Parallelism That's Actually Serial
Writing for block in response.content: result = await run_tool(block) looks parallel because of await, but it's serial. Always gather: asyncio.gather(*[run_tool(b) for b in blocks]).
Pitfall 2: Mismatched tool_result Ordering
asyncio.gather returns results in input order, which usually keeps you safe. But if you ever log or display results, match on tool_use_id explicitly. The Claude API doesn't require tool_result order to match tool_use order — it requires each tool_use_id to be paired with its tool_result.
Pitfall 3: Forgetting the Semaphore
My first production incident with parallel tools: Claude emitted eight parallel calls, my code fired them all at once, the downstream API's rate limit tripped, and the whole agent went down. Cap concurrency. Always.
Pitfall 4: Letting Exceptions Bubble Through gather
A single raised exception in gather(return_exceptions=False) discards the other results. Catch inside each tool function and return is_error: True instead — that keeps the turn alive and feeds Claude useful context.
Pitfall 5: Client-Side Idempotency Keys
If you generate uuid.uuid4() fresh every call, retries produce new keys and you lose deduplication. Use tool_use_id or a deterministic function of the request (e.g., (user_id, sku, time_bucket)).
Nudging Parallelism Through Prompting
How aggressively Claude parallelizes depends on your system prompt. From what I've measured:
Prompts that increase parallelism:
When answering the user's question, fetch all required information in parallel whenever possible. Tools that don't depend on each other should be called simultaneously to minimize latency.
Prompts that don't move the needle much:
Use tools efficiently.
Explicit words like "in parallel" and "simultaneously" moved my average parallel degree from ~1.3 to ~2.8. Claude is unusually responsive to direct language here, so be specific about what you want.
Production Checklist
Before shipping parallel tool use, verify:
- Your
asyncio.gatheractually runs tools concurrently (profile it) - Your Semaphore caps concurrency below the tightest downstream limit
- Every tool function catches exceptions and returns
is_error: True - Every write tool accepts an idempotency key, and you're injecting
tool_use_idinto it - Your partial-failure policy (strict, lenient, or hybrid) fits the product
- You have per-tool latency, success rate, and parallel-degree metrics
- Your system prompt asks for parallel fetching in explicit language
A Real-World Case: Cutting Agent Latency from 4.2s to 1.6s
Let me walk through a concrete refactor from one of my production agents so the patterns above feel less abstract. The agent was a customer-facing product assistant with four tools: a product search, a stock lookup, a review aggregator, and a recommendation generator. Each tool hit a different backend, and the original implementation called them serially in a loop as the user's question required them.
Measured p50 latency for a typical query: 4,200ms. That's slow enough that users perceive noticeable hang time between "send" and the first token of response. The breakdown was roughly:
- First Claude turn (generate tool calls): 600ms
- Product search API: 800ms
- Stock lookup API: 600ms
- Review aggregator: 1,400ms (the slowest, with DB-heavy queries)
- Recommendation API: 400ms
- Second Claude turn (write response): 400ms
Total: 600 + 800 + 600 + 1,400 + 400 + 400 = 4,200ms.
After the refactor to use parallel tool use and asyncio.gather:
- First Claude turn: 600ms
- All four tools in parallel, limited by the slowest (review aggregator): 1,400ms
- Second Claude turn: 400ms
Parallel delivery put us at roughly 2,400ms. Still slower than I wanted. The next round of optimization came from caching: the product search result is cache-friendly for 60 seconds, which dropped it from 800ms to 20ms most of the time. The stock lookup was inherently cache-hostile but tiny. Final numbers landed around 1,600ms for cache hits and 2,400ms for cache misses. That's the kind of compound improvement that only becomes visible once tools run in parallel — serial caching wouldn't have moved the needle the same way because the review aggregator dominated total time.
The point of the story isn't the specific numbers; it's that your slowest tool is now your latency floor, which changes how you think about optimization. Before parallelism, you optimize whichever tool takes the most total-duration share. After parallelism, you optimize whichever tool is the tail — everything else is free.
Parallel Tool Use + Extended Thinking: a Subtle Interaction
If you're using Claude's extended thinking (for example, claude-sonnet-4-5 with extended thinking enabled), there's an interaction with parallel tool use that's easy to miss. When extended thinking is on, Claude tends to produce fewer parallel tool calls per turn — it prefers to think carefully and call tools one at a time, using each result to refine its plan.
This is usually the right behavior for complex reasoning, but it means you don't get the same parallel-execution speedup when thinking is enabled. I've seen teams benchmark a parallel tool pipeline with thinking enabled, find it disappointing, and disable parallel tool use entirely — when the real issue was that thinking was serializing the calls.
Rule of thumb: if the task requires deep reasoning and has a clear chain of dependent tool calls, extended thinking wins. If the task is "fetch a bunch of independent data and compose an answer," disable thinking and enable parallel tool use explicitly in the prompt.
Tuning the Semaphore: a Practical Process
How do you pick the right number for asyncio.Semaphore(n)? Here's the empirical process I use.
Start with n = 3. This is conservative enough that most downstream APIs won't be disturbed.
Run a load test: simulate realistic user traffic for 30 minutes and watch two metrics — your own p95 latency and the downstream API's error rate. If both are healthy, raise n by 1 and repeat.
Stop raising n when any of these happens:
- The downstream API starts returning 429s or 503s with nontrivial frequency (more than 0.1% of calls)
- Your own p95 latency flattens (increasing concurrency no longer helps, which means you've saturated the bottleneck)
- Memory or CPU pressure on your service climbs to uncomfortable levels
In my experience, n between 4 and 8 covers almost every real product. n > 10 usually means you're fighting the downstream service, not helping it. The interesting thing is that tool parallelism within a single Claude turn is usually small (2-5 tools), so the Semaphore matters most during concurrent agent sessions, not within a single turn.
When to Deliberately Disable Parallel Tool Use
Not every agent benefits from parallelism. Cases where I set disable_parallel_tool_use: True:
- Stateful write-heavy workflows: When every tool call writes to the same record, serializing prevents write-write conflicts without requiring sophisticated locking.
- Dependency-heavy reasoning: When each tool's result informs the next tool's parameters, parallel execution is impossible anyway, and telling Claude so up front reduces wasted tokens.
- Strict audit trails: Regulated domains where every tool call needs to appear in a single, ordered log. Parallelism complicates ordering guarantees.
- Tight rate limits: If your downstream API allows only one request per second from your account, there's no point producing parallel tool calls that will immediately serialize on the network.
The cost of disabling parallelism is additional turns — instead of one Claude turn with three tool calls, you get three turns with one tool call each, which adds ~1.5-2s of Claude generation overhead. That's usually a fair trade for the scenarios above.
Handling Very Wide Parallelism (10+ Concurrent Tools)
Claude occasionally produces surprisingly wide tool_use responses — I've seen up to 12 parallel calls when asked to compare attributes across many products at once. When this happens, a few extra patterns help.
Batch within batch: if you can group the tool_use blocks by target service, you can deduplicate and batch. Ten calls to the same product-details API for ten different SKUs can become one batched call if the backend supports it. Write a small coalescing layer that inspects incoming tool_use blocks before executing.
Priority-ordered execution: when you have a hard ceiling on concurrency and Claude produces more tool calls than that, decide which go first. User-visible data (names, prices) should finish before background enrichment (recommendations). asyncio.gather doesn't give you priority, but you can split into two gathers — one for high-priority tools, then one for the rest.
Streaming partial results: if the user is waiting on the response, you can stream back partial tool results as they complete. This is more invasive (you need a WebSocket or SSE channel to the client) but it substantially improves perceived latency for wide parallel responses.
Testing Parallel Tool Use
Testing parallel tools is more subtle than testing serial ones because failure modes emerge from timing, not just logic. A few patterns I've found useful.
Deterministic test doubles with controllable delay: replace each tool with a fake that sleeps a configurable duration and returns a fixed result. This lets you unit-test the concurrency behavior directly — if your code is supposedly parallel, total test duration should match the slowest tool's delay, not the sum.
async def make_fake_tool(name, delay, result):
async def fake(**kwargs):
await asyncio.sleep(delay)
return result
return fake
# In a test
fake_inventory = await make_fake_tool("inventory", 0.2, {"stock": 10})
fake_price = await make_fake_tool("price", 0.2, {"price": 99})
# Measure total — it should be ~0.2s, not ~0.4sChaos-style failure injection: randomly make one tool time out or raise. The test then asserts your partial-failure policy is working: strict mode raises, lenient mode returns a usable result, hybrid retries appropriately.
Contract tests against real downstream APIs: once per day, run a small contract test suite against staging endpoints to catch cases where the downstream service changed its rate-limit, timeout, or response shape. This is unglamorous but catches the nastiest production bugs early.
Migrating an Existing Serial Agent to Parallel
If you already have a working agent that runs tools serially, here's the migration path I recommend.
First, measure current latency per turn and per tool. You need a baseline to verify the migration actually helps.
Second, make each tool function async. If any tool is currently synchronous, wrap it with loop.run_in_executor for now — you can convert it to a proper async implementation later. What you cannot do is mix sync and async tool execution in asyncio.gather, because a blocking call inside an async context will serialize everything.
Third, add the Semaphore before you add parallelism. Run the existing serial code inside a Semaphore that's set to 1 (equivalent to no change). Verify behavior is unchanged. Then raise it gradually.
Fourth, switch from the serial loop to asyncio.gather. Do this in a feature flag so you can roll back instantly. Deploy to 1% of traffic, watch metrics, and ramp up.
Fifth, update the system prompt to explicitly request parallel execution. This is the step that actually changes Claude's behavior — without it, the parallel infrastructure is in place but Claude will still emit serial tool calls.
I've found that teams who skip the gradual rollout often discover that one of their tools wasn't safe for concurrent execution (usually a shared connection pool or a global cache that didn't have the locking it needed). The feature flag catches these issues before they affect most of your users.
Where to Start
Measure the average number of tool_use blocks per Claude turn in your current agent. If it's close to 1.0, you have significant parallelism headroom. Add one line to your system prompt asking for parallel fetching where possible, and you'll feel the latency change immediately. From there, layer in Semaphores, idempotency, and observability, and you'll have an agent that's both fast and production-robust.