I was building a small agent to check the store listings of my wallpaper app across every language it ships in. One tool fetched the description for a locale, another pulled recent reviews, a third read the metadata of the wallpapers currently being served. Claude lined them up in a single turn, and the response came back noticeably faster than the serial version I'd had before.
The next morning I opened the logs. One locale had timed out, and at that instant the results for every other locale had been thrown away with it. What I understood that morning was that the speedup and the failures arrive together — you don't get one without the other.
So the first thing I'd like to say is this: parallel tool use is not just a feature that raises your concurrency. It's a feature that multiplies the ways a turn can fail. What follows is the order in which I worked through it: control, partial failure, and observability.
Why Parallelism Pays Off — Sum Versus Max
Picture an agent that checks inventory, fetches a price, and calculates shipping through three separate APIs. Serially, that's 200ms + 180ms + 150ms = 530ms. In parallel, it's max(200, 180, 150) = 200ms. Even after Claude's own generation time, the perceived latency lands close to half.
The official docs describe it plainly: with tool_choice left at auto (the default), Claude may call several tools it judges to be independent within the same turn. Whether it judges them independent depends on your prompt, though. If that judgment tips toward serial, all the async plumbing you've built stays idle.
Where parallelism helps
- Search and comparison flows that hit several external APIs (product lookups, flight comparison, news)
- Batch work with one function over many inputs (fetching several URLs, checking descriptions in several languages)
- Independent evaluations (scoring along multiple axes, querying several models at once)
Where it doesn't, or shouldn't
- Chains where each tool depends on the previous result (Search → Read → Analyze)
- Writes or file operations that might touch the same resource
- Downstream services with tight rate limits, where you'd rather throttle than fan out
There Are Two Axes of Control, Not One
There's the question of whether Claude emits several tool_use blocks in a turn, and the separate question of whether your code executes them concurrently. Blur the two and you'll spend an afternoon asking why the parallel version isn't any faster — I did.
Axis 1: Claude's decision — tool_choice and disable_parallel_tool_use
# Explicitly forbid parallel tool use
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
tools=[
{"name": "get_inventory", "description": "Get inventory", "input_schema": {"type": "object", "properties": {"sku": {"type": "string"}}}},
{"name": "get_price", "description": "Get price", "input_schema": {"type": "object", "properties": {"sku": {"type": "string"}}}},
],
tool_choice={
"type": "auto",
"disable_parallel_tool_use": True # Exactly one tool per turn
},
messages=[{"role": "user", "content": "What's the inventory and price for SKU-A123?"}]
)With disable_parallel_tool_use: True, Claude emits at most one tool_use block per turn. That's what I reach for when ordering matters, or when side-effectful tools need to run one at a time.
The default (False) lets Claude go wide whenever it judges the tools independent, and a line in the prompt such as "fetch whatever you need in parallel before answering" raises that rate visibly. The four tool_choice modes and what each one costs are covered in Using tool_choice to Cut Wasted Inference: Four Modes and Cost Patterns for Production.
Axis 2: Your execution — asyncio.gather and a Semaphore
Even when Claude returns five tool_use blocks, a for loop with an await inside runs them one after another. This is exactly where I was stuck at first.
import anthropic
import asyncio
import httpx
from typing import Any
client = anthropic.AsyncAnthropic()
# Each tool implementation needs to be async
async def get_inventory(sku: str) -> dict:
async with httpx.AsyncClient(timeout=5.0) as http:
r = await http.get(f"https://api.example.com/inventory/{sku}")
r.raise_for_status()
return r.json()
async def get_price(sku: str) -> dict:
async with httpx.AsyncClient(timeout=5.0) as http:
r = await http.get(f"https://api.example.com/price/{sku}")
r.raise_for_status()
return r.json()
async def get_shipping(sku: str, zip_code: str) -> dict:
async with httpx.AsyncClient(timeout=5.0) as http:
r = await http.get("https://api.example.com/shipping", params={"sku": sku, "zip": zip_code})
r.raise_for_status()
return r.json()
TOOL_MAP = {
"get_inventory": get_inventory,
"get_price": get_price,
"get_shipping": get_shipping,
}
# Cap the number of concurrent outbound calls
SEMAPHORE = asyncio.Semaphore(5)
async def run_tool(block: Any) -> dict:
"""Execute one tool_use block and return a tool_result."""
async with SEMAPHORE:
try:
fn = TOOL_MAP[block.name]
result = await fn(**block.input)
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": str(result),
"is_error": False,
}
except Exception as e:
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Error: {type(e).__name__}: {e}",
"is_error": True,
}
async def run_parallel_tools(response) -> list[dict]:
"""Execute every tool_use block in the response concurrently."""
tool_blocks = [b for b in response.content if b.type == "tool_use"]
results = await asyncio.gather(
*[run_tool(b) for b in tool_blocks],
return_exceptions=False # each run_tool already handles its own exceptions
)
return resultsThree things carry this code.
First, gather. Unless the coroutines are handed to asyncio.gather in one call, a row of awaits stays serial.
Second, cap concurrency with a Semaphore. Every external API has a rate limit, and Claude calls more tools at once than you'd expect. Without a cap, one busy turn can knock over the service downstream.
Third, catch exceptions inside run_tool and turn them into is_error: True results. Let one exception escape through gather(return_exceptions=False) and every other result in the batch is lost. The morning I described at the top — one locale timing out, all locales gone — was precisely this.
Designing for Partial Failure — Where Most of the Incidents Live
Five tools fire, three succeed, two fail. What should the agent do? The answer depends on the product, and I keep three policies side by side.
Policy A: Answer only when everything arrived (strict)
For payment amounts, medical data, anything where acting on a partial picture is itself the hazard. One failure fails the turn, and the turn is retried.
class PartialFailureError(Exception):
def __init__(self, failed_tools: list[str]):
super().__init__(f"failed: {failed_tools}")
self.failed_tools = failed_tools
async def strict_multi_tool(messages, tools):
"""Strict mode — every tool must succeed."""
response = await client.messages.create(
model="claude-sonnet-4-6",
max_tokens=2048,
tools=tools,
messages=messages,
)
if response.stop_reason != "tool_use":
return response
results = await run_parallel_tools(response)
if any(r["is_error"] for r in results):
raise PartialFailureError(
failed_tools=[r["tool_use_id"] for r in results if r["is_error"]]
)
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": results})
return await strict_multi_tool(messages, tools)Policy B: Hand the errors back and let Claude decide (lenient)
For search and information gathering. Put is_error: True and the error text into the failed tool_result, and Claude will usually say something like "I couldn't reach that source, so here's what I found elsewhere" — or ask the user for the missing piece.
async def lenient_multi_tool(messages, tools):
"""Lenient mode — tolerate partial failure and let Claude reason about it."""
response = await client.messages.create(
model="claude-sonnet-4-6",
max_tokens=2048,
tools=tools,
messages=messages,
)
if response.stop_reason != "tool_use":
return response
results = await run_parallel_tools(response)
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": results})
return await lenient_multi_tool(messages, tools)The store-listing checker moved to this policy. On a morning when one locale is unreachable it now tells me that English and Korean were verified and German could not be fetched, and I keep the results I did get instead of losing all of them.
Policy C: Retry only the tools that failed (hybrid)
For transient trouble — a network blip that will very likely succeed on the second try.
async def run_tool_with_retry(block, max_retries=3):
"""Retry a single tool with exponential backoff."""
for attempt in range(max_retries):
async with SEMAPHORE:
try:
fn = TOOL_MAP[block.name]
result = await fn(**block.input)
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": str(result),
"is_error": False,
}
except (httpx.TimeoutException, httpx.ConnectError) as e:
if attempt < max_retries - 1:
await asyncio.sleep(2 ** attempt)
continue
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Error after {max_retries} retries: {e}",
"is_error": True,
}
except Exception as e:
# Non-retryable — fail fast
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Non-retryable error: {e}",
"is_error": True,
}Be deliberate about which exceptions you retry. A 401 won't heal on the second attempt, and a 400 is a bug in your own arguments — fix it before you retry it.
Idempotency — Can the Tool Survive Running Twice?
The failure I fear most with parallel tools is the same tool running twice with the same arguments — through a wobble in Claude's plan, a retry on my side, or a resubmitted turn. For reads it's harmless. For writes it can be unrecoverable.
async def create_order(user_id: str, sku: str, quantity: int, idempotency_key: str) -> dict:
"""Create an order with deduplication via idempotency key."""
async with httpx.AsyncClient(timeout=10.0) as http:
r = await http.post(
"https://api.example.com/orders",
json={"user_id": user_id, "sku": sku, "quantity": quantity},
headers={"Idempotency-Key": idempotency_key}
)
r.raise_for_status()
return r.json()Claude assigns every tool_use block a unique id, which makes tool_use_id the simplest and safest idempotency key you'll find. Asking Claude to generate the key as a tool argument is riskier — a retry may well produce a different value.
async def run_tool_idempotent(block) -> dict:
"""Inject tool_use_id as the idempotency key."""
if block.name == "create_order":
input_with_key = {**block.input, "idempotency_key": block.id}
result = await create_order(**input_with_key)
else:
fn = TOOL_MAP[block.name]
result = await fn(**block.input)
return {"type": "tool_result", "tool_use_id": block.id, "content": str(result)}The one line I've drawn for myself here: reads may run in parallel; writes run one at a time, and each one carries a key. Agents that include a write tool get disable_parallel_tool_use: True. Only read-only agents are allowed to go wide.
Streaming and Parallel Tools
With streaming (SSE), a tool's arguments arrive as input_json_delta fragments, and the JSON for that block is only complete at its content_block_stop. Blocks arrive in index order, so ordering isn't the issue — the rule is simply that you must never execute a block before its input is finalized.
import anthropic
import asyncio
client = anthropic.AsyncAnthropic()
async def stream_and_execute_parallel(messages, tools):
async with client.messages.stream(
model="claude-sonnet-4-6",
max_tokens=2048,
tools=tools,
messages=messages,
) as stream:
async for text in stream.text_stream:
print(text, end="", flush=True) # text goes to the UI as it arrives
final = await stream.get_final_message()
tool_blocks = [b for b in final.content if b.type == "tool_use"]
if not tool_blocks:
return final
# Once the stream finishes, run the tools concurrently
results = await asyncio.gather(*[run_tool(b) for b in tool_blocks])
messages.append({"role": "assistant", "content": final.content})
messages.append({"role": "user", "content": results})
return await stream_and_execute_parallel(messages, tools)You can start executing each block at its content_block_stop instead of waiting for the whole turn. The difference is usually a few hundred milliseconds, and in my experience waiting for get_final_message() produced far fewer bugs. Stream the text early, gather the tools once they're final — that split has been enough for me.
Observability — Watch the Parallel Degree as a Distribution
If parallel tools are in production, per-tool latency, success rate, and parallel degree need to be metrics. Without them you can't tell which tool is slow or which one is failing.
import time
from prometheus_client import Histogram, Counter
TOOL_LATENCY = Histogram(
"claude_tool_duration_seconds",
"Tool execution duration",
["tool_name", "status"]
)
TOOL_CALLS = Counter(
"claude_tool_calls_total",
"Total tool calls",
["tool_name", "status"]
)
PARALLEL_DEGREE = Histogram(
"claude_parallel_tool_degree",
"Number of tools called in parallel per turn",
buckets=[1, 2, 3, 5, 8, 13, 21]
)
async def run_tool_observed(block) -> dict:
start = time.monotonic()
status = "success"
try:
async with SEMAPHORE:
fn = TOOL_MAP[block.name]
result = await fn(**block.input)
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": str(result),
"is_error": False,
}
except Exception as e:
status = "error"
return {
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Error: {e}",
"is_error": True,
}
finally:
duration = time.monotonic() - start
TOOL_LATENCY.labels(tool_name=block.name, status=status).observe(duration)
TOOL_CALLS.labels(tool_name=block.name, status=status).inc()
async def run_parallel_tools_observed(response):
tool_blocks = [b for b in response.content if b.type == "tool_use"]
PARALLEL_DEGREE.observe(len(tool_blocks))
return await asyncio.gather(*[run_tool_observed(b) for b in tool_blocks])Keeping the parallel degree as a distribution shows how often Claude actually goes wide versus emitting a single call. Change one line of the prompt, watch the distribution move — that's been the most reliable way to tune it.
You don't need Prometheus to start. If you keep conversation logs, the degree can be counted from the saved messages:
from collections import Counter
def parallel_degree_histogram(messages: list[dict]) -> Counter:
"""Count tool_use blocks per assistant turn from saved messages."""
hist: Counter = Counter()
for m in messages:
if m["role"] != "assistant" or not isinstance(m["content"], list):
continue
n = sum(1 for b in m["content"] if getattr(b, "type", None) == "tool_use" or (isinstance(b, dict) and b.get("type") == "tool_use"))
if n:
hist[n] += 1
return hist
# e.g. Counter({1: 41, 2: 12, 3: 7, 5: 1}) — a mean near 1 means plenty of headroomA Note on Extended Thinking
When I measured parallel degree with extended thinking enabled, my agents tended to emit somewhat fewer tool_use blocks per turn than with thinking off. They lean toward looking at one result before deciding the next call.
For work that needs careful reasoning, that's the behavior I want. For "fetch a handful of independent things and compose an answer," it means the parallel speedup doesn't show up. I may be missing a subtlety here, but measuring the degree separately with thinking on and off has saved me from misdiagnosing "parallelism isn't working" more than once.
Choosing the Semaphore Value
For asyncio.Semaphore(n), this is the process I use.
Start at n = 3; almost no downstream API is disturbed by that. Run realistic traffic for half an hour and watch two numbers: your own p95 latency and the downstream error rate. If both stay healthy, raise n by one and repeat.
Stop when any of three things happens: the downstream service starts returning 429 or 503 at a rate you can't ignore, your p95 flattens (more concurrency no longer helps, so you've hit the bottleneck), or memory and CPU on your side get uncomfortable. For me n settles between 4 and 8 nearly every time, and going past 10 has always meant I was fighting the downstream service rather than helping it.
Since a single turn rarely exceeds 2–5 parallel tools, the Semaphore matters most when several agent sessions are running at once.
When I Turn Parallelism Off on Purpose
- Write-heavy flows on the same record: serializing avoids write-write conflicts without building locks
- Dependency-heavy reasoning: if each result feeds the next call's arguments, parallelism was never available — telling Claude so up front saves tokens
- Ordered audit trails: parallel execution makes ordering guarantees harder
- A downstream limit of one request per second: parallel calls would only serialize again on the wire
The cost of turning it off is extra turns — three single-tool turns instead of one three-tool turn, with Claude's generation time stacking up accordingly. In the cases above, I'd rather pay that.
Testing That It's Actually Parallel
Parallel tools fail on timing, not logic, so they need a slightly different kind of test. What I rely on is a fake tool with a configurable delay: if the code is truly parallel, total time should sit near the slowest tool's delay rather than the sum.
import asyncio
import time
from types import SimpleNamespace
def make_fake_tool(delay: float, result: dict):
async def fake(**kwargs):
await asyncio.sleep(delay)
return result
return fake
async def test_tools_actually_run_in_parallel():
TOOL_MAP["get_inventory"] = make_fake_tool(0.2, {"stock": 10})
TOOL_MAP["get_price"] = make_fake_tool(0.2, {"price": 99})
response = SimpleNamespace(content=[
SimpleNamespace(type="tool_use", id="toolu_01", name="get_inventory", input={"sku": "A"}),
SimpleNamespace(type="tool_use", id="toolu_02", name="get_price", input={"sku": "A"}),
])
start = time.monotonic()
results = await run_parallel_tools(response)
elapsed = time.monotonic() - start
assert len(results) == 2
assert elapsed < 0.3, f"this ran serially: {elapsed:.2f}s" # ~0.2s when parallel
asyncio.run(test_tools_actually_run_in_parallel())Layer random failure injection on top — make one fake tool raise — and you can verify that strict, lenient, and hybrid each behave the way you intended.
Pitfalls I've Walked Into
I thought it was parallel when it was serial. An await inside a for loop. I didn't see it until I profiled.
I relied on ordering to match results. The Claude API doesn't require tool_result order to match tool_use order, but it does require every tool_use_id to be paired. gather returns in input order, yet matching on tool_use_id in logs and UI is the safer design. The recovery side of a missing tool_result is its own topic, written up in Why tool_result could not be submitted Keeps Coming Back, and How to Build a Recovery Handler That Actually Holds.
I ran without a Semaphore. Claude called eight tools at once, the downstream rate limit tripped, and the whole agent stopped.
I generated idempotency keys on the client, every call. uuid.uuid4() makes a new key each time, so the moment of a retry is the moment an order goes through twice. Use tool_use_id, or derive the key deterministically from what makes the request unique.
Nudging Parallelism Through the Prompt
How wide Claude goes depends on the words in your system prompt.
What worked for me was explicit language: "Fetch the information you need simultaneously whenever possible. Call tools that don't depend on each other in parallel." A vague "use tools efficiently" barely moved anything. In my own records, that one line moved the average parallel degree from roughly 1.3 to roughly 2.8 — the value will differ with your tool set, so please check it against your own distribution.
Before You Ship
asyncio.gatherreally runs the tools concurrently (fake-tool test or profiler)- The Semaphore cap sits below the tightest downstream limit
- Every tool function catches its exceptions and returns
is_error: True - Every write tool takes an idempotency key, and
tool_use_idis injected into it - The partial-failure policy (strict, lenient, hybrid) matches the product
- Parallel degree, latency, and success rate are being recorded
- The system prompt asks for parallel fetching in plain words
Where to Start
Count the tool_use blocks per assistant turn in your current agent's logs. If the mean is close to 1.0, there is a lot of parallelism left on the table. Add one line to the prompt, then layer in the Semaphore, idempotency, and observability in that order, and you'll have a parallel agent that holds up in production.
Reads in parallel, writes one at a time — that's the one line I don't cross, even on a night when I'm in a hurry.