A Long Non-Streaming Response Was Billed Twice Past the 10-Minute Wall: Redesigning the SDK's Default Timeout and Retries

Fix 1: switch long outputs to streaming

The most effective move is to never trigger the read timeout in the first place. With streaming, the server sends generation progress as a steady flow of small events. The connection always has data moving through it, so it never hits the 10-minute read-timeout wall.

from anthropic import Anthropic
 
client = Anthropic()
 
def summarize_streaming(system_prompt: str, user_text: str) -> str:
    """Receive long output over a stream to avoid the read timeout."""
    chunks = []
    with client.messages.stream(
        model="claude-haiku-4-5",
        max_tokens=16000,
        system=system_prompt,
        messages=[{"role": "user", "content": user_text}],
    ) as stream:
        for text in stream.text_stream:
            chunks.append(text)
        final = stream.get_final_message()
 
    # Always check stop_reason for truncation, too
    if final.stop_reason == "max_tokens":
        raise RuntimeError("Output was cut off at max_tokens; continuation needed")
 
    return "".join(chunks)

Streaming has a second benefit: if the connection drops midway, the text collected so far is still in chunks, so you do not throw away the whole thing. I made every task with max_tokens above 8,000 use this shape by default. The rule is simple — if the length could plausibly cross ten minutes, stream it.

Fix 2: if you stay non-streaming, set timeout and max_retries explicitly

Some places still want to stay non-streaming, like a short post-processing pass. For those, stop relying on the defaults and override them for that specific call. with_options lets you swap settings for a single invocation.

from anthropic import Anthropic, APITimeoutError, APIConnectionError
 
client = Anthropic()
 
def create_long_safe(system_prompt: str, user_text: str, job_id: str, ledger) -> str:
    """Run a long non-streaming request without risking a double charge."""
 
    # If there is already a record of this job, refuse a blind resend
    if ledger.is_inflight_or_done(job_id):
        raise RuntimeError(f"job {job_id} already ran / is running; needs a human check")
 
    ledger.mark_inflight(job_id)
 
    # This task is long: widen the timeout, turn OFF automatic retries
    scoped = client.with_options(timeout=1800.0, max_retries=0)
 
    try:
        msg = scoped.messages.create(
            model="claude-haiku-4-5",
            max_tokens=16000,
            system=system_prompt,
            messages=[{"role": "user", "content": user_text}],
        )
    except APITimeoutError:
        # A timeout is "undetermined". Never auto-resend it myself
        ledger.mark_uncertain(job_id)
        raise
    except APIConnectionError:
        # A connect failure most likely never processed; safe to resend later
        ledger.mark_failed(job_id)
        raise
 
    ledger.mark_done(job_id, input_tokens=msg.usage.input_tokens,
                     output_tokens=msg.usage.output_tokens)
    return msg.content[0].text

max_retries=0 is the crux. It turns off the SDK's automatic retries and takes the decision to resend back into my own hands. An APITimeoutError is only recorded as "undetermined" — it is never thrown straight back at the API, because while a timeout is unresolved the server may still be generating, and a blind resend walks right into the double charge. An APIConnectionError, by contrast, most likely never reached the server, so it can be resent safely after a ledger check. Same word, "failure" — very different handling.

Fix 3: let a local ledger answer "is a resend allowed?"

The last piece is holding enough state that, even unattended, the system can answer whether a job may be sent again. With no idempotency key on the Messages API, that responsibility is ours. I keep one light JSONL ledger that records the life of each request.

import json
import time
from pathlib import Path
 
class JobLedger:
    """A lightweight ledger tracking each request's life. The last line of defense."""
 
    def __init__(self, path: str):
        self.path = Path(path)
        self._state = {}
        if self.path.exists():
            for line in self.path.read_text().splitlines():
                rec = json.loads(line)
                self._state[rec["job_id"]] = rec["status"]
 
    def _append(self, job_id: str, status: str, **extra):
        rec = {"job_id": job_id, "status": status, "ts": time.time(), **extra}
        with self.path.open("a") as f:
            f.write(json.dumps(rec, ensure_ascii=False) + "\n")
        self._state[job_id] = status
 
    def is_inflight_or_done(self, job_id: str) -> bool:
        # inflight / uncertain / done: states the machine must not resend on its own
        return self._state.get(job_id) in {"inflight", "uncertain", "done"}
 
    def mark_inflight(self, job_id): self._append(job_id, "inflight")
    def mark_uncertain(self, job_id): self._append(job_id, "uncertain")
    def mark_failed(self, job_id): self._append(job_id, "failed")
    def mark_done(self, job_id, **usage): self._append(job_id, "done", **usage)

The point is holding uncertain as its own distinct state. If you fold a timed-out request into failed, the next morning's retry pass reads "failed means resend" and reproduces the double charge. uncertain is the marker for "the machine must not touch this until a human looks." I route only those requests into a morning review queue, reconcile them against usage to see whether a charge really landed twice, and only then decide by hand whether to resend.

Recording input and output tokens in the ledger also gives you real numbers on the cost side later. The per-token cost accounting itself is covered in building a cost ledger that buckets Claude API usage by purpose, so sharing the same ledger keeps operations in one place.

How much it actually helped

I compared logs over the same seven days and the same volume of work, before and after these three changes.

On paper it is 19 requests a week, a little under 2%. But because it runs unattended, that 2% had been quietly stacking up every week with no one to notice. The trickiest kind of indie-development cost, I was reminded, is the sort that leaks silently where you cannot see it.

If you are processing a large volume of requests in bulk anyway, another option is to lean on the asynchronous Message Batches, which never wait on a live response and so never hit this 10-minute wall by construction. The trade-offs are in designing cost around the async Message Batches API, and the general backoff design sits on top of an exponential backoff that honors the Retry-After header.

A first step to try

If you have even one call today that sets max_tokens above 10,000 without streaming, rewrite that one call first. Swap messages.create for messages.stream and iterate over text_stream. Once the possibility of touching the 10-minute wall is gone, the entrance to the double charge closes on the spot.

Until I hit this myself, I had trusted "if it times out, resend it" without a second thought. In unattended operation, that trust quietly compounds into extra charges. Hold enough state that the machine can answer whether a resend is allowed. Starting there today would be my one suggestion. Thank you for reading.