When Claude API Streaming Stops Without an Error: Detecting Silent Stalls and Resuming Mid-Stream

Carry the received text forward and continue "mid-stream"

Once you can detect a silent stall, the next problem is resuming. The naive fix — resend the same request — makes the model regenerate the entire response from scratch. For a long article or report, that wastes both tokens and time.

Instead, use a prefill: pass the text you already received as the final assistant message. Claude continues from the prior assistant turn, so it picks up roughly where it stopped.

def resume_stream(client, base_messages, received_text, **kw):
    """Pass received_text as an assistant prefill and generate the continuation."""
    messages = base_messages + [{"role": "assistant", "content": received_text}]
    for text in stream_with_watchdog(client, messages, **kw):
        yield text
 
 
def robust_generate(client, base_messages, max_resumes=3, **kw):
    full = ""
    for attempt in range(max_resumes + 1):
        try:
            gen = (resume_stream(client, base_messages, full, **kw)
                   if attempt else stream_with_watchdog(client, base_messages, **kw))
            for text in gen:
                full = stitch(full, text) if attempt else full + text
            return full   # completed normally
        except StreamStalled as e:
            full = stitch(full, e.args[0]) if e.args and isinstance(e.args[0], str) else full
            continue       # next loop resumes via prefill
    return full

One caveat: a prefill resume does not guarantee a seamless join. If the stop landed mid-word or mid-symbol, the start of the continuation may slightly overlap the previous text or fail to connect. In practice, add a step that trims the overlap at the seam.

def stitch(prev: str, cont: str, max_overlap: int = 80) -> str:
    """Detect and drop a duplicated boundary between prev's tail and cont's head."""
    window = prev[-max_overlap:]
    for size in range(min(len(window), len(cont)), 0, -1):
        if window.endswith(cont[:size]):
            return prev + cont[size:]
    return prev + cont

It is not a perfect splice, but it is far better than "the last few paragraphs are silently missing." For me, eliminating the loss itself was worth far more than the slight roughness at the seam. I still run the resumed text through a single read-through check (the verification step below) before saving.

Design timeouts as a budget, split into four layers

Once you face a silent stall, a timeout stops being one number and becomes several layers with different jobs. I manage four:

The connect and total layers are easy to guard with the SDK or asyncio.wait_for. The first-token and inter-token layers can share one mechanism: branch on whether last_delta["t"] has been updated yet.

def deadline_for(received_any: bool, first_token_s: float, inter_token_s: float) -> float:
    """First-token budget until any delta has arrived; inter-token budget after."""
    return first_token_s if not received_any else inter_token_s

The important part is to set thresholds from measured distributions, not by feel. Emit the elapsed time for each layer as a metric, watch p50 / p95 / p99 for about a week, then draw the lines. In my automation the first-token p95 was about 18 seconds, so I set the first-token budget to 45 (more than double p95); the inter-token p99 was around 6 seconds, so I capped it at 25. Drawing the line clear of the healthy p99 is how you avoid resuming needlessly on a false positive.

import logging, time
logger = logging.getLogger("claude.stream")
 
def timed_stream(client, messages, **kw):
    t0 = time.monotonic(); first_t = None; gaps = []; last = t0
    for text in stream_with_watchdog(client, messages, **kw):
        now = time.monotonic()
        if first_t is None: first_t = now - t0
        else: gaps.append(now - last)
        last = now
        yield text
    p99 = sorted(gaps)[int(len(gaps) * 0.99)] if gaps else 0.0
    logger.info("first_token=%.2fs total=%.2fs max_gap=%.2fs p99_gap=%.2fs",
                first_t or -1, time.monotonic() - t0, max(gaps) if gaps else 0.0, p99)

Once that one log line accumulates, you can review both the validity of your thresholds and any degradation of the path in numbers. "It stalls sometimes" becomes "first-token p95 is 6 seconds higher than last week," and the precision of your response changes entirely.

Reduce silence at the path (proxy buffering)

The app-side watchdog notices a stall and continues; settings that avoid generating silence in the first place help too. The most common culprit is proxy buffering. If Nginx sits in the path, leaving response buffering on batches the deltas, so the inter-token gaps look artificially spiky and tempt the watchdog into false positives.

location /api/chat {
    proxy_pass http://backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
 
    proxy_buffering off;          # without this, deltas arrive batched and delayed
    proxy_cache off;
    chunked_transfer_encoding on;
 
    proxy_connect_timeout 10s;
    proxy_read_timeout 600s;      # socket-layer backstop; silent stalls are guarded in the app
    proxy_send_timeout 600s;
}

If your app relays SSE itself, add X-Accel-Buffering: no to the response headers to disable Nginx buffering explicitly. Not buffering at the path and measuring content progress in the app are two separate defenses; you need both to be resilient to silent stalls.

Add one final check before saving

Once resume is in place, you want to confirm the resumed response actually finished. I put a one-line gate that inspects stop_reason right before saving.

def assert_complete(final_message) -> None:
    sr = final_message.stop_reason
    if sr == "max_tokens":
        raise RuntimeError("Hit max_tokens; a continuation may remain.")
    if sr not in ("end_turn", "stop_sequence"):
        raise RuntimeError(f"Incomplete stop reason: {sr}")

After a prefill resume, inspect the final message's stop_reason. If it is end_turn, the model at least considers itself done. The thing that hurt most about silent stalls was that the absence of an error meant I never noticed the output was incomplete. This one line closes that blind spot.

Looking back

None of these pieces is flashy on its own: a watchdog that measures content progress, a prefill resume of received text, a four-layer timeout budget, disabling buffering at the path, and a stop_reason check before saving. Each is a small move. Together, though, they nearly eliminated the most easily-missed failure — the response that silently loses its tail.

The longer something runs unattended, the more I value "it doesn't quietly break partway" over any flashy feature. A bug that raises no error is worse than one that does. That is exactly why, for me, starting by making the silence observable looked like the long way around but turned out to be the shortest path. If you recognize the same missing-tail symptom, I hope this gives you a thread to pull on.