The Day a Non-Responding MCP Call Swallowed an Entire Unattended Run — Owning the Stop With Your Own Deadline

An asyncio wrapper that wraps the call in your own deadline

The implementation starts by wrapping the MCP tool call in your own deadline and sorting the result into three buckets: it hung and you cut it off, the server returned an explicit error, or a response came back normally. This distinction is what supports the retry decision later.

import asyncio
import enum
from dataclasses import dataclass
from typing import Any, Awaitable, Callable
 
 
class Outcome(enum.Enum):
    OK = "ok"
    DEADLINE = "deadline"   # cut off by our own deadline (hang-equivalent)
    ERROR = "error"         # server returned an explicit failure
 
 
@dataclass
class CallResult:
    outcome: Outcome
    value: Any = None
    detail: str = ""
    elapsed: float = 0.0
 
 
async def call_with_deadline(
    fn: Callable[[], Awaitable[Any]],
    deadline_s: float,
) -> CallResult:
    """Wrap an MCP tool call in our own deadline, returning a 3-way result.
 
    Keep deadline_s inside the platform's default cutoff.
    """
    loop = asyncio.get_event_loop()
    start = loop.time()
    try:
        value = await asyncio.wait_for(fn(), timeout=deadline_s)
        return CallResult(Outcome.OK, value=value, elapsed=loop.time() - start)
    except asyncio.TimeoutError:
        # reached our own deadline with no response
        return CallResult(
            Outcome.DEADLINE,
            detail=f"no response within {deadline_s}s",
            elapsed=loop.time() - start,
        )
    except Exception as e:  # noqa: BLE001 — caller decides on the failure kind
        return CallResult(
            Outcome.ERROR,
            detail=f"{type(e).__name__}: {e}",
            elapsed=loop.time() - start,
        )

Here asyncio.wait_for sends CancelledError to the coroutine to cut it off, but there's no guarantee the other side's processing stops immediately. That's exactly why it matters not to fire the next call at the same connector right after cutting off. Fire again and it tends to collide with the prior call that hasn't fully stopped, leaving the next one non-responsive as well.

A hang and a normal error have different premises for retrying

I separated DEADLINE from ERROR because the meaning of a retry differs.

An ERROR the server returned usually leaves you information: an expired credential, an invalid argument, a transient upstream fault. For some kinds a retry is meaningful; for others, throwing it again gets the same result every time.

A DEADLINE, by contrast, is a cutoff made without knowing the other side's current state. A retry here can also work in the direction of crowding the other side further. So I keep DEADLINE retries to wider intervals and a more modest count. And I mix jitter into each retry so that several scheduled runs don't re-enter at the same instant and build a wave.

import random
 
 
async def call_with_retry(
    fn: Callable[[], Awaitable[Any]],
    deadline_s: float,
    max_attempts: int = 3,
    base_backoff_s: float = 2.0,
) -> CallResult:
    """Retry according to the 3-way result. Use only for idempotent calls."""
    last = CallResult(Outcome.ERROR, detail="not attempted")
    for attempt in range(1, max_attempts + 1):
        last = await call_with_deadline(fn, deadline_s)
        if last.outcome is Outcome.OK:
            return last
        if attempt == max_attempts:
            break
        # DEADLINE: other side's state is unknown, so widen the interval
        factor = 2.0 if last.outcome is Outcome.DEADLINE else 1.0
        backoff = base_backoff_s * (2 ** (attempt - 1)) * factor
        backoff += random.uniform(0, backoff)  # full jitter
        await asyncio.sleep(backoff)
    return last

It's worth stressing that retries are limited to idempotent calls. A call you cut off on DEADLINE may well have completed on the other side. Retrying a write tool unconditionally risks applying it twice. For writes, attach an idempotency key or leave them out of the retry set entirely. That line is continuous with the "what to allow" design I covered in MCP policy enforcement and allowlist design for unattended agents.

Don't let a broken connector drag the rest of the run with it

Even once you can bound a single call, if the same connector returns DEADLINE repeatedly, every added retry eats more of the run's budget. So insert a per-connector circuit breaker. After a set number of consecutive failures, hold that connector open for the rest of this run and skip it immediately before calling.

import time
 
 
class ConnectorBreaker:
    """A minimal per-connector circuit breaker.
 
    Open when consecutive failures hit the threshold; skip until cooldown passes.
    """
 
    def __init__(self, fail_threshold: int = 2, cooldown_s: float = 120.0):
        self.fail_threshold = fail_threshold
        self.cooldown_s = cooldown_s
        self._fails = 0
        self._opened_at = 0.0
 
    def is_open(self) -> bool:
        if self._fails < self.fail_threshold:
            return False
        if time.monotonic() - self._opened_at >= self.cooldown_s:
            self._fails = 0  # half-open: let one attempt through
            return False
        return True
 
    def record(self, result: CallResult) -> None:
        if result.outcome is Outcome.OK:
            self._fails = 0
            return
        self._fails += 1
        if self._fails == self.fail_threshold:
            self._opened_at = time.monotonic()
 
 
async def guarded_call(
    breaker: ConnectorBreaker,
    fn: Callable[[], Awaitable[Any]],
    deadline_s: float,
) -> CallResult:
    if breaker.is_open():
        return CallResult(Outcome.ERROR, detail="breaker open: skipped")
    result = await call_with_retry(fn, deadline_s)
    breaker.record(result)
    return result

Whether an open breaker makes the step a whole-run failure or just lets later steps proceed depends on the step's nature. In my runs, steps like fetching reference data — where the rest still holds even if it's missing — get skipped and continue, while steps tied to finalizing output give up for that cycle. Either way, I log the fact that the breaker opened. Skip it silently and you can't reach the cause next time the same thing happens.

Leave the fact that you waited in the log

What made hangs nasty was the thin trace. So when I switched to cutting off on my own, I made sure to also write out "how many seconds it waited and why it cut off."

For each call, I flatten the result bucket, elapsed seconds, attempt count, and breaker state into one line. Whether DEADLINE skews toward a particular connector, or toward a time of day, only becomes visible once these lines accumulate. Back when things were killed while still frozen, I had no material even to ask the question.

How far to raise the log's granularity I treat as separate from the interactive triage procedure. The steps for a person chasing a connector that suddenly stopped working I'll leave to the diagnosis guide for when an MCP server suddenly stops in Claude Code; here I narrow to the minimum an unattended run needs to explain itself afterward.

Where it tends to trip you up

Set your own deadline longer than the outer one and the cutoff never fires, leaving you swallowed by the outermost layer after all. Keeping the inner one shorter is the prerequisite.

Re-firing the same connector right after a DEADLINE cutoff is also a source of accidents. If the prior call is still alive on the other side, it collides and the next one tends to go non-responsive too. Insert a backoff before retrying and break the wave with jitter.

Unconditionally retrying write calls is dangerous. Because DEADLINE includes the possibility of completion on the other side, attach an idempotency key or leave writes out of the retry set.

Set the circuit breaker's cooldown longer than the run budget and, once open, it never tries again this cycle. That's fine if intended — but it's often unintended, so decide it against the run's length.

The next step

First, check one value: the cutoff you rely on at the outermost layer — the platform limit or the run timeout. Then add just one deadline, clearly shorter than that, to your innermost MCP call. Once you've slotted call_with_deadline into one place and the result lands in the log as three buckets, the step that used to vanish while frozen finally starts to tell you "how many seconds it waited before being cut off." From there, adding retries and the breaker becomes a grounded starting point.

How you give up on a counterpart that won't respond matters more the more unattended the machinery is. I'm still mid-tuning at Dolice, but just deciding not to leave the stop to the platform made my scheduled-run logs noticeably quieter. I hope it helps anyone building unattended operations the same way.