When a Connector Starts Slowing Down at Night: A Health-Aware Circuit Breaker for Solo Automation

Implementing the three-state breaker

A circuit breaker holds three states: Closed for normal operation (let calls through), Open while tripped (block them), and Half-Open to probe for recovery (let exactly one through and watch). I start with the core that holds the state and the observation window.

import time
from collections import deque
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional
 
 
class State(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"
 
 
@dataclass
class Sample:
    ok: bool
    latency_ms: float
    ts: float
 
 
@dataclass
class ConnectorBreaker:
    name: str
    window_sec: float = 900.0        # observe health over the last 15 minutes
    min_samples: int = 8             # hold off deciding below this count
    error_rate_open: float = 0.5     # open above this error rate
    p95_open_ms: float = 8000.0      # open above this p95 latency
    cooldown_sec: float = 300.0      # wait this long before probing after opening
    half_open_success: int = 3       # consecutive successes needed to confirm recovery
 
    state: State = State.CLOSED
    opened_at: float = 0.0
    probe_success: int = 0
    _samples: deque = field(default_factory=deque)
 
    def _prune(self, now: float) -> None:
        limit = now - self.window_sec
        while self._samples and self._samples[0].ts < limit:
            self._samples.popleft()
 
    def _error_rate(self) -> float:
        if not self._samples:
            return 0.0
        failed = sum(1 for s in self._samples if not s.ok)
        return failed / len(self._samples)
 
    def _p95_latency(self) -> float:
        if not self._samples:
            return 0.0
        values = sorted(s.latency_ms for s in self._samples)
        idx = max(0, int(len(values) * 0.95) - 1)
        return values[idx]

Because _prune drops samples that fall outside the window on every touch, the decision always looks only at the most recent 15 minutes. The p95 here is a naive "sort and pick the top-5% boundary," but at the few-dozen-calls scale of a solo setup, that was plenty.

Open, closed, or probing

Next I add the two methods that move the state around each call. allow answers whether a call may go through, and record takes the outcome and updates the state.

    def allow(self, now: Optional[float] = None) -> bool:
        """Ask before every call. If False, send the work to the fallback."""
        now = time.time() if now is None else now
        self._prune(now)
        if self.state is State.OPEN:
            if now - self.opened_at >= self.cooldown_sec:
                self.state = State.HALF_OPEN
                self.probe_success = 0
                return True   # let exactly one probe through
            return False
        return True
 
    def record(self, ok: bool, latency_ms: float,
               now: Optional[float] = None) -> None:
        now = time.time() if now is None else now
        self._samples.append(Sample(ok=ok, latency_ms=latency_ms, ts=now))
        self._prune(now)
 
        if self.state is State.HALF_OPEN:
            if ok:
                self.probe_success += 1
                if self.probe_success >= self.half_open_success:
                    self.state = State.CLOSED
            else:
                self.state = State.OPEN
                self.opened_at = now
            return
 
        if len(self._samples) < self.min_samples:
            return
        if (self._error_rate() >= self.error_rate_open
                or self._p95_latency() >= self.p95_open_ms):
            self.state = State.OPEN
            self.opened_at = now

The thing I was careful about here is letting the Open-to-Half-Open transition happen only inside allow. If you drive a state transition off a passive event like elapsed time, you get hard-to-trace behavior — a breaker that "opened on its own with nobody calling it." Instead, when it's called, it checks the elapsed time and lets exactly one probe through. Once I made the design passive like this, state changes lined up one-to-one with the log, and debugging got noticeably easier.

Wrapping the call in the breaker

With the core in place, I wrap the actual connector call in a thin adapter. While the breaker is open, it never touches the connector and returns the supplied fallback instead.

import time
 
 
def call_with_breaker(breaker, fn, fallback):
    """Wrap a connector call in the breaker. Return fallback while open."""
    if not breaker.allow():
        return fallback()
 
    start = time.perf_counter()
    try:
        result = fn()
    except Exception:
        elapsed_ms = (time.perf_counter() - start) * 1000
        breaker.record(ok=False, latency_ms=elapsed_ms)
        return fallback()
 
    elapsed_ms = (time.perf_counter() - start) * 1000
    breaker.record(ok=True, latency_ms=elapsed_ms)
    return result

The design of the fallback is what decides whether the whole mechanism is worth anything. What I settled on is a retry queue that gets picked up "next cycle, not next morning." Instead of discarding work that couldn't be processed while open, I write it out as line-oriented JSON, and a later job picks it back up on a connector that has regained its health.

import json
from pathlib import Path
 
 
def enqueue_for_retry(task, path="retry_queue.jsonl"):
    """Divert work that fell during an open state; pick it up next cycle."""
    line = json.dumps(task, ensure_ascii=False)
    with Path(path).open("a", encoding="utf-8") as f:
        f.write(line + "\n")

With this in place, tripping the breaker becomes "deferring that work" rather than "losing the whole night." In my nightly jobs, only the handful of items tied to the slow connector flowed into the queue, and the rest proceeded as usual. I had confined the collateral damage to a single connector.

Carrying health across process boundaries

Here is a pitfall unique to solo automation. A scheduled job starts in a fresh process every time, so the breaker's observation window also starts from zero on each run. Last night's memory of a needle in the red is gone by the next launch.

So I carry only the observation samples to disk. Persisting the raw samples rather than the state itself, then reloading them at startup back into the window, was the clean approach.

import json
from pathlib import Path
 
 
def save_samples(breaker, path):
    rows = [{"ok": s.ok, "latency_ms": s.latency_ms, "ts": s.ts}
            for s in breaker._samples]
    Path(path).write_text(json.dumps(rows), encoding="utf-8")
 
 
def load_samples(breaker, path, now):
    p = Path(path)
    if not p.exists():
        return
    for row in json.loads(p.read_text(encoding="utf-8")):
        breaker._samples.append(
            Sample(ok=row["ok"], latency_ms=row["latency_ms"], ts=row["ts"])
        )
    breaker._prune(now)

There's a reason I persist samples rather than the Open/Closed state. If you save only "it was Open," the cooldown origin becomes ambiguous at the next launch and you lose track of how long to stay shut. Carry the raw samples, and you can rebuild the window right at startup and redo the decision from the same thresholds. The source of truth stays with the observations. That "carry the facts, not the state" stance is the same call I made in noticing from the outside when a scheduled generation silently comes up empty.

Don't trust recovery too quickly in Half-Open

The single most effective thing in production was not confirming recovery on one success in Half-Open.

At first, the moment one probe went through, I flipped straight back to Closed. But a slowed connector will occasionally return one call quickly by chance, and trusting that one to reopen fully led to a back-and-forth: the next wave of latency would arrive right after and trip it open again. A breaker that keeps flapping open and shut is barely different from having no breaker at all.

Setting half_open_success to 3 — only returning to Closed after three consecutive successes — settled the flapping. Conversely, a single failure during Half-Open sends it straight back to Open and restarts the cooldown clock. Doubt recovery and verify it; trust a single sign of decline. That asymmetry was the knack for staying reliably closed.

I dialed in the thresholds while running. Across my four sites' nightly processing, a 15-minute window, a 5-minute cooldown, and an 8-second p95 ceiling landed in the sweet spot — neither too twitchy nor too dull. Since each connector's baseline speed differs, I keep the p95 threshold per connector. Applying the same 8 seconds to one that usually returns in 900ms and one that takes 3 seconds felt too blunt.

Common pitfalls

A few I stepped on while building this.

Not confusing this with model-side fallback mattered too. Tiered fallback for when a model goes down is the territory of graceful degradation design for the Claude API; this is confined to the health of the connector — the plumbing. The two live on different layers, and fixing one doesn't protect the other. As an indie developer at Dolice, I deliberately keep a separate breaker for models and for connectors.

Your next step

If your nightly job is greeting the morning with the "not down, just slow" failure mode right now, start by wrapping just your most-used connector in this ConnectorBreaker. Leave the thresholds at their defaults and simply log a week of p95 and error rate — even that alone will show you how that connector breathes at night. Once the numbers are visible, the thresholds decide themselves.

Since I slipped one breaker in, I've stopped leaving a red needle untouched until morning. The range I can trust to steer itself through an unattended night has widened a little again. I'm still feeling my way through it, but I'd be glad to keep working at it alongside you. Thank you for reading.