I Could Detect Failures — Then the Alerts Grew So Loud I Stopped Noticing

Deduplicate and group — fold one root cause into one message

The next lever was deduplication. When the same connector fails eight times in five minutes, I do not need eight messages — I need one: that connector is down.

So I gave each event a fingerprint, built from "site x tier x failure kind." Variable parts of the error text (timestamps, request IDs) are excluded from the fingerprint. Events sharing a fingerprint are folded into a single message within a grouping window (I use ten minutes), counting only the number of occurrences.

import hashlib, re
 
def fingerprint(site: str, stage: str, err: str) -> str:
    # Drop volatile parts so "same cause" collapses to one key
    norm = re.sub(r"\d{2,}", "N", err.lower())          # squash digit runs
    norm = re.sub(r"req_[a-z0-9]+", "req_X", norm)       # squash request IDs
    key = f"{site}|{stage}|{norm[:80]}"
    return hashlib.sha1(key.encode()).hexdigest()[:12]

Before fingerprints, my notifications arrived one per event. After, they shrink to one per cause. That difference lands directly on the cognitive load of whoever reads them — which, here, is me.

Silence the flapping — hysteresis

Even after folding, flapping remains. A connector drops, recovers, drops again. Ring naively on state transitions and you get "down," "up," "down," "up" in alternation.

I borrowed hysteresis from control theory: use different thresholds for firing and clearing.

Setting a high bar to fire and a steady bar to clear silences the back-and-forth near the boundary. One thing I felt strongly: the recovery notification is the one to suppress. "Recovered" looks reassuring, but it actually steals your attention once more. I routed recoveries to the digest and reserved paging for firing only.

Quiet hours, and the single line that breaks through them

I sleep at night, so I cannot page on everything after midnight. But muting everything until morning erases the whole point of a dead man's switch.

So I drew exactly one breakthrough rule through quiet hours.

• During the overnight window (00:00–07:00 JST), events are demoted to notify at most and sent to the digest.
• Except for a hard-down (all sites zero output / auth expired / billing path halted), which breaks through quiet hours to page.

The axis of the decision is: "does the wound spread if it waits until morning?" A night of empty runs cannot be recovered, but noticing next morning caps the damage at one night. Auth expiry or a halted billing path, by contrast, compounds lost opportunity the longer it sits. So the former waits, the latter wakes me. That is where I drew the line.

On the app side of Dolice Labs I also watch AdMob revenue daily, and an event like fill rate suddenly pinning to zero sits on this same "the wound spreads if it waits" side. A failure that stops the revenue plumbing is a different kind of thing from one article shipping late.

The escalation ladder — self-heal, notify, page

The final layer is the escalation staircase. In my setup, before anything reaches page, automatic recovery runs first (retry pull --rebase, recreate the repo, fall back to another model). If so, the alert should be coupled to the result of that self-heal.

1. Try self-heal: on detecting a failure, run the predefined recovery first. Ring nothing here.
2. If the heal works, log it: if output recovers via self-heal, keep only a record.
3. If the heal fails, notify; if it still does not recover, page: only when attempts hit their ceiling and output is still zero do I get woken.

The point of this staircase is not to summon a human while the machine is still trying to fix it. Summon only at the moment the machine gives up. Only then does a page become a message worth waking for.

Implementation: a thin notification gate

Folding the four pieces above (tiering, fingerprint collapse, hysteresis, quiet-hours breakthrough) into a single gate looks like this. State lives in an external JSONL so it survives even if the job process itself dies.

import json
 
class AlertGate:
    def __init__(self, state_path, quiet=(0, 7)):
        self.state_path = state_path
        self.quiet = quiet
        self.state = self._load()
 
    def _load(self):
        try:
            return json.load(open(self.state_path))
        except FileNotFoundError:
            return {}
 
    def _save(self):
        json.dump(self.state, open(self.state_path, "w"))
 
    def _in_quiet(self, hour):
        lo, hi = self.quiet
        return lo <= hour < hi
 
    def observe(self, fp, ok, severity, hard_down, hour):
        s = self.state.setdefault(fp, {"fail": 0, "succ": 0, "firing": False})
        if ok:
            s["succ"] += 1; s["fail"] = 0
            if s["firing"] and s["succ"] >= 2:      # clear on 2 successes
                s["firing"] = False; self._save()
                return ("recover", "digest")         # recoveries go to digest
            self._save(); return (None, None)
        # failure side
        s["fail"] += 1; s["succ"] = 0
        if not s["firing"] and s["fail"] >= 3:      # fire on 3 failures
            s["firing"] = True
        self._save()
        if not s["firing"]:
            return (None, "log")                     # still within the wobble
        # decide destination while firing
        if severity == "page":
            if self._in_quiet(hour) and not hard_down:
                return ("fire", "digest")            # demote during quiet hours
            return ("fire", "page")                  # hard-down breaks through
        return ("fire", "notify")

Call this gate against each step result of the generation job. Based on the returned destination (page / notify / digest / log), hand the actual send to a separate function. Keeping the gate focused purely on the "ring or not" decision — decoupled from the send mechanism (phone push, digest append, ledger write) — is what pays off later when you want to change quiet hours or swap the delivery channel.

Combine fingerprint() with this AlertGate and notifications settle into a shape that "silences the wobble per cause, folds the overnight, and wakes you only for the real thing."

The numbers I saw, and where I stumbled

Here are the results of running this gate over nightly generation across four sites for three weeks.

What mattered more than the numbers was that trust came back: if my phone rings, something is genuinely happening. Silence itself becomes reassurance. Only once you reach that state does automation get close to "handed off completely."

Let me leave the stumble on record too. At first I included the entire error message in the fingerprint, so variable IDs scattered the fingerprints and grouping did nothing at all. Only after adding normalization to squash digit runs and request IDs did folding start to work. The difficulty of "treating the same cause as the same" concentrates entirely in the tuning of that normalization: squash too much and you lump distinct failures together; squash too little and nothing folds. Expect to keep adjusting it for a while, watching your own logs.

For the connector-side observability (making error and latency visible), I wrote separately in instrumenting solo operations around the observability public beta. Measure with the instruments, fold with the gate. Only once those two sit together, I now believe, can you keep running overnight automation with any peace of mind.

Spending my time on the design that stays silent, rather than the design that rings, has been the best-value investment I have made so far. May your nights be a little quieter.