My background session sat at running all night — adding heartbeats to the agents view

Stamping the heartbeat from a hook

The next question is who writes the stamp.

The obvious instinct is to stand up a separate watcher process alongside each session. I avoided that. A watcher introduces a nested problem — now who watches whether the watcher died? Twelve background sessions would mean twelve watchers, doubling what I have to manage.

So I use a PostToolUse hook. Claude Code can fire a hook on every tool call. Put the heartbeat write there, and the stamp refreshes as a side effect every time the session does any work. No extra process. When the session dies, the hook stops with it, so no nested supervision is born.

Here is the small script that writes the stamp.

#!/usr/bin/env bash
# heartbeat.sh - refreshes the liveness stamp for the current session on each call
# usage: heartbeat.sh <phase>
set -euo pipefail
 
# pass the session name in as an env var at launch (keep it aligned with the readable name)
SESSION="${CC_SESSION_NAME:-unknown}"
PHASE="${1:-tool}"
 
# avoid the cloud-synced folder; use local tmpfs (explained later)
HB_DIR="/tmp/cc-heartbeat"
mkdir -p "$HB_DIR"
 
# write epoch seconds, never a date string (avoids the timezone accident)
NOW=$(date +%s)
 
# atomically replace a single-line JSON (never let a reader see a half-written line)
TMP="$(mktemp "${HB_DIR}/.tmp.XXXXXX")"
printf '{"session":"%s","ts":%s,"phase":"%s"}\n' "$SESSION" "$NOW" "$PHASE" > "$TMP"
mv -f "$TMP" "${HB_DIR}/${SESSION}.json"

Writing to mktemp and then replacing with mv removes any chance the reconciler reads a half-written line. A mv within the same directory is atomic, so the reader always sees exactly one complete line.

Wire the script into the hook. Add this to the project's .claude/settings.json.

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "heartbeat.sh tool:${TOOL_NAME}"
          }
        ]
      }
    ]
  }
}

The matcher is * so it fires on every tool, and the phase carries the tool name. That way the stamp alone tells you what the session was last doing. If a stalled session's phase reads tool:web_fetch, you immediately suspect a network hang.

The reconciler that judges a stall

Once stamps accumulate, you need the watching side. The reconciler can be a tiny program that sweeps the heartbeat directory and reports stale stamps as stall candidates.

#!/usr/bin/env node
// reconcile.mjs - reads heartbeat freshness and reports stall candidates
import { readdirSync, readFileSync } from "node:fs";
 
const HB_DIR = "/tmp/cc-heartbeat";
const NOW = Math.floor(Date.now() / 1000);
 
// thresholds: set well above your longest single tool (see the pitfalls section)
const STALE_SEC = 300;   // 5 min of silence => "suspected stall"
const DEAD_SEC = 900;    // 15 min of silence => "likely stopped"
 
const rows = [];
for (const f of readdirSync(HB_DIR)) {
  if (!f.endsWith(".json")) continue;
  let hb;
  try {
    hb = JSON.parse(readFileSync(`${HB_DIR}/${f}`, "utf8"));
  } catch {
    continue; // skip a corrupt line (atomic replace should prevent this, but be safe)
  }
  const age = NOW - hb.ts;
  let state = "alive";
  if (age >= DEAD_SEC) state = "dead?";
  else if (age >= STALE_SEC) state = "stale?";
  rows.push({ session: hb.session, ageSec: age, phase: hb.phase, state });
}
 
// oldest first; signal via non-zero exit only when something is suspicious
rows.sort((a, b) => b.ageSec - a.ageSec);
const suspect = rows.filter((r) => r.state !== "alive");
for (const r of rows) {
  console.log(`${r.state.padEnd(6)} ${String(r.ageSec).padStart(5)}s  ${r.session}  (${r.phase})`);
}
process.exit(suspect.length > 0 ? 1 : 0);

Run it every 3 to 5 minutes from cron or launchd. Tie the non-zero exit to a notification and a stall reaches you the moment it happens. I call it from launchd on macOS and surface a notification if even one session looks suspicious. For verifying the completion itself after the fact, I wrote earlier about doubting a silent success with an end-of-run assertion; the heartbeat sits before that, watching whether the run is alive while it is still moving. Keep both and you cover both an in-flight stall and an after-the-fact no-op.

The output looks like this.

dead?    7412s  wallpaper-ios-deps-0706-a3f  (tool:web_fetch)
alive      41s  gemilab-changelog-0706-9c2  (tool:bash)
alive      12s  blog-crash-summary-0706-77e  (tool:edit)

That first line is exactly the session I left stalled for seven hours at the start. Its phase is frozen at tool:web_fetch, so at a glance it was hung waiting on an external fetch.

How much it actually shrank

I recorded how the time-to-notice changed before and after. These are two weeks of real numbers from my own setup — an average of ten to twelve sessions a night, of which roughly one or two stall in a week.

More than the numbers, the real change for me was that I stopped opening the agents view every morning with a knot in my stomach. Because I know a stall will page me within minutes, I hand off the overnight runs more calmly than before. Investment in observability usually pays back as peace of mind before it shows up in a metric.

Where I tripped

Getting to a clean version took three stumbles. None of them were visible until I actually ran it.

Missing silent thinking

A PostToolUse hook, as the name says, only fires after a tool call. So while a session is thinking hard without calling any tool at all, the heartbeat does not refresh. Misjudge that as a stall and you risk killing legitimate deep reasoning.

I settled this by setting the threshold well above the longest legitimate single tool. Keeping the tool: prefix in the phase also helps: it tells you the last activity was a tool, so you can partly distinguish long thinking from a true hang. If you want to be strict, write a phase:thinking stamp once right at session start and overwrite it per tool afterward, which also closes the gap in the first moments after launch.

Write contention on a cloud-synced folder

At first I put the heartbeats directly under my workspace, and that was a mistake. My working folder is cloud-synced, and the small writes every few seconds contended with the sync process, so the reconciler occasionally read stale content.

The fix is simple: keep heartbeats in local /tmp, which is not synced. The stamp is disposable, transient information — there is nothing to persist. The traps around file materialization in cloud-synced folders run deep, and I sorted them out separately in automating file materialization for a cloud-synced workspace. High-frequency writes for the living state belong outside anything that syncs.

False positives from long single tools

Some tools, like code_execution, can take close to 90 seconds in a single call. Set the threshold shorter than that and you will sound "stall" in the middle of a perfectly valid long run.

So set STALE_SEC comfortably longer than the longest single tool your workload can produce. Mine top out around two minutes, so I use a five-minute threshold. This depends entirely on your workload; start wide to kill false positives, then tighten once you trust it. For threshold design in general, the alerting chapter of a solid SRE text treats the tradeoff between false alarms and alert fatigue carefully, and I reread it whenever I set a threshold.

Why I settled on this shape

Let me leave one design rationale.

This heartbeat layer does not replace the agents view. It complements it. The agents view owns presence, commit correlation owns traceability, end-of-run assertions own completion, and the heartbeat owns liveness. Each answers a different question, and any attempt to cover all of them with one mechanism always leaves a hole somewhere. The RSS watchdog design that watches memory growth in long sessions is another node in this same observability mesh.

Reliability under unattended operation rises not through one dramatic move but by layering small nodes like these — something I keep feeling as I run these pipelines day after day at Dolice.

If you want to try it, write a single heartbeat.sh, wire it to the PostToolUse hook, and watch just one of tonight's background sessions. If the stamp keeps refreshing by morning, that is your starting point for a liveness layer.

Thank you for reading, and I hope it lets you hand off a few more of your nightly runs with confidence.