Half of My Scheduled Runs Vanished Without a Single Error

Write the Record at Launch, Not at Completion

The first fix is to change when the execution record gets written. Instead of writing results after the work finishes, write the "this run started" line the moment it starts, then fill in only the status on exit. A run that gets killed halfway through still leaves a trace.

#!/usr/bin/env bash
# Write the record at launch; settle only the status on exit.
# Even a killed run leaves a "it did start" line behind.
set -uo pipefail
 
LOG_DIR="${LOG_DIR:-$HOME/logs/batch}"; mkdir -p "$LOG_DIR"
LOG="$LOG_DIR/run.jsonl"
STARTED="$(date +%Y-%m-%dT%H:%M:%S)"   # same clock cron uses: host local time
TMP="$(mktemp)"
 
finish() {
  code=$?
  case "$code" in
    0)  status=ok ;;
    64) status=skipped ;;   # preconditions unmet, deliberately did nothing
    *)  status=failed ;;
  esac
  printf '{"started_at":"%s","status":"%s","exit":%d,"tail":%s}\n' \
    "$STARTED" "$status" "$code" \
    "$(tail -c 200 "$TMP" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))')" \
    >> "$LOG"
  rm -f "$TMP"
}
trap finish EXIT
 
"$@" > "$TMP" 2>&1

Four paths, run locally: clean exit, deliberate skip, error exit, and a SIGTERM kill.

{"started_at":"2026-08-31T12:10:38","status":"ok","exit":0,"tail":"done\n"}
{"started_at":"2026-08-31T12:10:38","status":"skipped","exit":64,"tail":"nothing to do\n"}
{"started_at":"2026-08-31T12:10:38","status":"failed","exit":3,"tail":"boom\n"}
{"started_at":"2026-08-31T12:10:38","status":"failed","exit":143,"tail":""}

That last line, exit 143, is the SIGTERM case. The job body wrote nothing at all, yet the record survived because the trap sits outside it. This is the whole reason for using trap rather than appending to the log at the end of the script: if the record lives inside the work, it dies with the work.

Mapping exit code 64 to "deliberately did nothing" matters more than it looks. Rate limit reached, zero items in the queue, previous run still holding the lock. All of those are correct decisions, but without a record they are indistinguishable from a run that never happened. Doing nothing has to write something.

The wrapper does not care what it wraps. It simply runs whatever command it is handed, so alongside the image classifier I route my unattended claude -p headless runs through the same script. Since claude -p exits non-zero on failure, the recording side needed no changes at all.

Reconcile Expected Firings Against Real Records

The second half is the reconciliation itself: match expanded expectations against the records that actually exist.

#!/usr/bin/env python3
"""Reconcile expected firing times against real execution records."""
import json
import sys
from datetime import datetime, timedelta
from pathlib import Path
 
from occ import expected
 
TOLERANCE = timedelta(minutes=20)   # how much launch delay we tolerate
 
 
def load_records(log_dir: Path):
    """One JSONL line per run; status is ok / skipped / failed."""
    recs = []
    for f in sorted(log_dir.glob("*.jsonl")):
        for line in f.read_text(encoding="utf-8").splitlines():
            line = line.strip()
            if not line:
                continue
            try:
                r = json.loads(line)
                recs.append((datetime.fromisoformat(r["started_at"]), r))
            except (ValueError, KeyError):
                # Do not let one bad line kill the report, but do not eat it silently either
                print(f"  ! skipping malformed line in {f.name}: {line[:60]}", file=sys.stderr)
    return recs
 
 
def reconcile(cron, log_dir, since, until):
    recs = load_records(Path(log_dir))
    unmatched = list(recs)
    missing = []
    for want in expected(cron, since, until):
        hit = next((r for r in unmatched if abs(r[0] - want) <= TOLERANCE), None)
        if hit:
            unmatched.remove(hit)      # never reuse one record for two expectations
        else:
            missing.append(want)
    return missing, unmatched, recs
 
 
if __name__ == "__main__":
    cron, log_dir = sys.argv[1], sys.argv[2]
    until = datetime.fromisoformat(sys.argv[3]) if len(sys.argv) > 3 else datetime.now()
    since = until - timedelta(days=7)
 
    missing, extra, recs = reconcile(cron, log_dir, since, until)
    by = {}
    for _, r in recs:
        by[r.get("status", "?")] = by.get(r.get("status", "?"), 0) + 1
 
    print(f"expected {len(missing) + len(recs) - len(extra)} / recorded {len(recs)} {by}")
    print(f"missing {len(missing)}")
    for m in missing[:5]:
        print("  -", m.strftime("%Y-%m-%d %H:%M"))
    if len(missing) > 5:
        print(f"  ... and {len(missing) - 5} more")
    sys.exit(1 if missing else 0)

Fed a week where only the morning runs left records, it prints:

expected 14 / recorded 7 {'ok': 6, 'skipped': 1}
missing 7
  - 2026-08-25 16:30
  - 2026-08-26 16:30
  - 2026-08-27 16:30
  - 2026-08-28 16:30
  - 2026-08-29 16:30
  ... and 2 more

Every missing timestamp lands on 16:30. Once the output looks like that, you know to stop reading the config and start looking at the launcher. Reading individual log entries one at a time never surfaces that shape.

Two implementation details are worth keeping. Remove a record from the candidate pool once it matches, or a single delayed run will satisfy several expectations and under-report the gap. And pick the tolerance deliberately: 20 minutes absorbs queueing and startup contention without reaching the neighbouring slot. If your slots sit closer together, shrink it to well under half the interval.

There is a third detail I only learned by stepping in it. My first version of the wrapper stamped started_at with date -u, in UTC. Cron, meanwhile, runs on the host's local clock, and the expanded expectations follow that same clock. On my JST machine the two drifted apart by nine hours, and the reconciler reported "missing 14, extra 7" for a week in which all seven morning runs had actually happened. A reconciliation that declares a healthy job dead is worse than no reconciliation.

The fix is a single rule: expectations and records must be written against the same clock. That is why the wrapper above uses plain local date rather than date -u. If you expect the host's timezone to change later, you can store an offset in the timestamp and let fromisoformat handle it, but I chose the simpler path of matching cron's clock exactly.

Retrofitting This onto a Job You Already Run

If the job is already in production, this order keeps your existing records usable throughout.

1. Start by counting, and change nothing else

Leave the record format alone and just count expected firings first. If your existing logs carry usable filenames or timestamps, swapping out load_records for that format is enough to make the reconciliation run. A large gap at this stage lets you start investigating without waiting on a logging rewrite.

2. Move the write outside the work

Then switch to the trap wrapper. Wrapping the existing job from the outside keeps the diff small and the rollback trivial. I changed both pieces at once the first time and could no longer tell whether a gap came from missing old records or from a flaw in the new wrapper.

3. Put the reconciliation at the front of your morning

Finally, run the reconciler once a day. It exits 0 when nothing is missing and non-zero otherwise, so your alerting condition is a single line: notify when this command fails. You add a new kind of monitoring without adding a new thing to watch.

The Same Reconciliation Surfaces Three Different Problems

Once expectations and records sit side by side, three distinct symptoms fall out of the same comparison.

Extras are easy to shrug off and expensive to ignore. Two copies of the same job writing to the same destination can corrupt state quietly, which is why I treat duplicates as seriously as gaps.

Chronic lateness is a forecast rather than a fault. A slot that keeps matching at the very edge of tolerance is telling you it will cross into the next window soon, which is a good moment to split the work or move the start time.

What the Fix Turned Out to Be

The remedy on the cause side was unglamorous: if you want a job to run twice a day, register it twice rather than folding both slots into one line like 30 4,16 * * *.

This is not a claim about crontab syntax. As shown above, the syntax expands to 14 firings exactly as written. How a given execution platform handles multiple slots in one entry varies, and on my setup only one of the two was ever launched. Rather than reverse-engineer the platform, splitting into one registration per slot removes the ambiguity outright.

I kept the reconciliation in place afterwards. Knowing the cause of this gap is a different thing from being protected against the next one.

Do This Tomorrow Morning

Pick one scheduled job you own and count its expected firings over the last seven days. Then put that number next to the count of execution records.

If the two numbers disagree, something is happening outside your monitoring. If they agree, you can finally say that nothing is wrong and point at a record when you say it. Zero errors was never evidence of that.

The more automation you add, the quieter the runs that did not happen become. Collect only the records of runs that did happen, and you will keep mistaking silence for health. I did.