Articles/Claude Code

⟐ Claude Code/2026-08-31Intermediate

Half of My Scheduled Runs Vanished Without a Single Error

A batch job set to run twice a day was only firing once. No errors, no failure alerts. Here is how to expand your own schedule, count expected runs, and reconcile them against execution records to catch silent misses.

Claude Code²⁴⁰ scheduled tasks¹⁰ cron³ operations²⁵ automation¹⁰⁷

✦ Premium Article

An asset classification batch was set to run twice a day, morning and late afternoon, a few hundred images per pass.

What tipped me off was the backlog, not the logs. Throughput was running at roughly half of what the schedule implied. The error log was empty. No failure notification had arrived. Every run that did leave a record had exited cleanly, with nothing unusual in its output.

The afternoon run had never started at all.

A run that never starts never fails. Because it never fails, nothing that watches for failure will ever mention it. This piece is about making that blank space visible.

The Crontab Syntax Was Fine

The schedule was written like this:

30 4,16 * * *

As crontab syntax, that is unambiguously twice a day. Still, I wanted to count it myself rather than trust my reading, so I wrote a small expander that takes a cron expression and lists every moment a firing is expected inside a window.

#!/usr/bin/env python3
"""Expand a cron expression into the moments a firing is expected."""
from datetime import datetime, timedelta
 
 
def _field(spec: str, lo: int, hi: int) -> set:
    out = set()
    for part in spec.split(","):
        step = 1
        if "/" in part:
            part, s = part.split("/", 1)
            step = int(s)
        if part in ("*", ""):
            start, end = lo, hi
        elif "-" in part:
            a, b = part.split("-", 1)
            start, end = int(a), int(b)
        else:
            start = end = int(part)
            if step != 1:
                end = hi
        out |= set(range(start, end + 1, step))
    return {v for v in out if lo <= v <= hi}
 
 
def expected(cron: str, start: datetime, end: datetime):
    mi, ho, dom, mon, dow = cron.split()
    M, H = _field(mi, 0, 59), _field(ho, 0, 23)
    DOM, MON = _field(dom, 1, 31), _field(mon, 1, 12)
    DOW = {d % 7 for d in _field(dow, 0, 7)}      # Sunday is both 0 and 7
    dom_restricted = dom.strip() != "*"
    dow_restricted = dow.strip() != "*"
 
    t = start.replace(second=0, microsecond=0)
    while t <= end:
        if t.minute in M and t.hour in H and t.month in MON:
            d_ok = t.day in DOM
            w_ok = ((t.weekday() + 1) % 7) in DOW
            # Both day-of-month and day-of-week restricted means OR, not AND
            ok = (d_ok or w_ok) if (dom_restricted and dow_restricted) else (d_ok and w_ok)
            if ok:
                yield t
        t += timedelta(minutes=1)

Over seven days it expands to 14 firings. My reading of the syntax was correct, which meant the loss was happening somewhere below the syntax, in whatever actually launches the job.

That was the fork in the road. Rereading the config file would never have gotten me there. Once you have confirmed the configuration says what you think it says, the next thing to look at is not the configuration but the record of what happened.

One detail I would not have trusted without expanding it: when both day-of-month and day-of-week are restricted, cron treats them as OR. So 0 9 1 * 1 fires on the first of the month and on every Monday. Expanding September gives 09-01 (Tue) plus each Monday, five firings, which matched what I expected only after I saw it printed.

A Failure Class That Never Reaches Your Failure Log

Most monitoring around automation is shaped to catch failures. Check the exit code, forward the exception, inspect the output. All three require a process to have existed.

When the launch itself is what goes missing, none of that shape applies.

Failure	Shows in failure log?	How you find out
Exception during processing	Yes	Alert fires immediately
Crash or timeout	Yes, via exit code	Alert fires
Preconditions unmet, did nothing	No	Intended behaviour, no problem
Never launched	No	Only by counting output volume

As an indie developer running long unattended jobs, I had built my monitoring entirely around the first two rows. There was no place in the design for a question about what did not happen.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦You will be able to prove, from records rather than from the absence of errors, that an automated job actually ran

✦You will be able to expand your own schedule, count expected firings, and detect the gap against real execution records with a single morning command

✦You will be able to catch failures where the launch itself disappears, before the missing work quietly piles up

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Write the Record at Launch, Not at Completion

The first fix is to change when the execution record gets written. Instead of writing results after the work finishes, write the "this run started" line the moment it starts, then fill in only the status on exit. A run that gets killed halfway through still leaves a trace.

#!/usr/bin/env bash
# Write the record at launch; settle only the status on exit.
# Even a killed run leaves a "it did start" line behind.
set -uo pipefail
 
LOG_DIR="${LOG_DIR:-$HOME/logs/batch}"; mkdir -p "$LOG_DIR"
LOG="$LOG_DIR/run.jsonl"
STARTED="$(date -u +%Y-%m-%dT%H:%M:%S)"
TMP="$(mktemp)"
 
finish() {
  code=$?
  case "$code" in
    0)  status=ok ;;
    64) status=skipped ;;   # preconditions unmet, deliberately did nothing
    *)  status=failed ;;
  esac
  printf '{"started_at":"%s","status":"%s","exit":%d,"tail":%s}\n' \
    "$STARTED" "$status" "$code" \
    "$(tail -c 200 "$TMP" | python3 -c 'import json,sys;print(json.dumps(sys.stdin.read()))')" \
    >> "$LOG"
  rm -f "$TMP"
}
trap finish EXIT
 
"$@" > "$TMP" 2>&1

Four paths, run locally: clean exit, deliberate skip, error exit, and a SIGTERM kill.

{"started_at":"2026-08-31T03:10:38","status":"ok","exit":0,"tail":"done\n"}
{"started_at":"2026-08-31T03:10:38","status":"skipped","exit":64,"tail":"nothing to do\n"}
{"started_at":"2026-08-31T03:10:38","status":"failed","exit":3,"tail":"boom\n"}
{"started_at":"2026-08-31T03:10:38","status":"failed","exit":143,"tail":""}

That last line, exit 143, is the SIGTERM case. The job body wrote nothing at all, yet the record survived because the trap sits outside it. This is the whole reason for using trap rather than appending to the log at the end of the script: if the record lives inside the work, it dies with the work.

Mapping exit code 64 to "deliberately did nothing" matters more than it looks. Rate limit reached, zero items in the queue, previous run still holding the lock. All of those are correct decisions, but without a record they are indistinguishable from a run that never happened. Doing nothing has to write something.

Reconcile Expected Firings Against Real Records

The second half is the reconciliation itself: match expanded expectations against the records that actually exist.

#!/usr/bin/env python3
"""Reconcile expected firing times against real execution records."""
import json
import sys
from datetime import datetime, timedelta
from pathlib import Path
 
from occ import expected
 
TOLERANCE = timedelta(minutes=20)   # how much launch delay we tolerate
 
 
def load_records(log_dir: Path):
    """One JSONL line per run; status is ok / skipped / failed."""
    recs = []
    for f in sorted(log_dir.glob("*.jsonl")):
        for line in f.read_text(encoding="utf-8").splitlines():
            line = line.strip()
            if not line:
                continue
            try:
                r = json.loads(line)
                recs.append((datetime.fromisoformat(r["started_at"]), r))
            except (ValueError, KeyError):
                # Do not let one bad line kill the report, but do not eat it silently either
                print(f"  ! skipping malformed line in {f.name}: {line[:60]}", file=sys.stderr)
    return recs
 
 
def reconcile(cron, log_dir, since, until):
    recs = load_records(Path(log_dir))
    unmatched = list(recs)
    missing = []
    for want in expected(cron, since, until):
        hit = next((r for r in unmatched if abs(r[0] - want) <= TOLERANCE), None)
        if hit:
            unmatched.remove(hit)      # never reuse one record for two expectations
        else:
            missing.append(want)
    return missing, unmatched, recs
 
 
if __name__ == "__main__":
    cron, log_dir = sys.argv[1], sys.argv[2]
    until = datetime.fromisoformat(sys.argv[3]) if len(sys.argv) > 3 else datetime.now()
    since = until - timedelta(days=7)
 
    missing, extra, recs = reconcile(cron, log_dir, since, until)
    by = {}
    for _, r in recs:
        by[r.get("status", "?")] = by.get(r.get("status", "?"), 0) + 1
 
    print(f"expected {len(missing) + len(recs) - len(extra)} / recorded {len(recs)} {by}")
    print(f"missing {len(missing)}")
    for m in missing[:5]:
        print("  -", m.strftime("%Y-%m-%d %H:%M"))
    if len(missing) > 5:
        print(f"  ... and {len(missing) - 5} more")
    sys.exit(1 if missing else 0)

Fed a week where only the morning runs left records, it prints:

expected 14 / recorded 7 {'ok': 6, 'skipped': 1}
missing 7
  - 2026-08-25 16:30
  - 2026-08-26 16:30
  - 2026-08-27 16:30
  - 2026-08-28 16:30
  - 2026-08-29 16:30
  ... and 2 more

Every missing timestamp lands on 16:30. Once the output looks like that, you know to stop reading the config and start looking at the launcher. Reading individual log entries one at a time never surfaces that shape.

Two implementation details are worth keeping. Remove a record from the candidate pool once it matches, or a single delayed run will satisfy several expectations and under-report the gap. And pick the tolerance deliberately: 20 minutes absorbs queueing and startup contention without reaching the neighbouring slot. If your slots sit closer together, shrink it to well under half the interval.

Retrofitting This onto a Job You Already Run

If the job is already in production, this order keeps your existing records usable throughout.

1. Start by counting, and change nothing else

Leave the record format alone and just count expected firings first. If your existing logs carry usable filenames or timestamps, swapping out load_records for that format is enough to make the reconciliation run. A large gap at this stage lets you start investigating without waiting on a logging rewrite.

2. Move the write outside the work

Then switch to the trap wrapper. Wrapping the existing job from the outside keeps the diff small and the rollback trivial. I changed both pieces at once the first time and could no longer tell whether a gap came from missing old records or from a flaw in the new wrapper.

3. Put the reconciliation at the front of your morning

Finally, run the reconciler once a day. It exits 0 when nothing is missing and non-zero otherwise, so your alerting condition is a single line: notify when this command fails. You add a new kind of monitoring without adding a new thing to watch.

The Same Reconciliation Surfaces Three Different Problems

Once expectations and records sit side by side, three distinct symptoms fall out of the same comparison.

Discrepancy	How it appears	Where to look
Missing	An expectation with no matching record	Launcher, unapplied config, host downtime
Extra	A record that matches no expectation	Duplicate registration, manual runs, timezone handling
Chronic lateness	Matches, but always near the tolerance edge	Growing runtime, slots stacked on the same minute

Extras are easy to shrug off and expensive to ignore. Two copies of the same job writing to the same destination can corrupt state quietly, which is why I treat duplicates as seriously as gaps.

Chronic lateness is a forecast rather than a fault. A slot that keeps matching at the very edge of tolerance is telling you it will cross into the next window soon, which is a good moment to split the work or move the start time.

What the Fix Turned Out to Be

The remedy on the cause side was unglamorous: if you want a job to run twice a day, register it twice rather than folding both slots into one line like 30 4,16 * * *.

This is not a claim about crontab syntax. As shown above, the syntax expands to 14 firings exactly as written. How a given execution platform handles multiple slots in one entry varies, and on my setup only one of the two was ever launched. Rather than reverse-engineer the platform, splitting into one registration per slot removes the ambiguity outright.

I kept the reconciliation in place afterwards. Knowing the cause of this gap is a different thing from being protected against the next one.

Do This Tomorrow Morning

Pick one scheduled job you own and count its expected firings over the last seven days. Then put that number next to the count of execution records.

If the two numbers disagree, something is happening outside your monitoring. If they agree, you can finally say that nothing is wrong and point at a record when you say it. Zero errors was never evidence of that.

The more automation you add, the quieter the runs that did not happen become. Collect only the records of runs that did happen, and you will keep mistaking silence for health. I did.

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.