Trusting a Three-Day-Old Mirror: Stopping Unattended Tasks from Acting on a Stale Working Copy

A sub-second freshness preflight

You don't need a heavy git fetch. git ls-remote pulls only the refs, so you can read the branch tip SHA and match it against your local SHA. While you're at it, confirm you can even write (ownership and permissions).

# preflight_clone_fresh: is the working tree writable AND equal to the remote tip?
# return: 0=fresh / 10=stale(needs pull) / 20=unwritable / 30=missing
preflight_clone_fresh() {
  local work="$1" remote_url="$2" branch="${3:-main}"
 
  # (a) does the clone exist?
  [ -d "$work/.git" ] || return 30
 
  # (b) is .git writable? (detect nobody-owned dirs / missing permissions)
  if ! ( touch "$work/.git/.wtest" 2>/dev/null && rm -f "$work/.git/.wtest" ); then
    return 20
  fi
 
  # (c) match local HEAD against the remote tip SHA
  local local_sha remote_sha
  local_sha="$(git -C "$work" rev-parse HEAD 2>/dev/null)"
  remote_sha="$(git ls-remote "$remote_url" "refs/heads/$branch" 2>/dev/null | awk '{print $1}')"
 
  # if the remote can't be read, treat "unknown" as stale and fail safe
  [ -z "$remote_sha" ] && return 10
  [ -z "$local_sha" ]  && return 10
 
  [ "$local_sha" = "$remote_sha" ] && return 0
  return 10
}

The key move is deliberately treating "unknown" — couldn't read the remote SHA, couldn't read the local SHA — as the stale side (needs sync). Optimistically calling something "latest" when you can't confirm its freshness is precisely how this incident started. When you lack proof, don't advance. That's what failing safe means here.

A self-heal flow once you detect it

Branch on the preflight's return code: catch up, or rebuild. If pull --rebase can catch you up, that's fast. If you can't write, the clone is missing, or you can't catch up, make a fresh clone under $HOME (sidestepping /tmp ownership trouble).

ensure_fresh_clone() {
  local work="$1" remote_url="$2" branch="${3:-main}"
 
  preflight_clone_fresh "$work" "$remote_url" "$branch"
  local code=$?
 
  case "$code" in
    0)
      echo "fresh (local==remote): $work"
      ;;
    10)
      echo "stale detected — catching up with pull --rebase"
      if git -C "$work" pull --rebase origin "$branch" >/dev/null 2>&1; then
        # re-match after catching up to get proof
        preflight_clone_fresh "$work" "$remote_url" "$branch" \
          && echo "synced" \
          || { echo "still mismatched after pull — re-cloning"; reclone_fresh "$work" "$remote_url" "$branch"; }
      else
        echo "pull failed — re-cloning"
        reclone_fresh "$work" "$remote_url" "$branch"
      fi
      ;;
    20|30)
      echo "unwritable/missing — fresh clone under \$HOME"
      reclone_fresh "$work" "$remote_url" "$branch"
      ;;
  esac
}
 
reclone_fresh() {
  local stale="$1" remote_url="$2" branch="${3:-main}"
  local fresh="$HOME/repos/$(basename "$stale")"
  rm -rf "$fresh" 2>/dev/null
  git clone --depth 1 --branch "$branch" "$remote_url" "$fresh" >/dev/null 2>&1
  git -C "$fresh" config user.email "you@example.com"
  git -C "$fresh" config user.name  "you"
  # everything downstream uses this fresh tree as the working copy
  export WORK="$fresh"
  echo "fresh clone ready: $fresh"
}

reclone_fresh overwrites and exports WORK on purpose: from this point on, both decision reads and writes must run against the verified tree. Keep referencing the old WORK by accident and your preflight is wasted. Swap the ground, swap the variable. It's unglamorous, but skip it and the incident comes back.

Before / After: one line before the decision

My task preamble used to read like this:

# Before: implicitly trusting the persistent clone as "latest"
WORK="/tmp/repos/claudelab.net"
cd "$WORK" && git pull --rebase origin main || true   # keeps going even on failure
# then decides "does this slug exist?" <- possibly on an old tree

|| true is the trap: if pull fails on an ownership problem, the step still advances, and the later slug check and count run on a stale tree. Insert the preflight and it becomes:

# After: guarantee freshness as a contract before the decision
WORK="/tmp/repos/claudelab.net"
REMOTE="https://github.com/you/claudelab.net.git"
ensure_fresh_clone "$WORK" "$REMOTE" main   # may swap WORK to a fresh tree
cd "$WORK"
# decision reads from here run only against a verified tree
JA=$(find content/articles/ja -name '*.mdx' | wc -l)
EN=$(find content/articles/en -name '*.mdx' | wc -l)

The diff is barely a line, but the meaning shifts entirely. You replace the implicit assumption "I could read it, so it's latest" with the explicit contract "I confirmed it's latest, so I read it."

How far to take it — balancing cost and safety

A fresh clone every run is certain, but it's 2-3x slower for large repos and eats disk. So the realistic combination is "fast persistent clone plus cheap freshness check." git ls-remote reads refs once and finishes in a few hundred milliseconds; the write probe is instant. The heavy re-clone (tens of seconds) runs only when it truly must — unwritable, or can't catch up. I recommend that asymmetry: light all the time, heavy only rarely.

Drawing the line by the weight of the decision helps too. The priority order I actually bake into tasks:

1. Before decisions that corrupt what you publish (duplicate slugs, JA/EN count parity), always run the preflight
2. Before decisions that lead straight to double work (already revised? already published this topic?), run it too
3. For harmless steps like formatting a log, skip it

I run the same design in a separate task that aggregates AdMob revenue every morning, because getting the freshness of those numbers wrong skews the whole day's decision. Rather than guarding everything at maximum strength, concentrate verification where breakage hurts; that's what keeps a long-running operation from burning out.

One thing you can do tomorrow: add a single local HEAD == remote HEAD match to the top of an unattended task you already run. Since I added that one line, the particular dread of "it says success but the contents are old" has eased a great deal. I'd be glad if we could keep closing these quiet failures one at a time, together.