Existence Checks Pass, Writes Fail — Probing Capabilities Before an Unattended Run

Some Places Let You Write but Never Delete

Everything so far was still within what I expected. This next result was not.

I probed a directory on a mount I use for build artifacts. Permissions read drwx------, owned by me. os.access(d, os.W_OK) returns True.

Then I measured each operation separately.

Create, append, and rename all succeed. Delete is refused with EPERM.

I sat with that for a moment. I had assumed one write probe was sufficient, and here was a real filesystem where writability and deletability are governed separately.

The operational consequence is not small. Write a temp file, do the work, clean up afterward — a very ordinary shape. On this mount, the cleanup always fails. And it fails at the very end, after the artifact has already been produced, which yields the confusing state of "everything worked, exit code 1." Re-run it and last night's debris is still sitting there when the new run starts.

If the temp file had a fixed name, the next run would pick up that debris. I have been bitten by fixed-name temp files before, and since then I give every probe file a random suffix.

One principle came out of this: a probe should not ask "can I write here?" but "does the exact sequence of operations I am about to perform succeed?"

Can Static Clues Predict It?

Once you know such a place exists, the natural next thought is whether you can spot it without touching anything. A probe is small, but it still has side effects, and reading permissions and mount metadata would be the better-mannered approach if it worked.

So I lined up every static clue I could read and checked it against what the operations actually did: the mount options in /proc/mounts, the ST_RDONLY flag from os.statvfs, the sticky bit in st_mode, ownership, and os.access(W_OK). Three subjects — the artifact directory where only delete fails, a genuinely read-only mount, and /proc.

For the read-only mount, all four static clues point the same way. It reads ro, ST_RDONLY is set, and os.access returns False. Had I looked only at that case, I might well have concluded that reading mount metadata was enough.

But the delete-only column is healthy from top to bottom. Mounted rw, ST_RDONLY clear, no sticky bit, mode 0700, owned by me, os.access(W_OK) True. Nothing readable without touching the filesystem gives you a reason to doubt it. And unlink is still refused with EPERM.

The /proc column is the one that stayed with me. Creating a file in a directory that plainly exists comes back as ENOENT — "no such file or directory." The path is right there. If that single line were all I had in a log, I would have suspected a typo first, then a race in the creation order, and reached "this is a virtual filesystem that does not accept new entries" considerably later.

That clarified what the clues are actually good for. Static information tells you what is forbidden; it never confirms what is permitted. Prohibitions are declared. Permissions stay a hypothesis until you exercise them.

In practice I keep both. The probe decides pass or fail on its own, and the static clues are gathered only when something fails, as material for the remedy line.

def diagnose(path: str) -> str:
    """Context gathered *after* a probe fails, to explain why."""
    import stat
    bits = []
    try:
        vfs = os.statvfs(path)
        if vfs.f_flag & os.ST_RDONLY:
            bits.append("read-only mount")
    except OSError as exc:
        return f"statvfs failed: {exc.strerror}"
    try:
        st = os.stat(path)
    except OSError as exc:
        return f"stat failed: {exc.strerror}"
    if st.st_mode & stat.S_ISVTX:
        bits.append("sticky bit (you cannot unlink files you do not own)")
    if st.st_uid != os.getuid():
        bits.append(f"owner uid={st.st_uid}, self uid={os.getuid()}")
    if not bits:
        # The interesting case: statically healthy, operationally not.
        bits.append("statically healthy — suspect a mount-level restriction")
    return "; ".join(bits)

Keeping diagnosis separate from the check itself matters. Fold them together and a failing stat inside the diagnostic brings down the whole check. The probe renders the verdict alone; the diagnosis is a courtesy for whoever reads the log. Reverse that order and the machinery you built to explain failures becomes a new source of them.

Implementing the Preflight

With that settled, here is the module I run immediately before the real work. Checks are declared as data; each one performs its real operation at minimal scale and carries the remedy for its own failure.

#!/usr/bin/env python3
"""Preflight capability probes for unattended runs."""
from __future__ import annotations
 
import os
import subprocess
import time
import uuid
from dataclasses import dataclass, field
from typing import Callable
 
CREATE, REPLACE, DELETE, GIT_REPO, SECRET = "create", "replace", "delete", "git", "secret"
 
 
@dataclass
class Check:
    name: str
    kind: str
    target: str
    remedy: str          # what a human should do when this fails
    ok: bool = False
    detail: str = ""
    ms: float = field(default=0.0)
 
 
def _probe_path(kind: str, path: str) -> tuple[bool, str]:
    # Unique per run. A fixed name eventually grabs the debris left by a
    # previous run that could not delete its own probe file.
    probe = os.path.join(path, f".preflight-{uuid.uuid4().hex[:8]}")
    try:
        with open(probe, "w") as fp:
            fp.write("probe")
    except OSError as exc:
        return False, f"create: {exc.strerror}"
 
    try:
        if kind in (REPLACE, DELETE):
            os.replace(probe, probe + ".tmp")
            probe = probe + ".tmp"
        if kind == DELETE:
            os.remove(probe)
            return True, "create/replace/delete all permitted"
    except OSError as exc:
        try:                      # leave behind nothing we can still remove
            os.remove(probe)
        except OSError:
            pass
        return False, f"{kind}: {exc.strerror}"
 
    try:
        os.remove(probe)
    except OSError as exc:
        # If this check never asked for delete, an unlink failure is not fatal
        return kind != DELETE, f"created, but unlink failed: {exc.strerror}"
    return True, "ok"
 
 
def _probe_git(path: str) -> tuple[bool, str]:
    try:
        res = subprocess.run(
            ["git", "-C", path, "rev-parse", "--git-dir"],
            capture_output=True, text=True, timeout=15,
        )
    except (OSError, subprocess.TimeoutExpired) as exc:
        return False, f"git unavailable: {exc}"
    if res.returncode != 0:
        return False, res.stderr.strip().splitlines()[0] if res.stderr else "not a repository"
    return True, res.stdout.strip()
 
 
def _probe_secret(spec: str) -> tuple[bool, str]:
    # spec = "<file>::<label>". Read it the way production reads it, then
    # assert the shape. An empty string here resurfaces much later as a 403.
    path, _, label = spec.partition("::")
    try:
        with open(path, encoding="utf-8") as fp:
            lines = [ln.rstrip("\n") for ln in fp]
    except OSError as exc:
        return False, f"{exc.strerror}: {path}"
    for i, line in enumerate(lines):
        if line.startswith(label) and i + 1 < len(lines):
            value = lines[i + 1].strip()
            if len(value) < 20:
                return False, f"value for {label!r} is {len(value)} chars — too short"
            return True, f"{label}: {len(value)} chars"
    return False, f"label {label!r} not found in {os.path.basename(path)}"
 
 
PROBES: dict[str, Callable[[str], tuple[bool, str]]] = {
    CREATE: lambda p: _probe_path(CREATE, p),
    REPLACE: lambda p: _probe_path(REPLACE, p),
    DELETE: lambda p: _probe_path(DELETE, p),
    GIT_REPO: _probe_git,
    SECRET: _probe_secret,
}
 
 
def run(checks: list[Check]) -> list[Check]:
    for check in checks:
        start = time.perf_counter()
        try:
            check.ok, check.detail = PROBES[check.kind](check.target)
        except KeyError:
            check.ok, check.detail = False, f"unknown probe kind: {check.kind}"
        check.ms = (time.perf_counter() - start) * 1000
    return checks
 
 
def report(checks: list[Check]) -> int:
    failed = [c for c in checks if not c.ok]
    total = sum(c.ms for c in checks)
    for c in checks:
        mark = "PASS" if c.ok else "FAIL"
        print(f"[{mark}] {c.name:<26} {c.ms:6.2f}ms  {c.detail}")
    print(f"\n{len(checks) - len(failed)}/{len(checks)} passed in {total:.2f}ms")
    for c in failed:
        print(f"  -> {c.name}: {c.remedy}")
    return 1 if failed else 0

The caller lists only the places today's run will touch.

checks = [
    Check("workspace/clone-root", DELETE,   f"{HOME}/repos",               "move to another writable location"),
    Check("artifact-dir",         DELETE,   outputs_dir,                   "do not use undeletable dirs for re-runnable output"),
    Check("repo/claudelab.net",   GIT_REPO, f"{HOME}/repos/claudelab.net", "re-clone"),
    Check("credential",           SECRET,   f"{tokens}::Claude Lab",       "check the label and the value"),
]
sys.exit(report(run(checks)))

Real output:

[PASS] workspace/clone-root         0.14ms  create/replace/delete all permitted
[FAIL] artifact-dir                 3.50ms  delete: Operation not permitted
[PASS] repo/claudelab.net           2.15ms  .git
[FAIL] locked-dir                   0.04ms  create: Permission denied
[PASS] credential                   1.71ms  Claude Lab: 40 chars
[FAIL] credential(typo)             1.14ms  label 'ClaudeLab' not found in github_tokens.txt

3/6 passed in 8.68ms
  -> artifact-dir: do not use undeletable dirs for re-runnable output
  -> locked-dir: check ownership and pick another path
  -> credential(typo): check the label and the value

Six checks in 8.68 milliseconds. Against a first stage that alone costs 93 milliseconds, that is 9.3% overhead — noise, for an indie developer running this unattended every night.

I made remedy a required field deliberately. The person reading an unattended log is almost always me, several hours later. Reconstructing the original reasoning from a bare Permission denied is real work. The moment you write the check is the moment you understand the fix best, so write it down right there.

A Probe That Reads Differently From Production Will Lie

The first version of this module rejected a perfectly valid credential.

_probe_secret originally compared with if line.strip() == label, assuming the label line read Claude Lab. The actual line is Claude Lab (claudelab.net) — the domain is appended. Production read it with grep -A1 "^Claude Lab", a prefix match, so production was fine. Only the probe, matching exactly, threw it out.

[FAIL] credential  1.65ms  label 'Claude Lab' not found in github_tokens.txt

That one stung. The mechanism I wrote to catch broken assumptions had lied to me because of an assumption of its own.

And the lie has two directions. A probe stricter than production halts a healthy environment. A probe looser than production waves a broken one through. You notice the first kind immediately. You never notice the second — and if you have thinned out your production-side checks because you trust the preflight, the loose direction is the dangerous one.

The fix is unglamorous: share the reading code between probe and production. Same function, same regular expression, same defaults. The instant they are written separately, they are free to diverge.

Put differently, a preflight should be a miniature of the real work, never a summary of it. I understood that distinction intellectually before. I understand it differently now.

In the Shell, Assignment Swallows Failure

Hardening the Python side leaves the shell entry point untouched. Consider the line that extracts the credential.

GITHUB_TOKEN=$(grep -A1 "^Claude Lab" tokens.txt | tail -1 | tr -d '[:space:]')

When the label is missing, this command still succeeds. grep returns 1, tail returns 0, and a pipeline's exit status is the status of its last command. GITHUB_TOKEN moves on as an empty string.

Clone with an empty token and you get this:

fatal: could not read Username for 'https://github.com': terminal prompts disabled

Nothing there says the token was empty. In an unattended run there is no terminal to prompt on, so "no credential" is rendered as "cannot prompt." The loudest message appears at the point furthest from the cause.

I assumed set -e would stop it. Measured on bash 5.1.16, the behavior splits:

export and local do not stop because the line's exit status becomes that of the export or local builtin itself, which is always 0. A failure that would have propagated from a bare assignment disappears when you prepend a single word.

Three practical rules follow:

1. Start with set -euo pipefail. Without pipefail, every assignment containing a pipe passes its failures through silently.
2. Never write export VAR=$(...) or local VAR=$(...). Split the assignment and the export onto two lines.
3. Inspect the value immediately after obtaining it — its length and shape, not merely its existence.

set -euo pipefail
 
token=$(grep -A1 "^Claude Lab" "$TOKENS" | tail -1 | tr -d '[:space:]') || true
if [ "${#token}" -lt 20 ]; then
  echo "preflight: credential for 'Claude Lab' is empty or too short (${#token} chars)" >&2
  exit 78            # EX_CONFIG — signal a configuration fault in the exit code
fi
export GITHUB_TOKEN="$token"

The distinct exit code exists so retry logic can branch on it. A configuration fault will not heal on the third attempt, and it deserves different handling than a transient network blip.

Where It Belongs, and What I Deliberately Skip

It belongs immediately before the run. A check you performed last night describes last night's environment. When unattended work happens in a disposable sandbox, measuring again every time is not optional.

I probe only the things that fail silently and let the run continue. These I leave out:

On disk space, one clarification. I used to check df /tmp and feel reassured. If the path you actually write to lives on a different filesystem, that number guarantees nothing — and on this machine /tmp and the working directory are indeed separate devices. Measure the path you will write to. It is the same lesson as existence versus capability, wearing different clothes.

Running Claude Code headless makes these assumptions even more mobile. As releases tighten controls around Bash execution and subagent spawning, an operation that worked yesterday may not work today. Rather than tracking every specification change, I would rather measure the capabilities my run depends on, in place, every time. It has turned out to be less work overall.

You cannot prevent the ground from shifting. You can prevent a run from continuing for twenty minutes after it has.

Closing

An existence check answers only "is it there?" Unattended runs usually break on "it is there, but you cannot use it." In my five cases, three of those states walked straight through the existence check.

If you want one concrete next step: pick a single output directory in an automation you already run, and add three lines before the real work that create a small file and delete it. That alone surfaces two of the failure classes described here, probably today. Declaring the full set of checks can come later.

Since adding this, my mornings have gotten shorter. When a run fails, it fails in a tenth of a second, with the reason and the fix on one line. That turns out to matter more than I expected.

I appreciate you staying with a topic this unglamorous.