Resuming After a Budget Stop: What 36 Enumerated Stop Points Taught Me

A naive resume accepted a 100,006-byte artifact as finished

I ran naive mode through all 36 caps, each time doing budget stop, then resume with no cap, then verify.

Opening one of those eight made the weight of it clear. The run stopped at cap 3.0 left this behind:

-rw-r--r-- 1 ... 100006 alpha.md      <- should be 200013 bytes
-rw-r--r-- 1 ... 200013 bravo.md
-rw-r--r-- 1 ... 200015 charlie.md
-rw-r--r-- 1 ... 200013 delta.md
 
index.json:
{"items": [{"name": "bravo", ...}, {"name": "charlie", ...}, {"name": "delta", ...}]}

alpha.md is 100,006 bytes — exactly half of the expected 200,013 — yet it exists, so the resume skipped it. Because it was skipped, it never reached the index either. A half-written artifact simply stays on disk forever.

That was the truncated text I had found in the morning.

The danger is not the moment of the stop. It is the moment a later run misreads the partial state as complete. Introducing a budget cap means manufacturing that opportunity on a schedule, at positions the designer never chose.

Atomic writes and index-first resume

The remedy is a classic combination.

1. Write artifacts to .part, fsync, then os.replace into the final name. Within one filesystem the swap is atomic, so a half-written file never appears under the real name.
2. Sweep leftover .part files at startup so the previous interruption cannot leak into this run.
3. Base completion on the index, never on file presence. Update the index through the same atomic replace.
4. Append to the index after the artifact swap succeeds. The reverse order produces entries that are recorded but absent.

With safe mode, all 36 stop points resumed into a healthy final state.

The cost question remained. Extra fsync calls and a rename are not free. I ran a 200-item pipeline five times per mode and averaged.

That is a 29.1% difference, or 0.78 ms per item. When the real cost of each item is a model call measured in seconds, this vanishes into noise. I found no reason to trade away atomicity.

So far, everything matched expectations. The surprises came next.

Two results that ran against my intuition

Atomicity created a retry loop that burns budget forever

Having made the pipeline safe, wrapping it in "retry until complete" should have finished the job. I set the cap to 3.0 and ran both modes to completion.

Against an ideal of 18.0, safe spent 42.0 and produced nothing. At cap 3.5 it climbed to 48.0.

The reason is arithmetic. One item needs 4.5 units, and a cap of 3.0 never reaches the first commit. Because an atomic design refuses to commit partial work, it unwinds cleanly and stops in exactly the same place every time. Every retry buys zero progress.

The irony is that the mode which happily ships corrupt output was the cheaper failure.

What was missing was not atomicity but progress observation. Compare committed counts before and after each attempt, and bail the moment the count fails to move.

#!/usr/bin/env bash
# Progress-gated retry: bail out before burning budget on a non-advancing attempt
set -u
MODE=$1; CAP=$2; MAXATT=${3:-12}; D=$4
 
progress () {
  python3 - "$D" <<'PY'
import json, pathlib, sys
p = pathlib.Path(sys.argv[1]) / "index.json"
print(len(json.loads(p.read_text())["items"]) if p.exists() else 0)
PY
}
 
ATT=0; STALL=0
while [ "$ATT" -lt "$MAXATT" ]; do
  BEFORE=$(progress); ATT=$((ATT + 1))
  python3 pipeline.py "$D" "$CAP" "$MODE" >/dev/null 2>&1 && break
  AFTER=$(progress)
  if [ "$AFTER" = "$BEFORE" ]; then
    STALL=1                      # the cap sits below the cost of one unit of work
    echo "no forward progress after attempt ${ATT}: raise --max-budget-usd" >&2
    break
  fi
done
exit "$STALL"

With the gate in place, cap 3.0 bails after a single attempt having spent 3.5 units. 42.0 became 3.5. A budget cap and a progress gate are two halves of one part. Ship only the first half and the safety mechanism turns into a spending mechanism.

Total spend is not monotonic in the cap

With the gate active, I swept the cap from 4.0 to 12.0 in 0.5 steps and recorded total spend to completion. The ideal is 18.0.

Raising the cap from 8.0 to 8.5 — a change of 0.5 — dropped total spend from 30.0 to 18.0. A 40% reduction from loosening the budget.

Tracing it explained the shape. This pipeline charges immediately after committing an artifact. At cap 8.0 the threshold is crossed just before the second commit, so a full item's generation cost is discarded, and that loss repeats on every attempt. At cap 8.5 the commit lands first, so nothing is thrown away.

Waste, in other words, is determined by where the cap falls relative to internal charge points you cannot see from outside. There is no reliable way to know the granularity at which a real model call accrues cost. Which means the whole idea of calculating an optimal cap on paper was mistaken.

Apply the cap to the smallest resumable unit, not to the batch

The answer was not to choose the cap more cleverly. It was to change what the cap applies to.

A cap lands mid-work because one invocation processes the entire batch. Limit each invocation to a single item, and the cap can only ever land on a seam.

PER_RUN = int(os.environ.get("PER_RUN", "0"))   # 0 means unlimited
committed = 0
for name in ITEMS:
    if PER_RUN and committed >= PER_RUN:
        break                                    # exit normally after one commit
    ...
    write_index(items)
    committed += 1
    charge(0.5)

I re-measured across caps from 4.5 to 12.0.

Waste is 0% regardless of the cap value. The tuning exercise disappears entirely.

There is a useful side effect. Each invocation is short, so the cap changes meaning: it stops being a runaway brake and becomes a per-item cost ceiling. When one item demands three times its usual cost, you drop that item instead of the whole batch — and you get a signal pointing at the input responsible.

I settled on this shape. Running unattended overnight as an indie developer, having exactly one question to answer in the morning — how far did it get — is worth a great deal.

The price of one item per invocation was a 9.3x startup tax

This shape is not free. What used to be one process handling the whole batch becomes one process per item.

In the same rig I measured 200 items processed by a single process against 200 items processed one per process.

That is 9.3x. The extra time is interpreter startup and initialization.

When a stage waits seconds on a model call, 27.3 ms disappears entirely. My overnight job showed no perceptible difference.

Some setups are not so forgiving. If every start re-reads a large index, relaunches an MCP server, or re-acquires credentials, initialization creeps toward the cost of generating an item, and one-item invocations stop paying for themselves.

In that case, do not pin PER_RUN to 1 — use k. Choosing k is simple: pick a value that guarantees each invocation ends immediately after a commit. Then set the cap to "worst case for k items plus one item of headroom."

# Process k items per invocation; cap = k x worst-case + one item of headroom
PER_RUN=4
WORST_PER_ITEM=0.35            # highest per-item cost over the last 30 days -- not the average
CAP=$(python3 -c "print(round($PER_RUN * $WORST_PER_ITEM + $WORST_PER_ITEM, 2))")
 
PER_RUN=$PER_RUN claude -p "$PROMPT" --max-budget-usd "$CAP"

Using the maximum rather than the average for WORST_PER_ITEM is the part that matters. With an average, the runs that hit a heavy input land the cap in the middle of generation — and the non-monotonic waste comes straight back.

The moment I ran them in parallel, the index stopped being the single source of truth

Having split the work into single items, running them concurrently should have recovered the startup tax. I launched 24 at once.

Entries vanished from the index.

Across 24 workers times 5 trials — 120 items — 97 records fell out. Every artifact file was written correctly. Only the index lost them.

The cause sits outside atomic replacement. os.replace makes the swap atomic, but it does not protect the full read-modify-write sequence. Twenty-four processes read the same stale index, each appends its own single entry, and only the last writer survives.

The first row is a worse failure. With the temp file pinned to .part, one process would have its temp file taken by another before it could call os.replace, producing 40 exceptions — and some trials read an empty index. I believed I was writing atomically while concurrency quietly invalidated the premise of atomicity.

What makes the second row frightening is the zero in the exceptions column. Nothing lands in the log. You find out the next morning when the counts do not add up. The same quality of silence as the truncated text I opened at the start of this piece.

Two fixes worked, and both produced zero lost records.

Serializing writes with flock keeps the existing index structure intact. It cost 453 ms on average at 24 workers — roughly twice the marker approach below. The difference is time spent waiting on the lock.

The other fix drops the single index in favor of one marker file per item. Writes never collide, so no lock is needed. Same 24 workers: 231 ms.

def commit(root, name, artifact_bytes):
    """One marker file per item. Writes never collide, so no lock is required."""
    done = root / "done"
    done.mkdir(exist_ok=True)
    p = done / f"{name}.json"
    t = p.with_name(p.name + f".{os.getpid()}.part")   # temp name must be per-process
    with open(t, "w") as f:
        json.dump({"name": name, "bytes": artifact_bytes}, f)
        f.flush(); os.fsync(f.fileno())
    os.replace(t, p)
 
def completed(root):
    """On resume, gather the markers into a completion set."""
    d = root / "done"
    if not d.is_dir():
        return set()
    return {json.loads(p.read_text())["name"] for p in d.glob("*.json")}

If you still want a single index file, rebuild it from the markers. Rebuilding 1,000 entries took 32.1 ms — cheap enough to run once after the parallel phase finishes.

Making temp filenames per-process was required either way. The atomicity of os.replace assumes the source you are swapping in belongs only to you. Sequential runs satisfy that assumption for free, which is why it breaks the instant you go parallel.

A checklist before you adopt this

If you are assembling something similar, checking in this order should spare you the holes I fell into.

1. Are artifact writes atomic? Temp file, fsync, os.replace. No direct writes to the final path left anywhere.
2. Is completion decided by the index? Not by file presence or size. Presence checks will happily accept a half-written file.
3. Is the commit-then-record order correct? Append to the index only after the artifact swap succeeds. The reverse creates entries with no file behind them.
4. Do you sweep partials at startup? Delete leftover .part files before doing any work.
5. Is there a progress gate? Detect a non-advancing attempt and bail. Without it, the cap becomes a spending mechanism.
6. Does the cap apply to the smallest resumable unit? One item per invocation, not the whole batch.
7. Is the exit code interpreted correctly? A budget stop must not fall into a branch that retries transient failures unconditionally. The same cap reproduces the same stop.
8. Are temp filenames per-process? If concurrency is on the table, a fixed .part name breaks the premise atomic replacement rests on.
9. Does completion recording survive concurrency? No read-modify-write against a single index. Go with flock or one marker file per item.

Of those nine, I had two — items 1 and 3 — before I started. The rest only became visible once I enumerated the stop points. Items 8 and 9 never surface at all as long as you stay sequential.

When we build something that runs unattended, we naturally think about what to do when it fails. But the halt a budget cap produces is not a failure. It is a quiet seam, placed by correct behavior, somewhere entirely unrelated to the boundaries we drew.

Counting those seams seems to be the only place the work can start. If this saves even one person the morning I had, it will have been worth writing.