Articles/Claude Code

⟐ Claude Code/2026-08-09Advanced

One Transient 401 Replaced My Long-Lived Token — Giving Credentials a Provenance Field

A shared credential file can lose its long-lived token to a single transient 401. I injected exactly one 401 into a 24-worker fleet, measured the blast radius, and compared a provenance guard against full isolation.

Claude Code²¹⁴ headless¹⁴ credentials³ OAuth⁴ automation¹⁰¹

✦ Premium Article

Most of what I build as an indie developer runs on overnight schedules, so checking last night's runs is the first thing I do in the morning.

That particular morning, all four jobs had stopped at the same place. The first one had hit an auth error. The other three had never succeeded even once, right from startup.

Running the same command by hand worked immediately. Re-running the jobs worked too. Digging back through the logs, the first job had hit a transient 401 — a brief network hiccup, or something specific to that moment. That part I expected.

What did not add up was that one job's failure had dragged down three others that should have been completely independent.

A single line in the release notes stopped me

The Claude Code updates from August 8th and 9th include this fix:

Fixed an issue where a transient 401 could cause a long-lived CLAUDE_CODE_OAUTH_TOKEN to be replaced by a short-lived token from a saved login, leaving headless sessions broken until restart.

The shape of it matched my logs exactly. Since it is fixed, that specific path needs no further attention from me.

What kept me from moving on was that this reads less like one bug and more like a shape of failure. A long-lived credential and a short-lived one share the same storage slot. The path that refreshes one of them can silently overwrite the other. That structure almost certainly still exists in several places in my own automation.

So rather than speculating about Claude Code's internals, I built a minimal model that isolates only this shape of failure and turned the blast radius into numbers. Every measurement below comes from that model of mine, not from any reproduction of Anthropic's implementation. What I am measuring is a property of shared, mutable credential storage in general.

The test environment: Linux 6.8.0, Python 3.10.12, 4 vCPUs, in a sandbox.

Building the smallest version of the failure

Three parts turned out to be enough. One file holding the credential. Several worker processes that read and use it. And a refresh path that fires when a 401 shows up.

The credential carries a src field from the start. It does nothing yet — it becomes the basis for the guard later on.

# store.py — a minimal credential store
import json, os, time, fcntl, tempfile
 
def write_naive(path, obj):
    """A non-atomic write, split in two to reproduce the real crash window."""
    with open(path, "w") as f:
        s = json.dumps(obj)
        half = len(s) // 2
        f.write(s[:half]); f.flush()
        time.sleep(0.0008)          # a reader landing here gets broken JSON
        f.write(s[half:]); f.flush()
 
def read_naive(path):
    with open(path) as f:
        raw = f.read()
    return json.loads(raw)          # raises ValueError on a half-written file
 
def write_atomic(path, obj):
    """Write into the same directory, then rename. rename is atomic on POSIX."""
    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(path))
    with os.fdopen(fd, "w") as f:
        json.dump(obj, f); f.flush(); os.fsync(f.fileno())
    os.replace(tmp, path)
 
def with_lock(path, fn):
    """Exclusive access via flock. The lock file lives beside the credential."""
    with open(path + ".lock", "a+") as lf:
        fcntl.flock(lf, fcntl.LOCK_EX)
        try:
            return fn()
        finally:
            fcntl.flock(lf, fcntl.LOCK_UN)

I made the workers hold the credential in process memory once they read it, the way a real headless session would. A process that re-reads the file on every single request would be an odd thing to build.

def worker(wid, q):
    path = os.path.join(BASE, "credentials.json")
    cached = None; used_session = 0; poisoned_at = None
    for it in range(ITERS):
        if cached is None:
            try:
                cached = with_lock(path, lambda: read_naive(path))
            except Exception:
                torn += 1; time.sleep(0.001); continue
        if cached.get("kind") != "long_lived":
            used_session += 1
            if poisoned_at is None:
                poisoned_at = it        # which iteration contamination arrived
        if random.random() < P401:      # transient 401 -> refresh path
            new = {"token": session_token(wid), "kind": "session", "src": "saved_login"}
            with_lock(path, lambda: write_atomic(path, new))
            cached = None               # we just wrote, so re-read next time
        time.sleep(0.0005)

A worker that finishes with kind still set to long_lived is healthy. If it flips to session partway through, that worker is running on a credential the operator never intended it to use.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦Watch a single transient 401 contaminate all 24 workers in a fleet within ten iterations, with measured numbers at every step

✦Understand why adding flock takes torn reads from 25 to 0 while leaving the overwrite damage at 16 out of 16, unchanged

✦Get a provenance guard that drops contamination to zero, plus an isolation pattern that cuts per-session cost from 4.7 ms and 2.9 MB down to 0.2 ms and 4.5 kB

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Sixteen workers, and the first surprise

With the 401 probability at 2%, I ran 16 workers for 200 iterations each.

Approach	Torn reads	Workers contaminated	Runs on the wrong token	Final state
Naive read/write	25	16 / 16	2,919	session / saved_login
flock + atomic write	0	16 / 16	2,874	session / saved_login

This is the result that made me sit with the data for a while.

Adding flock takes torn reads from 25 down to 0. The number of contaminated workers stays at 16 out of 16 — not a single one saved.

I had honestly expected locking to cut the damage roughly in half. What it actually fixed was only the cosmetic failure mode, where a reader catches the file mid-write. The semantic overwrite — last writer wins — proceeds just as thoroughly inside the lock, only now in an orderly fashion.

The uncomfortable part is what happens to observability. With torn reads at 0, the one visible symptom disappears. Logs get cleaner after you add locking. You gain logs that look healthy while the running token stays wrong.

Varying the 401 rate showed that rarity offers no protection either.

401 probability	Torn reads	Workers contaminated	Runs on the wrong token
0.1%	2	7 / 16	1,295
0.5%	3	12 / 16	1,740
2%	18	16 / 16	2,920
10%	256	16 / 16	3,160

An event that fires once in a thousand iterations still contaminates 7 of 16 workers, while producing just 2 torn reads. The amount of anomaly your monitoring can catch and the amount of damage actually done are wildly out of proportion.

Injecting exactly one 401

Probabilistic runs make it impossible to attribute damage to a specific failure. So I switched to a deterministic setup: only worker 0, only at iteration 20, hits exactly one 401. Nobody else fails at all.

To stay close to a real fleet, workers also re-read the credential every 30 iterations, standing in for new processes starting up.

INJECT_AT = 20            # the single transient 401, on worker 0 only
 
if wid == 0 and it == INJECT_AT:
    new = {"token": "st_x", "kind": "session", "src": "saved_login"}
    with_lock(path, lambda: write_atomic(path, new))
    cached = None
if it % 30 == 29:         # stands in for a fresh process starting up
    cached = None

The results:

Condition	401s injected	Workers contaminated	First	Last	Runs on the wrong credential
No guard	1	24 / 24	iter 21	iter 30	2,889
Provenance guard	1	0 / 24	—	—	0

One transient error reached all 24 processes within nine iterations. What propagated was not the failure itself. It was the well-intentioned cleanup that followed it.

That finally explained the three unrelated jobs in my morning logs. Job one really did just have a momentary failure. What stopped the other three was the recovery credential job one had helpfully written back.

Giving the credential a provenance field

The fix turned out to be neither locking nor retries. It is simply looking at where the current value came from before writing over it.

def refresh_with_provenance(path, new_token):
    """A refresh that respects provenance.
    A long-lived credential supplied via env is never overwritten by a recovery path."""
    def guarded():
        cur = read_naive(path)
        if cur.get("src") == "env" and cur.get("kind") == "long_lived":
            # The operator handed us this one explicitly. Park the recovery token beside it.
            write_atomic(path + ".session", new_token)
            return "sidecar"
        write_atomic(path, new_token)
        return "replaced"
    return with_lock(path, guarded)

It is one additional if. The property it buys is real, though: every path that can write now decides for itself what it is allowed to write over.

The part worth getting right is not throwing the short-lived token away. Discard it and you cannot recover when a refresh genuinely is needed. Parking it in a .session sidecar next to the long-lived credential, with a fixed resolution order of "env long-lived first, sidecar short-lived second," lets both coexist.

Measured: with the guard, 24 workers, one injected 401 — zero contamination, zero runs on the wrong credential. The 16-worker, 200-iteration, 2% run also came back at zero.

For src I write the resolution path itself. I settled on four values.

`src` value	Meaning	Overwritable
`env`	Supplied explicitly by the operator via environment variable	No — goes to sidecar
`mount`	Mounted read-only	No — not writable at all
`saved_login`	Persisted by an interactive login	Yes
`refresh`	Issued by a refresh path	Yes

A read-only mount is the strongest option when it is available — the filesystem does the job of the if statement. That assumes containers, though, so for automation running on a plain machine the provenance field has been the more practical choice.

How far to take isolation

The other answer is to stop sharing at all. Give each session its own config directory and nobody can overwrite anyone else's credential.

But a config directory holds more than a token. Session history and caches live there too. I built a 2.89 MB directory modeled on my own setup and compared a full copy against keeping only the credential real and symlinking the rest.

Approach	8 sessions	32 sessions	Per session	Disk at 32
Full copy with `cp -a`	38.2 ms	151.1 ms	4.7 ms / 2.9 MB	92.5 MB
Credential real, rest symlinked	1.3 ms	4.9 ms	0.2 ms / 4.5 kB	142.4 kB

That is 24× faster and 650× smaller on disk.

import os, shutil
 
def isolated_config(src, dst):
    """Only the credential gets a real copy; everything else points back to the shared source."""
    os.makedirs(dst, exist_ok=True)
    for entry in os.listdir(src):
        if entry == "credentials.json":
            shutil.copy2(os.path.join(src, entry), os.path.join(dst, entry))
        else:
            os.symlink(os.path.abspath(os.path.join(src, entry)),
                       os.path.join(dst, entry))
    return dst

The dividing line is simply which files get written to. There is no reason to duplicate history and caches that are only ever read. At 142 kB for 32 sessions, per-session isolation is cheap enough to leave on permanently.

I would layer isolation and the provenance guard rather than choosing between them. Isolation prevents sharing the slot; the guard decides what may be written in slots you have no choice but to share. Headless runs still hit cases where the config directory has to be shared. I applied the same line of thinking to secrets on the sandbox side in the boundary where sandbox credential masking actually holds.

Re-reading faster is not prevention

There was one more idea worth testing.

If in-process caching is the problem, surely re-reading more often fixes it. Grab a bad credential, re-read shortly after, pick up the repaired value.

I injected contamination at iteration 10, had the operator repair the file by hand at iteration 40, and varied only the cache reload interval. 16 workers, 120 iterations.

Reload interval	Runs on the wrong credential	After the repair	Workers still broken after repair
Never reload	30	1	1 / 16
Every 30 iterations	480	301	16 / 16
Every 5 iterations	500	36	8 / 16

Two results came back opposite to what I expected.

First: shrinking the interval sixfold, from 30 to 5, does not reduce total damage at all (480 → 500, actually slightly worse). Reading more often means picking up the contamination sooner, so total time spent holding a bad credential barely moves. What did shrink was the tail after the repair: 301 → 36. The reload interval is a recovery parameter, not a prevention parameter.

Second: the smallest total damage came from never reloading — 30 runs, 1 worker. A process that never re-reads never receives the contamination. It carries the correct credential it grabbed at startup all the way to the end.

That cannot become a design principle, though. "Safe because it never re-reads" is riding on the luck of having started before the contamination landed. Any process that starts afterward holds the wrong credential for its entire life. Fittingly, the one worker that stayed broken in that row is worker 0 — the one that wrote the contamination itself. "Broken until restart" in the release notes is exactly this property.

So no amount of cache tuning gets you prevention. Only the provenance guard (zero) and isolation (zero) did.

A startup canary that costs 24 microseconds

The last piece I added just writes one line to the log before each run, stating which token is about to be used.

import json, hashlib
 
def credential_canary(path):
    """Returns a fingerprint and provenance. Never the token itself."""
    o = json.load(open(path))
    fp = hashlib.sha256(o["token"].encode()).hexdigest()[:12]
    return {"fingerprint": fp, "kind": o.get("kind"), "src": o.get("src")}
 
# Call before the run; refuse to start if it drifted
c = credential_canary(CRED_PATH)
if c["kind"] != "long_lived" or c["src"] != "env":
    raise SystemExit(f"credential drifted: {c}")   # e.g. eb031743498b / long_lived / env

Over 5,000 measurements: median 24.1 µs, p95 26.9 µs, p99 53.7 µs. A thousand startups add up to 24.1 ms. For scale, acquiring and releasing a flock alone costs a median of 16.5 µs on the same machine — the canary's real overhead is the same order as one lock.

Only the first twelve hex digits go into the log. The token itself never appears in logs or error messages. A change in fingerprint is still enough to tell you a different token was used today than yesterday.

The broader thing this exercise taught me is that credential incidents are rarely about a value being wrong. They are about a value quietly becoming a different value. Correctness you can verify. A substitution you cannot even detect unless you were recording.

What to add before tonight's run

This model has clear limits. It covers processes on a single host, with no network and no real authorization server. Crossing filesystems, or going through a shared container volume, would need its own round of measurement.

With that said, here is the order I would work through, ranked by measured impact.

Add a src check to every write-back path. One if statement, 24 microseconds measured. There is no reason to wait on anything else before doing this one. In my own automation, it alone closed the route by which one transient error reached 24 processes.
Log a startup canary line. Fingerprint, kind, src — nothing more. It prevents nothing, but it lets you establish after the fact that a substitution occurred.
Make only the written files real, per session. At 142 kB for 32 sessions you can leave it on permanently. There is no need to duplicate the whole config directory.

Isolation and canaries can come after step one. In terms of order of impact, what mattered first was making the paths that hold destructive power ask whether they are allowed to use it.

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.