One Transient 401 Replaced My Long-Lived Token — Giving Credentials a Provenance Field

Sixteen workers, and the first surprise

With the 401 probability at 2%, I ran 16 workers for 200 iterations each.

This is the result that made me sit with the data for a while.

Adding flock takes torn reads from 25 down to 0. The number of contaminated workers stays at 16 out of 16 — not a single one saved.

I had honestly expected locking to cut the damage roughly in half. What it actually fixed was only the cosmetic failure mode, where a reader catches the file mid-write. The semantic overwrite — last writer wins — proceeds just as thoroughly inside the lock, only now in an orderly fashion.

The uncomfortable part is what happens to observability. With torn reads at 0, the one visible symptom disappears. Logs get cleaner after you add locking. You gain logs that look healthy while the running token stays wrong.

Varying the 401 rate showed that rarity offers no protection either.

An event that fires once in a thousand iterations still contaminates 7 of 16 workers, while producing just 2 torn reads. The amount of anomaly your monitoring can catch and the amount of damage actually done are wildly out of proportion.

Injecting exactly one 401

Probabilistic runs make it impossible to attribute damage to a specific failure. So I switched to a deterministic setup: only worker 0, only at iteration 20, hits exactly one 401. Nobody else fails at all.

To stay close to a real fleet, workers also re-read the credential every 30 iterations, standing in for new processes starting up.

INJECT_AT = 20            # the single transient 401, on worker 0 only
 
if wid == 0 and it == INJECT_AT:
    new = {"token": "st_x", "kind": "session", "src": "saved_login"}
    with_lock(path, lambda: write_atomic(path, new))
    cached = None
if it % 30 == 29:         # stands in for a fresh process starting up
    cached = None

The results:

One transient error reached all 24 processes within nine iterations. What propagated was not the failure itself. It was the well-intentioned cleanup that followed it.

That finally explained the three unrelated jobs in my morning logs. Job one really did just have a momentary failure. What stopped the other three was the recovery credential job one had helpfully written back.

Giving the credential a provenance field

The fix turned out to be neither locking nor retries. It is simply looking at where the current value came from before writing over it.

def refresh_with_provenance(path, new_token):
    """A refresh that respects provenance.
    A long-lived credential supplied via env is never overwritten by a recovery path."""
    def guarded():
        cur = read_naive(path)
        if cur.get("src") == "env" and cur.get("kind") == "long_lived":
            # The operator handed us this one explicitly. Park the recovery token beside it.
            write_atomic(path + ".session", new_token)
            return "sidecar"
        write_atomic(path, new_token)
        return "replaced"
    return with_lock(path, guarded)

It is one additional if. The property it buys is real, though: every path that can write now decides for itself what it is allowed to write over.

The part worth getting right is not throwing the short-lived token away. Discard it and you cannot recover when a refresh genuinely is needed. Parking it in a .session sidecar next to the long-lived credential, with a fixed resolution order of "env long-lived first, sidecar short-lived second," lets both coexist.

Measured: with the guard, 24 workers, one injected 401 — zero contamination, zero runs on the wrong credential. The 16-worker, 200-iteration, 2% run also came back at zero.

For src I write the resolution path itself. I settled on four values.

A read-only mount is the strongest option when it is available — the filesystem does the job of the if statement. That assumes containers, though, so for automation running on a plain machine the provenance field has been the more practical choice.

How far to take isolation

The other answer is to stop sharing at all. Give each session its own config directory and nobody can overwrite anyone else's credential.

But a config directory holds more than a token. Session history and caches live there too. I built a 2.89 MB directory modeled on my own setup and compared a full copy against keeping only the credential real and symlinking the rest.

That is 24× faster and 650× smaller on disk.

import os, shutil
 
def isolated_config(src, dst):
    """Only the credential gets a real copy; everything else points back to the shared source."""
    os.makedirs(dst, exist_ok=True)
    for entry in os.listdir(src):
        if entry == "credentials.json":
            shutil.copy2(os.path.join(src, entry), os.path.join(dst, entry))
        else:
            os.symlink(os.path.abspath(os.path.join(src, entry)),
                       os.path.join(dst, entry))
    return dst

The dividing line is simply which files get written to. There is no reason to duplicate history and caches that are only ever read. At 142 kB for 32 sessions, per-session isolation is cheap enough to leave on permanently.

I would layer isolation and the provenance guard rather than choosing between them. Isolation prevents sharing the slot; the guard decides what may be written in slots you have no choice but to share. Headless runs still hit cases where the config directory has to be shared. I applied the same line of thinking to secrets on the sandbox side in the boundary where sandbox credential masking actually holds.

Re-reading faster is not prevention

There was one more idea worth testing.

If in-process caching is the problem, surely re-reading more often fixes it. Grab a bad credential, re-read shortly after, pick up the repaired value.

I injected contamination at iteration 10, had the operator repair the file by hand at iteration 40, and varied only the cache reload interval. 16 workers, 120 iterations.

Two results came back opposite to what I expected.

First: shrinking the interval sixfold, from 30 to 5, does not reduce total damage at all (480 → 500, actually slightly worse). Reading more often means picking up the contamination sooner, so total time spent holding a bad credential barely moves. What did shrink was the tail after the repair: 301 → 36. The reload interval is a recovery parameter, not a prevention parameter.

Second: the smallest total damage came from never reloading — 30 runs, 1 worker. A process that never re-reads never receives the contamination. It carries the correct credential it grabbed at startup all the way to the end.

That cannot become a design principle, though. "Safe because it never re-reads" is riding on the luck of having started before the contamination landed. Any process that starts afterward holds the wrong credential for its entire life. Fittingly, the one worker that stayed broken in that row is worker 0 — the one that wrote the contamination itself. "Broken until restart" in the release notes is exactly this property.

So no amount of cache tuning gets you prevention. Only the provenance guard (zero) and isolation (zero) did.

A startup canary that costs 24 microseconds

The last piece I added just writes one line to the log before each run, stating which token is about to be used.

import json, hashlib
 
def credential_canary(path):
    """Returns a fingerprint and provenance. Never the token itself."""
    o = json.load(open(path))
    fp = hashlib.sha256(o["token"].encode()).hexdigest()[:12]
    return {"fingerprint": fp, "kind": o.get("kind"), "src": o.get("src")}
 
# Call before the run; refuse to start if it drifted
c = credential_canary(CRED_PATH)
if c["kind"] != "long_lived" or c["src"] != "env":
    raise SystemExit(f"credential drifted: {c}")   # e.g. eb031743498b / long_lived / env

Over 5,000 measurements: median 24.1 µs, p95 26.9 µs, p99 53.7 µs. A thousand startups add up to 24.1 ms. For scale, acquiring and releasing a flock alone costs a median of 16.5 µs on the same machine — the canary's real overhead is the same order as one lock.

Only the first twelve hex digits go into the log. The token itself never appears in logs or error messages. A change in fingerprint is still enough to tell you a different token was used today than yesterday.

The broader thing this exercise taught me is that credential incidents are rarely about a value being wrong. They are about a value quietly becoming a different value. Correctness you can verify. A substitution you cannot even detect unless you were recording.

Looking for the same shape in your own setup

Everything so far came from my own model. Whether the same shape lurks in your automation is something an audit finds faster than any measurement.

There is one thing to check: do separate jobs read, as their credential, the same writable file in reality? Paths that look different often resolve to one file once you follow the symlinks.

# shared_slot_audit.py — find writable credential files shared across jobs
import os, collections
 
def shared_writable_slots(jobs):
    """jobs: {job name: [paths read as credentials, ...]}
    Resolve symlinks and return real files writable from two or more jobs."""
    slots = collections.defaultdict(set)
    for job, paths in jobs.items():
        for p in paths:
            real = os.path.realpath(os.path.expanduser(p))
            if os.path.isfile(real) and os.access(real, os.W_OK):
                slots[real].add(job)
    return {p: sorted(j) for p, j in slots.items() if len(j) > 1}

In a small check with three jobs, where a/c.json and b/c.json are both symlinks to one shared file, it reported:

shared/credentials.json ['nightly-a', 'nightly-b']
b/own.json ['nightly-b', 'nightly-c']

The first line is the shape of my own setup that morning: separate-looking files, one real file underneath. The second is a plain share without symlinks, but with write access it carries the same risk.

Empty output means this kind of sharing is not there, at least. Even one line gives you something to decide on — provenance guard or isolation first. I rank by the number of paths holding write access, not by the number of jobs.

What to add before tonight's run

This model has clear limits. It covers processes on a single host, with no network and no real authorization server. Crossing filesystems, or going through a shared container volume, would need its own round of measurement.

With that said, here is the order I would work through, ranked by measured impact.

1. Add a src check to every write-back path. One if statement, 24 microseconds measured. There is no reason to wait on anything else before doing this one. In my own automation, it alone closed the route by which one transient error reached 24 processes.
2. Log a startup canary line. Fingerprint, kind, src — nothing more. It prevents nothing, but it lets you establish after the fact that a substitution occurred.
3. Make only the written files real, per session. At 142 kB for 32 sessions you can leave it on permanently. There is no need to duplicate the whole config directory.

Isolation and canaries can come after step one. In terms of order of impact, what mattered first was making the paths that hold destructive power ask whether they are allowed to use it.