●2.1.301 — Adds the vim % motion, LaTeX math rendering in the VS Code panel, and a retry-watchdog limit for usage-limit waits (10/15)●2.1.299 — Conversations over 200K tokens left idle about 54 minutes get compacted; MCP allow rules now match only the server they name●SONNET 4.5 — Retires on the Claude API on 11/30, 45 days left. Migrate to Sonnet 5.5●WINDOWS — Reports say hooks break silently only on Japanese-locale Windows; check the character encoding first●FAST MODE — Opus 5.5 fast mode is out of research preview and no longer needs the beta header (10/15)●NEW — I Put a Ten-Line Promise File at the To…●2.1.301 — Adds the vim % motion, LaTeX math rendering in the VS Code panel, and a retry-watchdog limit for usage-limit waits (10/15)●2.1.299 — Conversations over 200K tokens left idle about 54 minutes get compacted; MCP allow rules now match only the server they name●SONNET 4.5 — Retires on the Claude API on 11/30, 45 days left. Migrate to Sonnet 5.5●WINDOWS — Reports say hooks break silently only on Japanese-locale Windows; check the character encoding first●FAST MODE — Opus 5.5 fast mode is out of research preview and no longer needs the beta header (10/15)●NEW — I Put a Ten-Line Promise File at the To…
Same Output, Different Path — Guarding Agent Trajectories with Invariants
When the default model changes, your final output can stay correct while the path your agent takes quietly shifts. Here is a trajectory regression harness built on recorded tool traces and deterministic invariants, with working code and measured numbers.
My nightly batch looked fine. The articles it generated were correct. The logs it posted were correct. Nothing was broken.
The billing noticed before I did. Tool calls per day had climbed to roughly 1.6x their previous level. The answers were right. The path to those answers had changed.
If you only watch output quality, this kind of change stays invisible forever. The output evaluation I described in detecting quality regressions with an eval harness measures the final string. It does not measure the trajectory.
Running unattended agents as an indie developer, this quiet shift in the path turned out to be the hard one. Nothing breaks, so nothing alerts you. And by the time you notice, either cost or wall-clock time has already grown.
What actually needs to be recorded
Record the process, not the final answer. In practice, I found five fields sufficient.
Recorded field
What it protects against
Sequence of tool names
Reordering, or an unexpected tool appearing
Normalized arguments per call
Repeated calls with identical arguments
Turn count and total tool calls
Loop inflation
Sequence of stop_reason values
Truncation or pauses slipping in
Count of write-tool invocations
Duplicated side effects
The important part is that you do not store raw arguments. Timestamps and UUIDs make every trace unique, which makes comparison useless. Strip the volatile values before recording, and the trajectory becomes a stable object you can diff.
A thin wrapper that records the trace
Rather than reaching into the agent loop, wrap only the tool dispatch boundary. Your production loop stays untouched, which keeps the tested code and the shipped code from drifting apart.
# trace.pyfrom __future__ import annotationsimport hashlibimport jsonimport refrom dataclasses import dataclass, field, asdictfrom typing import Any, Callable# Values that change on every run must be dropped before comparison.VOLATILE_KEYS = {"request_id", "timestamp", "trace_id", "nonce", "cursor"}UUID_RE = re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}")ISO_RE = re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}")def normalize(value: Any) -> Any: if isinstance(value, dict): return {k: normalize(v) for k, v in sorted(value.items()) if k not in VOLATILE_KEYS} if isinstance(value, list): return [normalize(v) for v in value] if isinstance(value, str): value = UUID_RE.sub("<uuid>", value) return ISO_RE.sub("<ts>", value) return value@dataclassclass ToolCall: name: str args_digest: str ok: bool@dataclassclass Trace: case_id: str model: str calls: list[ToolCall] = field(default_factory=list) stop_reasons: list[str] = field(default_factory=list) turns: int = 0 def record(self, name: str, args: dict, ok: bool) -> None: payload = json.dumps(normalize(args), ensure_ascii=False, sort_keys=True) digest = hashlib.sha256(payload.encode()).hexdigest()[:12] self.calls.append(ToolCall(name=name, args_digest=digest, ok=ok)) @property def tool_names(self) -> list[str]: return [c.name for c in self.calls] def to_json(self) -> str: return json.dumps(asdict(self), ensure_ascii=False, indent=2, sort_keys=True)def instrument(tool_fn: Callable[[str, dict], Any], trace: Trace) -> Callable[[str, dict], Any]: """Wrap the existing dispatcher. The loop itself needs no changes.""" def wrapped(name: str, args: dict) -> Any: try: result = tool_fn(name, args) except Exception: trace.record(name, args, ok=False) raise trace.record(name, args, ok=True) return result return wrapped
Storing a digest rather than the arguments themselves means you can commit traces to your repository without leaking anything sensitive. When you want to inspect a difference, a hash mismatch tells you enough.
The loop only swaps the dispatcher and appends stop_reason.
# run_case.pyimport anthropicclient = anthropic.Anthropic()MODEL = "claude-sonnet-5"def run_case(case_id: str, prompt: str, tools: list[dict], dispatch) -> Trace: trace = Trace(case_id=case_id, model=MODEL) traced_dispatch = instrument(dispatch, trace) messages = [{"role": "user", "content": prompt}] while trace.turns < 24: # the cap is itself one of the invariants trace.turns += 1 resp = client.messages.create( model=MODEL, max_tokens=4096, tools=tools, messages=messages, ) trace.stop_reasons.append(resp.stop_reason) if resp.stop_reason != "tool_use": break messages.append({"role": "assistant", "content": resp.content}) results = [] for block in resp.content: if block.type != "tool_use": continue output = traced_dispatch(block.name, block.input) results.append({ "type": "tool_result", "tool_use_id": block.id, "content": json.dumps(output, ensure_ascii=False), }) messages.append({"role": "user", "content": results}) return trace
Returning every tool_result within the same turn is the requirement I covered in using Parallel Tool Use in production. Here it is enough to confirm that instrumentation does not disturb that structure.
✦
Thank you for reading this far.
Continue Reading
What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.
WHAT YOU'LL LEARN
✦Complete implementation of a thin wrapper that records an agent's execution trace: tool sequence, normalized arguments, and side effects
✦Seven deterministic invariants that catch regressions without an LLM judge, plus how to set thresholds from measured variance
✦A measured record of median tool calls rising from 5 to 8 after a default model switch, and the procedure that surfaced it
Secure payment via Stripe · Cancel anytime
✦
Unlock This Article
Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.
Verifying a trajectory does not need an LLM. I would argue it should not have one. If the judgment wobbles, your regression detector has itself regressed.
These are the seven invariants I run in production.
# invariants.pyWRITE_TOOLS = {"publish_article", "commit_files", "post_log"}def assert_invariants(trace: Trace, *, max_calls: int, required: set[str]) -> list[str]: violations: list[str] = [] names = trace.tool_names # 1. Loop inflation if len(names) > max_calls: violations.append(f"tool calls {len(names)} > {max_calls}") # 2. A required tool never ran missing = required - set(names) if missing: violations.append(f"missing required tools: {sorted(missing)}") # 3. Identical call repeated (a sign the model lost a prior read) seen = set() for call in trace.calls: key = (call.name, call.args_digest) if key in seen: violations.append(f"duplicate call: {call.name}") seen.add(key) # 4. At most one write action per case writes = [n for n in names if n in WRITE_TOOLS] if len(writes) > 1: violations.append(f"multiple write actions: {writes}") # 5. Read before write if writes: first_write = names.index(writes[0]) if not any(n not in WRITE_TOOLS for n in names[:first_write]): violations.append("write before any read") # 6. A failed tool call was swallowed if any(not c.ok for c in trace.calls) and trace.stop_reasons[-1] == "end_turn": violations.append("ended successfully despite a failed tool call") # 7. Truncation crept in if "max_tokens" in trace.stop_reasons: violations.append("hit max_tokens mid-trajectory") return violations
Why these seven? Because none of them are visible in the output. Number five is the clearest case: if the agent happens to write the same content as last time, the result looks perfectly correct even though it never read anything first.
Diffing against a baseline
Invariants are an absolute fence. Alongside them you want a relative signal: how different is this run from the last accepted one? Edit distance over the tool sequence turned out to be the most workable form for me.
def levenshtein(a: list[str], b: list[str]) -> int: prev = list(range(len(b) + 1)) for i, x in enumerate(a, 1): cur = [i] for j, y in enumerate(b, 1): cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (x != y))) prev = cur return prev[-1]def drift_ratio(baseline: Trace, current: Trace) -> float: """0.0 means identical. 1.0 means an entirely different path.""" denom = max(len(baseline.tool_names), len(current.tool_names), 1) return levenshtein(baseline.tool_names, current.tool_names) / denom
Watch the per-case maximum, not the median across the golden set. A median happily hides the one case out of 72 that went completely off the rails.
What the default model switch actually changed
After Sonnet 5 became the default, I replayed my 72-case golden set against both the old and new setup. My notes look like this.
Metric
Before
After
Tool calls (median)
5
8
Max drift_ratio
—
0.62
Invariant violations
0
3 (duplicate calls)
Wall-clock per case (median)
41 s
63 s
Output eval pass rate
72/72
72/72
The last row says everything. Output evaluation stayed perfect while the path underneath it moved.
Digging into the extra calls, most of them were confirmation reads: the agent re-read a file before writing to it. That is a desirable behavior. The three cases with genuinely duplicated reads went back to the original call count once I put a small cache in front of the read tool.
A behavior change is not necessarily a bad change. What is bad is not being able to see it. So I draw the line at invariant violations rather than at a drift threshold.
Wiring it into CI without constant false failures
Drop trajectory tests into CI as-is and they will fail, because models are not deterministic. The compromise I settled on has three parts.
# test_trajectory.pyimport pytestGOLDEN = load_cases("golden/cases.jsonl") # 72 casesBASELINE = load_traces("golden/baseline/") # last accepted traces@pytest.mark.parametrize("case", GOLDEN, ids=lambda c: c["id"])def test_invariants_hold(case): """A single violation fails the build. No tolerance here.""" trace = run_case(case["id"], case["prompt"], TOOLS, dispatch) violations = assert_invariants(trace, max_calls=case["max_calls"], required=set(case["required"])) assert not violations, f"{case['id']}: {violations}"def test_drift_within_budget(): """Drift is judged in aggregate so one noisy case cannot block CI.""" ratios = [] for case in GOLDEN: trace = run_case(case["id"], case["prompt"], TOOLS, dispatch) ratios.append(drift_ratio(BASELINE[case["id"]], trace)) severe = [r for r in ratios if r > 0.5] assert len(severe) <= 2, f"{len(severe)} cases drifted severely: {ratios}"
First, invariants fail immediately with no tolerance. Give them slack and the fence stops being a fence.
Second, drift is judged in aggregate. Up to two severe deviations across 72 cases are accepted. I chose that "two" by running the set five times without changing the model and taking the worst count I saw. Set thresholds from your own measured variance, not from reasoning.
Third, baselines are never updated automatically. I replace them by hand, once I have looked at the deviation and decided it is a change I want. Automatic updates quietly ratify quiet degradation.
What I learned running it
Weak argument normalization makes every trace unique. For the first two weeks I had no <ts> substitution, so my diffs sat at 100% permanently. Normalization is the piece to get right first.
Adding cases where zero tool calls are expected paid off as well. A model that starts calling tools when it should not is exactly the cost inflation I wrote about in controlling cost with tool_choice. Without a case that asserts silence, that regression walks straight through.
And the stop_reason sequence is more eloquent than I expected. A case with max_tokens in the middle will break at the next model update, even if today's output is fine. Seeing it before it breaks made the seventh invariant the cheapest one I have.
Where to start
Build ten golden cases. Add trace.py and the seven invariants. You will not need to touch a single line of your existing agent loop.
Then run those ten cases five times without changing the model. The spread you observe is the starting point for every threshold in your environment. On the pipeline behind my own App Store releases, that single measurement replaced a lot of guesswork.
Until I built this, I assumed correct output was enough. It turns out correctness rests on the health of the path that produced it. I hope it proves useful in your own work, and thank you for reading.
Share
Thank You for Reading
Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.