⬡ API & SDK/2026-07-09Advanced

Same Output, Different Path — Guarding Agent Trajectories with Invariants

When the default model changes, your final output can stay correct while the path your agent takes quietly shifts. Here is a trajectory regression harness built on recorded tool traces and deterministic invariants, with working code and measured numbers.

claude-api⁷⁹ tool-use²² agent¹³ regression-testing² observability¹⁸

✦ Premium Article

My nightly batch looked fine. The articles it generated were correct. The logs it posted were correct. Nothing was broken.

The billing noticed before I did. Tool calls per day had climbed to roughly 1.6x their previous level. The answers were right. The path to those answers had changed.

If you only watch output quality, this kind of change stays invisible forever. The output evaluation I described in detecting quality regressions with an eval harness measures the final string. It does not measure the trajectory.

Running unattended agents as an indie developer, this quiet shift in the path turned out to be the hard one. Nothing breaks, so nothing alerts you. And by the time you notice, either cost or wall-clock time has already grown.

What actually needs to be recorded

Record the process, not the final answer. In practice, I found five fields sufficient.

Recorded field	What it protects against
Sequence of tool names	Reordering, or an unexpected tool appearing
Normalized arguments per call	Repeated calls with identical arguments
Turn count and total tool calls	Loop inflation
Sequence of stop_reason values	Truncation or pauses slipping in
Count of write-tool invocations	Duplicated side effects

The important part is that you do not store raw arguments. Timestamps and UUIDs make every trace unique, which makes comparison useless. Strip the volatile values before recording, and the trajectory becomes a stable object you can diff.

A thin wrapper that records the trace

Rather than reaching into the agent loop, wrap only the tool dispatch boundary. Your production loop stays untouched, which keeps the tested code and the shipped code from drifting apart.

# trace.py
from __future__ import annotations
import hashlib
import json
import re
from dataclasses import dataclass, field, asdict
from typing import Any, Callable
 
# Values that change on every run must be dropped before comparison.
VOLATILE_KEYS = {"request_id", "timestamp", "trace_id", "nonce", "cursor"}
UUID_RE = re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}")
ISO_RE = re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}")
 
 
def normalize(value: Any) -> Any:
    if isinstance(value, dict):
        return {k: normalize(v) for k, v in sorted(value.items()) if k not in VOLATILE_KEYS}
    if isinstance(value, list):
        return [normalize(v) for v in value]
    if isinstance(value, str):
        value = UUID_RE.sub("<uuid>", value)
        return ISO_RE.sub("<ts>", value)
    return value
 
 
@dataclass
class ToolCall:
    name: str
    args_digest: str
    ok: bool
 
 
@dataclass
class Trace:
    case_id: str
    model: str
    calls: list[ToolCall] = field(default_factory=list)
    stop_reasons: list[str] = field(default_factory=list)
    turns: int = 0
 
    def record(self, name: str, args: dict, ok: bool) -> None:
        payload = json.dumps(normalize(args), ensure_ascii=False, sort_keys=True)
        digest = hashlib.sha256(payload.encode()).hexdigest()[:12]
        self.calls.append(ToolCall(name=name, args_digest=digest, ok=ok))
 
    @property
    def tool_names(self) -> list[str]:
        return [c.name for c in self.calls]
 
    def to_json(self) -> str:
        return json.dumps(asdict(self), ensure_ascii=False, indent=2, sort_keys=True)
 
 
def instrument(tool_fn: Callable[[str, dict], Any], trace: Trace) -> Callable[[str, dict], Any]:
    """Wrap the existing dispatcher. The loop itself needs no changes."""
    def wrapped(name: str, args: dict) -> Any:
        try:
            result = tool_fn(name, args)
        except Exception:
            trace.record(name, args, ok=False)
            raise
        trace.record(name, args, ok=True)
        return result
    return wrapped

Storing a digest rather than the arguments themselves means you can commit traces to your repository without leaking anything sensitive. When you want to inspect a difference, a hash mismatch tells you enough.

The loop only swaps the dispatcher and appends stop_reason.

# run_case.py
import anthropic
 
client = anthropic.Anthropic()
MODEL = "claude-sonnet-5"
 
 
def run_case(case_id: str, prompt: str, tools: list[dict], dispatch) -> Trace:
    trace = Trace(case_id=case_id, model=MODEL)
    traced_dispatch = instrument(dispatch, trace)
    messages = [{"role": "user", "content": prompt}]
 
    while trace.turns < 24:  # the cap is itself one of the invariants
        trace.turns += 1
        resp = client.messages.create(
            model=MODEL, max_tokens=4096, tools=tools, messages=messages,
        )
        trace.stop_reasons.append(resp.stop_reason)
        if resp.stop_reason != "tool_use":
            break
 
        messages.append({"role": "assistant", "content": resp.content})
        results = []
        for block in resp.content:
            if block.type != "tool_use":
                continue
            output = traced_dispatch(block.name, block.input)
            results.append({
                "type": "tool_result",
                "tool_use_id": block.id,
                "content": json.dumps(output, ensure_ascii=False),
            })
        messages.append({"role": "user", "content": results})
 
    return trace

Returning every tool_result within the same turn is the requirement I covered in using Parallel Tool Use in production. Here it is enough to confirm that instrumentation does not disturb that structure.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦Complete implementation of a thin wrapper that records an agent's execution trace: tool sequence, normalized arguments, and side effects

✦Seven deterministic invariants that catch regressions without an LLM judge, plus how to set thresholds from measured variance

✦A measured record of median tool calls rising from 5 to 8 after a default model switch, and the procedure that surfaced it

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Assert invariants, do not appoint a judge

Verifying a trajectory does not need an LLM. I would argue it should not have one. If the judgment wobbles, your regression detector has itself regressed.

These are the seven invariants I run in production.

# invariants.py
WRITE_TOOLS = {"publish_article", "commit_files", "post_log"}
 
 
def assert_invariants(trace: Trace, *, max_calls: int, required: set[str]) -> list[str]:
    violations: list[str] = []
    names = trace.tool_names
 
    # 1. Loop inflation
    if len(names) > max_calls:
        violations.append(f"tool calls {len(names)} > {max_calls}")
 
    # 2. A required tool never ran
    missing = required - set(names)
    if missing:
        violations.append(f"missing required tools: {sorted(missing)}")
 
    # 3. Identical call repeated (a sign the model lost a prior read)
    seen = set()
    for call in trace.calls:
        key = (call.name, call.args_digest)
        if key in seen:
            violations.append(f"duplicate call: {call.name}")
        seen.add(key)
 
    # 4. At most one write action per case
    writes = [n for n in names if n in WRITE_TOOLS]
    if len(writes) > 1:
        violations.append(f"multiple write actions: {writes}")
 
    # 5. Read before write
    if writes:
        first_write = names.index(writes[0])
        if not any(n not in WRITE_TOOLS for n in names[:first_write]):
            violations.append("write before any read")
 
    # 6. A failed tool call was swallowed
    if any(not c.ok for c in trace.calls) and trace.stop_reasons[-1] == "end_turn":
        violations.append("ended successfully despite a failed tool call")
 
    # 7. Truncation crept in
    if "max_tokens" in trace.stop_reasons:
        violations.append("hit max_tokens mid-trajectory")
 
    return violations

Why these seven? Because none of them are visible in the output. Number five is the clearest case: if the agent happens to write the same content as last time, the result looks perfectly correct even though it never read anything first.

Diffing against a baseline

Invariants are an absolute fence. Alongside them you want a relative signal: how different is this run from the last accepted one? Edit distance over the tool sequence turned out to be the most workable form for me.

def levenshtein(a: list[str], b: list[str]) -> int:
    prev = list(range(len(b) + 1))
    for i, x in enumerate(a, 1):
        cur = [i]
        for j, y in enumerate(b, 1):
            cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (x != y)))
        prev = cur
    return prev[-1]
 
 
def drift_ratio(baseline: Trace, current: Trace) -> float:
    """0.0 means identical. 1.0 means an entirely different path."""
    denom = max(len(baseline.tool_names), len(current.tool_names), 1)
    return levenshtein(baseline.tool_names, current.tool_names) / denom

Watch the per-case maximum, not the median across the golden set. A median happily hides the one case out of 72 that went completely off the rails.

What the default model switch actually changed

After Sonnet 5 became the default, I replayed my 72-case golden set against both the old and new setup. My notes look like this.

Metric	Before	After
Tool calls (median)	5	8
Max drift_ratio	—	0.62
Invariant violations	0	3 (duplicate calls)
Wall-clock per case (median)	41 s	63 s
Output eval pass rate	72/72	72/72

The last row says everything. Output evaluation stayed perfect while the path underneath it moved.

Digging into the extra calls, most of them were confirmation reads: the agent re-read a file before writing to it. That is a desirable behavior. The three cases with genuinely duplicated reads went back to the original call count once I put a small cache in front of the read tool.

A behavior change is not necessarily a bad change. What is bad is not being able to see it. So I draw the line at invariant violations rather than at a drift threshold.

Wiring it into CI without constant false failures

Drop trajectory tests into CI as-is and they will fail, because models are not deterministic. The compromise I settled on has three parts.

# test_trajectory.py
import pytest
 
GOLDEN = load_cases("golden/cases.jsonl")     # 72 cases
BASELINE = load_traces("golden/baseline/")     # last accepted traces
 
 
@pytest.mark.parametrize("case", GOLDEN, ids=lambda c: c["id"])
def test_invariants_hold(case):
    """A single violation fails the build. No tolerance here."""
    trace = run_case(case["id"], case["prompt"], TOOLS, dispatch)
    violations = assert_invariants(trace, max_calls=case["max_calls"], required=set(case["required"]))
    assert not violations, f"{case['id']}: {violations}"
 
 
def test_drift_within_budget():
    """Drift is judged in aggregate so one noisy case cannot block CI."""
    ratios = []
    for case in GOLDEN:
        trace = run_case(case["id"], case["prompt"], TOOLS, dispatch)
        ratios.append(drift_ratio(BASELINE[case["id"]], trace))
    severe = [r for r in ratios if r > 0.5]
    assert len(severe) <= 2, f"{len(severe)} cases drifted severely: {ratios}"

First, invariants fail immediately with no tolerance. Give them slack and the fence stops being a fence.

Second, drift is judged in aggregate. Up to two severe deviations across 72 cases are accepted. I chose that "two" by running the set five times without changing the model and taking the worst count I saw. Set thresholds from your own measured variance, not from reasoning.

Third, baselines are never updated automatically. I replace them by hand, once I have looked at the deviation and decided it is a change I want. Automatic updates quietly ratify quiet degradation.

What I learned running it

Weak argument normalization makes every trace unique. For the first two weeks I had no <ts> substitution, so my diffs sat at 100% permanently. Normalization is the piece to get right first.

Adding cases where zero tool calls are expected paid off as well. A model that starts calling tools when it should not is exactly the cost inflation I wrote about in controlling cost with tool_choice. Without a case that asserts silence, that regression walks straight through.

And the stop_reason sequence is more eloquent than I expected. A case with max_tokens in the middle will break at the next model update, even if today's output is fine. Seeing it before it breaks made the seventh invariant the cheapest one I have.

Where to start

Build ten golden cases. Add trace.py and the seven invariants. You will not need to touch a single line of your existing agent loop.

Then run those ten cases five times without changing the model. The spread you observe is the starting point for every threshold in your environment. On the pipeline behind my own App Store releases, that single measurement replaced a lot of guesswork.

Until I built this, I assumed correct output was enough. It turns out correctness rests on the health of the path that produced it. I hope it proves useful in your own work, and thank you for reading.

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.