Make Agent Failures Reproducible: Deterministic Replay and Event Sourcing

A wrapper that switches between record and replay

Instead of calling a nondeterministic boundary directly, always route it through this single boundary() function. In record mode it runs the real work and writes the result to the log; in replay mode it never calls the real work and returns the value from the log.

// boundary.ts
import { AgentEvent, BoundaryKind, EventStore } from "./events";
 
export type Mode = "record" | "replay";
 
export class RunContext {
  private seq = 0;
  private cursor = 0;
  private recorded: AgentEvent[] = [];
 
  constructor(
    public readonly runId: string,
    public readonly mode: Mode,
    private readonly store: EventStore,
    private replayLog: AgentEvent[] = [],
  ) {}
 
  static async forReplay(runId: string, store: EventStore) {
    const log = await store.load(runId);
    return new RunContext(runId, "replay", store, log);
  }
 
  // every nondeterministic boundary passes through this one point
  async boundary<T>(
    kind: BoundaryKind,
    key: string,
    input: unknown,
    real: () => Promise<{ value: T; tokensIn?: number; tokensOut?: number; costUsd?: number }>,
  ): Promise<T> {
    const seq = this.seq++;
 
    if (this.mode === "replay") {
      const e = this.replayLog[this.cursor++];
      if (!e || e.kind !== kind || e.key !== key || e.seq !== seq) {
        // branch diverged from record time = the replay premise is broken
        throw new ReplayDivergenceError(this.runId, seq, kind, key, e);
      }
      return e.output as T;
    }
 
    const r = await real();
    const event: AgentEvent = {
      runId: this.runId, seq, kind, key,
      input, output: r.value,
      tokensIn: r.tokensIn, tokensOut: r.tokensOut, costUsd: r.costUsd,
      ts: new Date().toISOString(),
    };
    await this.store.append(event);
    this.recorded.push(event);
    return r.value;
  }
}
 
export class ReplayDivergenceError extends Error {
  constructor(runId: string, seq: number, kind: string, key: string, got: AgentEvent | undefined) {
    super(`Replay diverged at run=${runId} seq=${seq} expected ${kind}:${key} got ${got?.kind}:${got?.key}`);
    this.name = "ReplayDivergenceError";
  }
}

Deliberately throwing ReplayDivergenceError is the part of this design that earns its keep. If replay encounters a different order or a different call than was recorded, it tells you, at that exact moment, whether the code logic changed or the recording was incomplete. That's far better for a postmortem than silently returning a plausible value: it points at the divergence by seq, so you reach the root cause dramatically faster.

Routing the model call and tools through the boundary

From here it's just a matter of wrapping the model call and tool execution in boundary(). At record time the Claude request actually hits the Messages API; at replay time it returns the recorded response as-is.

// model.ts
import Anthropic from "@anthropic-ai/sdk";
import { RunContext } from "./boundary";
 
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }); // e.g. "YOUR_API_KEY"
 
// approximate cost (Sonnet 4.6 example; refresh the unit prices from the pricing page)
const PRICE_IN = 3.0 / 1_000_000;   // USD / input token
const PRICE_OUT = 15.0 / 1_000_000; // USD / output token
 
export async function callClaude(ctx: RunContext, messages: Anthropic.MessageParam[]) {
  return ctx.boundary("model", "claude:messages", { messages }, async () => {
    const res = await client.messages.create({
      model: "claude-sonnet-4-6",
      max_tokens: 2048,
      temperature: 0,
      messages,
    });
    const tin = res.usage.input_tokens;
    const tout = res.usage.output_tokens;
    return {
      value: res,
      tokensIn: tin,
      tokensOut: tout,
      costUsd: tin * PRICE_IN + tout * PRICE_OUT,
    };
  });
}

Tools follow the same pattern. Treat the clock and randomness as boundaries too, and enforce one discipline: agent code must never call Date.now() or Math.random() directly.

// tools.ts
import { RunContext } from "./boundary";
 
export const now = (ctx: RunContext) =>
  ctx.boundary("clock", "now", null, async () => ({ value: Date.now() }));
 
export const rand = (ctx: RunContext) =>
  ctx.boundary("random", "rand", null, async () => ({ value: Math.random() }));
 
export async function fetchOrders(ctx: RunContext, customerId: string) {
  return ctx.boundary("tool", "tool:fetch_orders", { customerId }, async () => {
    const res = await fetch(`https://api.example.com/orders?c=${customerId}`);
    return { value: await res.json() };
  });
}

Hold that discipline and the agent's main loop — call the model, run a tool, call again — works unchanged for both recording and replay. Your production code and your postmortem code become one and the same, which structurally eliminates the nastiest mismatch of all: writing a separate debug path that quietly behaves differently from production.

Replaying a single failed run

This is the operational goal. When runId=run_2026_0531_0203 fails at 2 a.m., the next morning I reproduce that one execution offline on my own machine.

// replay.ts
import { RunContext } from "./boundary";
import { jsonlStore } from "./store";
import { runAgent } from "./agent"; // the exact same loop used in production
 
async function replay(runId: string) {
  const ctx = await RunContext.forReplay(runId, jsonlStore);
  try {
    const result = await runAgent(ctx); // touches no network and no LLM
    console.log("replay complete:", result);
  } catch (e) {
    if (e instanceof Error && e.name === "ReplayDivergenceError") {
      console.error("divergence detected — code change or missing recording:", e.message);
    } else {
      console.error("reproduced the failure at the same point as record time:", e);
    }
  }
}
 
replay(process.argv[2]);

Because replay touches nothing external, you can run it as many times as you like, setting breakpoints wherever you want. You can separate whether the failure came from model output that wobbled even at temperature: 0, from a specific gap in the order data, or from a clock-based branch that only fires at night — all without changing a single line of code. In my own experience, intermittent failures that used to take two or three days to isolate now generally get pinned down the same morning once replay is in place.

Attributing cost per run from the same log

This log isn't only for postmortems — it doubles as your cost attribution source. Since model events carry tokensIn / tokensOut / costUsd, aggregating a run tells you which job spent how many tokens for how much money.

// cost.ts
import { EventStore } from "./events";
 
export async function runCost(store: EventStore, runId: string) {
  const events = await store.load(runId);
  const model = events.filter((e) => e.kind === "model");
  const tokensIn = model.reduce((s, e) => s + (e.tokensIn ?? 0), 0);
  const tokensOut = model.reduce((s, e) => s + (e.tokensOut ?? 0), 0);
  const costUsd = model.reduce((s, e) => s + (e.costUsd ?? 0), 0);
  return { runId, calls: model.length, tokensIn, tokensOut, costUsd };
}

You don't need a separate dashboard — the execution log itself becomes the primary source. In the backend automation for my wallpaper apps, I run this aggregation at the start of each month and consistently find one or two runs burning three times the expected tokens. Most turn out to be a retry that looped or a context stuffed with unnecessary history, and in both cases opening the model input at the offending seq in replay makes the cause obvious at a glance.

Things that actually tripped me up in production

A few notes you only learn by operating this, not from the official samples.

First, handling personal data. Recording the input / output of a boundary verbatim means customer data ends up in tool outputs. Run a per-key masking function just before append() and redact PII that isn't needed for reproduction. What reproduction needs is the value that decided a branch, which is often not the raw personal data itself.

Second, schema evolution. Adding fields to AgentEvent can break replay of older runIds. Carry a schemaVersion on each event and keep an adapter on the replay side that can read older versions. Just as a shrine carpenter leaves joints that anticipate future repair, building in consideration for whoever inspects this later — usually your future self — ends up being the cheapest path.

Finally, retention. Keeping every run forever bloats storage, so my default is "successful runs for 7 days, failed runs for 90 days." Failures live longer because the subject of a postmortem is always the failing side. Embedding the date in the runId also makes automatic deletion via a lifecycle policy straightforward to express.

A minimal rollout order, and what I recommend

If you're retrofitting this onto an agent that's already running, I recommend this order. Don't try to do it all at once — close one kind of boundary at a time.

1. Introduce RunContext and boundary(), and route only the model call first (smallest blast radius, biggest payoff).
2. Grep for direct Date.now() and Math.random() calls and replace them with clock / random boundaries.
3. Wrap external-I/O tools in a tool boundary one at a time. For non-idempotent side effects (payments, sends), execute only at record time and return the logged value on replay.
4. Make the EventStore append-only JSON Lines and carry schemaVersion from day one.
5. Always include the failing runId in alerts, and provide a one-command path to reproduce it via replay.ts.

As a guiding recommendation, pin temperature to 0 for both record and replay. It won't guarantee exact reproduction, but keeping the setting identical between record and replay reduces false divergence detections. For retention, default to "short for success, long for failure," and when in doubt keep the failing side — that's my conclusion on the best cost/benefit for postmortems. Conversely, I'd advise against uniformly long retention for every run, since it rarely justifies the storage cost.

Deterministic replay isn't a flashy feature. But being able to take a failure that happened once, unseen, in the dead of night, faithfully reproduce it the next morning, and actually reach its cause — that quiet confidence is a solid foundation for software meant to run for a long time. As a next step, find one place in your current automation where you call Date.now() directly and replace it with boundary("clock", ...). That single change is your first doorway into a reproducible design.

If you're wrestling with the reproducibility of your own nightly jobs, I hope this helps. Thank you for reading.