Let me start with one piece of context. Earlier this year I ran a six-step Claude Agent SDK workflow in production. Step five timed out against an external API, and the four side effects already in flight — a payment hold, a draft email, a CRM update, a Slack post — had no defined way to come back. I had been so focused on the SDK's reasoning loop that "what happens when we fail halfway" had drifted to the bottom of my list. That night I wrote four ad-hoc rollback scripts by hand, and the next morning I sat down and started designing this properly.
This article is the design that came out of that small incident. The Agent SDK docs tell you to "make tools idempotent" and "design for retries," but I have not seen a worked example of the saga pattern — the discipline of unwinding side effects across multiple services — written for agents specifically. Below is the structure I now use, with the production lessons that shaped it.
Why agents need the saga pattern
The Agent SDK is, at its core, a loop that interleaves model reasoning with tool calls. Inside one loop you might trigger:
- A payment reservation (Stripe
payment_intent) - An inventory hold in your own database
- A shipping API booking with a 3PL
- A customer email
- An audit log write
Each of these is a side effect on a separate system. When step three fails, your local database transaction cannot roll back the Stripe charge. You are now in the world of distributed side effects, and the standard answer there is the saga pattern: a sequence of forward steps, each paired with a compensating step that semantically undoes it.
Agent workflows are an unusually good fit for sagas because external APIs, your own DB, and LLM calls are typically interleaved. They are an unusually bad fit, too, because nothing in the SDK enforces the saga structure for you. You have to assemble it yourself, and the cost of assembling it badly is an entire class of bugs that only surface in production.
The other reason this matters specifically for agents: an autonomous loop will aggressively keep trying to make forward progress if you let it. A regular service has one human-facing path that fails loudly. An agent has a model that may "decide" to retry a side effect through a different tool, which is exactly the recipe for double-charging a customer. Saga discipline gives you a place to express "this entire group of side effects either committed or fully unwound; there is no middle state."
Choreography vs. orchestration: pick orchestration
There are two classic implementations of sagas. In choreography, services emit events and react to them autonomously. In orchestration, a central coordinator drives the order of execution.
For agent workflows I almost always pick orchestration, for three reasons. First, the Agent SDK already has a central reasoning loop, and a saga coordinator slots into the same place naturally. Second, agent steps frequently need a model decision in the middle, and event-driven choreography fragments that decision logic across handlers in ways that make it almost impossible to audit. Third, ordering compensations correctly is much easier when one place owns the state — there is no "we both think the other handled the rollback" failure mode.
If your saga grows past about ten steps, the orchestrator starts to bloat. That is the moment to split it into sub-sagas owned by different agents — multi-agent territory. I covered the multi-agent operating model in detail in Claude API multi-agent production patterns and recommend reading that alongside this piece if you are headed in that direction.
A short note on naming, because the literature is inconsistent: I use "step" for a forward action and "compensation" for its inverse. Some authors call the pair a "transaction" and the inverse a "compensating transaction." Either is fine; pick one and be consistent in code review.
A minimal saga coordinator
First, the type for a single step. Each step has a forward execute and a compensate to undo it.
// saga/types.ts
export interface SagaStep<TInput, TResult> {
name: string;
execute: (input: TInput, ctx: SagaContext) => Promise<TResult>;
compensate: (input: TInput, executeResult: TResult | null, ctx: SagaContext) => Promise<void>;
// Idempotency-key prefix used by the compensator (covered below).
idempotencyKeyPrefix: string;
}
export interface SagaContext {
sagaId: string; // One ID per saga run; UUIDv7 is a great fit.
attempt: number; // Current retry attempt.
store: SagaStateStore; // Where step status is persisted.
}Notice that compensate receives executeResult: TResult | null. That distinction matters: a compensation called before execute ran (because some earlier step failed first) should behave differently than one called after execute succeeded. Burying that into the type forces every author to think about it. I have written compensations that crashed because they assumed the result was always present; this signature now prevents that whole category of bug.
Now the coordinator itself.
// saga/coordinator.ts
import { SagaStep, SagaContext } from "./types";
interface RunOptions {
sagaId: string;
store: SagaStateStore;
}
export async function runSaga<I>(
steps: SagaStep<I, unknown>[],
initialInput: I,
opts: RunOptions
): Promise<{ ok: boolean; failedAt?: string }> {
const ctx: SagaContext = { sagaId: opts.sagaId, attempt: 1, store: opts.store };
const executed: { step: SagaStep<I, unknown>; result: unknown }[] = [];
for (const step of steps) {
try {
// Resume safely: skip steps already marked succeeded.
const cached = await ctx.store.getStepResult(ctx.sagaId, step.name);
if (cached?.status === "succeeded") {
executed.push({ step, result: cached.result });
continue;
}
await ctx.store.markStepRunning(ctx.sagaId, step.name);
const result = await step.execute(initialInput, ctx);
await ctx.store.markStepSucceeded(ctx.sagaId, step.name, result);
executed.push({ step, result });
} catch (err) {
await ctx.store.markStepFailed(ctx.sagaId, step.name, err);
// Compensate in reverse order.
for (const done of executed.reverse()) {
try {
await done.step.compensate(initialInput, done.result, ctx);
} catch (compErr) {
// Compensation failures are the worst class of failure: surface for humans.
await ctx.store.markCompensationFailed(ctx.sagaId, done.step.name, compErr);
}
}
return { ok: false, failedAt: step.name };
}
}
return { ok: true };
}Expected output: { ok: true } when everything succeeds, or { ok: false, failedAt: "step-name" } after the in-flight steps have been compensated in reverse order.
The two non-obvious rules embedded here are: compensate in reverse order, and record compensation failures as a distinct category. Reverse order matters because, if you cancel the payment before releasing the inventory hold, you can end up with reserved inventory you no longer have any record of. And compensation failures must not be folded into the generic "saga failed" log line — they almost always need a human, and bundling them with normal failures makes them invisible at exactly the moment you want them loud.
Bridging Agent SDK tool calls to the saga
This is the agent-specific part. In the Agent SDK, the model decides to call tools via tool_use events. The cleanest pattern I have used is a tool whose body simply invokes runSaga.
// tools/checkout-saga-tool.ts
import { Tool } from "@anthropic-ai/claude-agent-sdk";
import { runSaga } from "../saga/coordinator";
import { reservePayment, capturePayment, cancelPayment } from "./payment";
import { allocateInventory, releaseInventory } from "./inventory";
import { scheduleShipping, cancelShipping } from "./shipping";
export const checkoutSagaTool: Tool = {
name: "execute_checkout",
description: "Run a customer checkout: payment, inventory, and shipping in a single saga.",
input_schema: {
type: "object",
properties: {
orderId: { type: "string" },
customerId: { type: "string" },
amountJpy: { type: "number" },
},
required: ["orderId", "customerId", "amountJpy"],
},
async run(input, sdkCtx) {
const sagaId = `chk_${input.orderId}`; // Tying sagaId to orderId makes resumption deterministic.
const result = await runSaga(
[
{
name: "reserve-payment",
idempotencyKeyPrefix: "pay-reserve",
execute: (i) => reservePayment(i.amountJpy),
compensate: (_i, r) => (r ? cancelPayment(r as { paymentIntentId: string }) : Promise.resolve()),
},
{
name: "allocate-inventory",
idempotencyKeyPrefix: "inv-alloc",
execute: (i) => allocateInventory(i.orderId),
compensate: (i) => releaseInventory(i.orderId),
},
{
name: "schedule-shipping",
idempotencyKeyPrefix: "ship-schedule",
execute: (i) => scheduleShipping(i.orderId),
compensate: (i) => cancelShipping(i.orderId),
},
],
input,
{ sagaId, store: sdkCtx.runtime.kv }
);
if (!result.ok) {
return {
status: "compensated",
failedAt: result.failedAt,
message: "Side effects to this point have been rolled back. Please offer the user an alternative.",
};
}
return { status: "ok" };
},
};The reasoning loop now only sees three outcomes from this tool: success, failure with rollback complete, or failure with compensation pending. Add a single sentence to the system prompt — "If a tool returns compensated, propose an alternative path to the user" — and the user-facing experience becomes coherent again.
Notice that sagaId is derived deterministically from orderId. This is what makes the saga resumable and idempotent at the outer boundary too: if the agent reasoning loop crashes and the same tool_use is replayed, the second call lands on the same sagaId, finds the partial state, and either resumes or no-ops. Without that determinism, every retry of the tool call creates a brand-new saga, and you end up multiplying side effects across sagas.
A worked failure scenario
Let me walk through what happens with one specific failure. Say the agent calls execute_checkout with orderId = "ord_42" and amountJpy = 5000. The saga starts:
reserve-paymentruns and returns{ paymentIntentId: "pi_AAA" }. The store now holds:chk_ord_42:reserve-payment = succeeded.allocate-inventoryruns and succeeds. Store:chk_ord_42:allocate-inventory = succeeded.schedule-shippingcalls the 3PL, which returns 504 after 30 seconds.
The coordinator catches the error, marks schedule-shipping as failed, and walks the executed list in reverse. It calls releaseInventory("ord_42"), which writes a row into the compensations table and decrements the reserved counter. Then it calls cancelPayment({ paymentIntentId: "pi_AAA" }), which voids the payment intent at Stripe.
The tool returns { status: "compensated", failedAt: "schedule-shipping" }. The agent's reasoning loop sees this and, guided by the system prompt, says to the user: "We weren't able to schedule shipping with our usual carrier. Would you like to try a different carrier or wait a few hours?"
Now consider the variant where, between steps 2 and 3, the coordinator process itself crashes — perhaps an OOM, perhaps a deploy. On restart, the agent sees the previous tool call had no result and retries execute_checkout. Same orderId, same sagaId. The new runSaga invocation reads the store, finds steps 1 and 2 already succeeded, skips them, and tries schedule-shipping directly. If it succeeds this time, we are done. If it fails again, we compensate exactly the same two steps that committed previously — not twice, because the resumed coordinator only sees two succeeded steps in the journal.
That is the piece that makes the saga safe under arbitrary process restarts, and it only works because of the deterministic sagaId.
Idempotency keys: stop double-compensating
The single most painful saga bug I have written is double compensation. A network blip swallows the response from cancelPayment, the saga resumes, and the same compensation runs again. With external APIs that already support idempotency keys (Stripe, Stripe-clones), this often hides itself. Against your own database it does not: a naive UPDATE inventory SET reserved = reserved - 1 runs twice and you go negative.
The convention I use everywhere:
// saga/idempotency.ts
export function compensateKey(sagaId: string, stepName: string, prefix: string): string {
// e.g. "pay-reserve:chk_abc123:reserve-payment"
return `${prefix}:${sagaId}:${stepName}`;
}
// In your DB-backed compensator:
async function releaseInventory(orderId: string): Promise<void> {
const key = compensateKey(`chk_${orderId}`, "allocate-inventory", "inv-alloc");
// Skip if this compensation has already been applied.
const seen = await db.query("SELECT 1 FROM compensations WHERE key = $1", [key]);
if (seen.rowCount > 0) return;
await db.transaction(async (trx) => {
await trx.query("UPDATE inventory SET reserved = reserved - 1 WHERE order_id = $1", [orderId]);
await trx.query("INSERT INTO compensations(key, applied_at) VALUES ($1, now())", [key]);
});
}Expected behaviour: no matter how many times releaseInventory is invoked for the same orderId, reserved decreases by exactly one. The compensations table is the only proof that a compensation has run, so it must be written in the same transaction as the side effect itself. If you split them, you can end up with the inventory restored but the idempotency record missing — the worst possible state, because the next retry will compensate again.
I keep the compensations table tiny — key TEXT PRIMARY KEY, applied_at TIMESTAMPTZ NOT NULL — and prune entries older than the longest possible saga lifetime (in our case, 30 days). The pruning is the only operational chore this introduces, and it is well worth the simplicity.
For more on the idempotency machinery on the forward side, Idempotency design patterns for the Claude Agent SDK is a companion piece I'd recommend reading after this one.
Where to put saga state: Redis, Durable Objects, or RDBMS
When the process running your coordinator dies, your saga is not "failed" — it is paused. If you cannot resume it on a fresh process, you cannot run long-lived workflows safely. That is what SagaStateStore is for, and the choice of backing store is one of the more consequential decisions you will make.
The three options I have used in production:
- Redis: low-latency writes, easy TTLs, and ubiquitous. Durability is weaker than a real database —
appendfsync everysecwill lose at most one second of writes on crash, which for sagas is sometimes acceptable and sometimes not. I always shadow-write the journal to a more durable store. Best for sagas that finish within a day. - Cloudflare Durable Objects: a single strongly-consistent writer per saga ID, no shard reasoning required. The least painful option for multi-region deployments, because the DO is automatically pinned to one region and reads/writes are linearizable by construction. The per-request cost is real, but operational savings make it cheap in practice for typical workloads.
- PostgreSQL with advisory locks: easiest to retrofit into an organisation that already runs Postgres. The journal is a table with a primary key on
(saga_id, step_name), plus a status enum and a JSON column for the step result. With a careful index on(status, started_at)you can list stuck sagas in milliseconds. This option happily handles sagas that run for weeks.
The decision matrix I use:
- Saga lifetime under one hour and single-region: Redis is fine.
- Multi-region or strong durability needed: Durable Objects.
- Already running Postgres and want one fewer dependency: Postgres.
For an end-to-end agent workflow built on Durable Objects, Claude Agent SDK and Temporal for durable AI workflows walks through a similar architecture and may help you choose more quickly.
Common mistakes and traps
These are the ones I have either lived through or pulled out of code review.
1. Treating timeouts as failure. A timeout means "I do not know the result." If your reservePayment times out, the server may have already created the reservation. Naively retrying without an idempotency key gives you double charges. Compensations and retries must always assume the unknown-state case and pass an idempotency key downstream. When in doubt, query the upstream system to find out the true state before deciding whether to compensate.
2. No plan for compensation failure. A failed compensation is a real distributed-systems incident. I keep compensation_failed as its own state, with an immediate Slack alert and a small ops dashboard. Trying to auto-retry away a compensation failure means you stop noticing the incidents that need humans. The right design is "auto-retry up to three times in five minutes, then page someone."
3. Combining side effects to "save a step." "Why have separate reservePayment and allocateInventory when one tool can do both?" Because the moment you do, the saga loses its meaning. If both succeed but a later step fails, you cannot compensate one without compensating the other, and you have to write a custom rollback that is twice as complex as two compensations would have been. Keep one side effect per step. Test ergonomics, observability, and compensability all degrade by an order of magnitude when you stuff steps.
4. Letting the LLM make decisions inside the saga. I tried this once. Mixing model decisions into the deterministic step list means your compensation set changes between runs and you cannot reason about correctness. Keep the LLM strictly outside the saga: it decides whether to call the saga and how to interpret the result, and nothing more. Sagas are the part of your system that must be deterministic; that is non-negotiable.
5. No retry budget. Infinite retries against an unhealthy dependency just consume your saga coordinator. I always cap consecutive failures of the same step (typically three) and escalate to a human after that. The point of sagas is to make automation safe; that includes knowing when to stop being automated.
6. Mixing forward and compensation logic in one function. A cancelPayment(intentId) function that "also retries the original payment if it fails" is a future incident waiting to happen. Compensations should do nothing but undo. If you need a "retry on a different rail" path, that is a new saga, not a branch inside compensation.
Testing sagas without a chaos cluster
The fastest way to gain confidence in saga correctness is property-based fault injection. The setup is small:
- Wrap every step with a fault-injection harness that, with some probability, throws an exception or stalls a configured duration.
- Run the saga against an in-memory store and assert two invariants at the end: every committed forward step has a matching
compensatedorsucceededoutcome in the journal, and the underlying "world" (an in-memory mock of inventory, payments, etc.) is in a state consistent with the journal. - Repeat for thousands of seeded runs.
// saga/test-harness.ts
function withFaultInjection<I, R>(step: SagaStep<I, R>, probability: number): SagaStep<I, R> {
return {
...step,
execute: async (i, ctx) => {
if (Math.random() < probability) throw new Error(`injected:${step.name}`);
return step.execute(i, ctx);
},
};
}
// In your test:
for (let seed = 0; seed < 5000; seed++) {
const world = createMockWorld(seed);
await runSaga(
steps.map((s) => withFaultInjection(s, 0.2)),
{ orderId: `ord_${seed}`, customerId: "c1", amountJpy: 100 },
{ sagaId: `chk_${seed}`, store: makeInMemoryStore() }
);
expectConsistent(world); // payments == 0 OR (payments == 100 AND inventory == reserved AND shipping booked)
}The first time I ran this against a saga I thought was correct, two of five thousand runs ended in an inconsistent state. Both turned out to be the same bug: a compensator that swallowed an exception silently. I would never have found that bug in regular unit tests.
Observability — make sagas visible
The minimum telemetry I want for any saga in production:
- Trace spans by sagaId and step name. I emit an OpenTelemetry span inside
runSagaand a child span per step, with thesagaIdas a span attribute. Without this, debugging a stuck saga is grep-archaeology. - Compensation rate metrics. A rising compensation rate is a design smell. I dashboard it next to error rate. A sustained 5% compensation rate is usually a sign that an upstream is degrading or a step's preconditions are wrong.
- Compensation-failure alerts. Direct to Slack or PagerDuty. Humans should see these in seconds, not hours. The alert text should always include the
sagaIdand the step name; nothing else matters at 2 a.m. - Long-running saga inventory. A scheduled query that lists every saga
runningfor more than an hour. The cost is tiny; the catch rate is high.
The first incident I had with sagas, I lost an hour at 2 a.m. trying to figure out which saga ID had stalled because I had no traces. Investing in the observability piece up front will pay for itself the first weekend you are on call.
If you'd like to keep the observability pipeline cleanly separated from the saga itself — emitting events from a separate outbox rather than inline — Claude Agent SDK and the Transactional Outbox pattern describes a useful split. For sagas with long-tail steps, decoupling event publication from the step execution prevents observability latency from becoming saga latency.
When the model and the saga disagree
One last edge case worth naming, because it has bitten me twice. The reasoning loop calls execute_checkout, which compensates and returns { status: "compensated" }. The model, instead of proposing an alternative to the user, re-calls execute_checkout with the same inputs because the system prompt was ambiguous about the meaning of "compensated."
If the saga is correctly idempotent on its outer boundary (same sagaId), this is harmless: the second call sees a journal full of compensated steps, sees no work to do, and immediately returns { status: "compensated" } again. But if the saga is not idempotent — for example, if a developer added "always run pre-checks first, regardless of journal state" — you can end up running the saga forward a second time, locking inventory you cannot pay for, and so on.
The defensive moves I take now are these:
- The outer
runSagaalways reads the journal first and refuses to start work when the previous run ended in a compensated state, unless an explicitrestart: trueflag is passed. The flag is never set by the model; only by a human-triggered retry. - The system prompt that drives the agent contains an explicit example: a tool result of
compensatedmeans "do not retry; explain the situation to the user." Without that example, the model occasionally treatscompensatedas a transient error. - I unit-test the agent prompt in a small harness that calls the tool in a loop with
compensatedresponses and asserts the model emits a user-facing message instead of re-calling the tool. This sounds excessive; it took me one prod incident to learn the test was cheap insurance.
The general lesson is: when a probabilistic component (the model) drives a deterministic component (the saga), every interface between them deserves explicit defensive design. The saga's outer boundary is one such interface, and treating it as a public API of your system — versioned, tested, documented — pays back quickly.
How to roll this out without overbuilding
Resist the temptation to build a full coordinator on day one. The progression I now recommend:
Start with a two- or three-step saga and lock in the compensation contract and idempotency convention. Persist state in Redis until you actually need a long-lived workflow, then move to Durable Objects or Postgres. Wire up tracing and the compensation-failure alert before you grow past four steps. Don't split into sub-sagas until you cross ten. Each of these is a small, reversible decision; the value of the saga pattern is mostly already paid back by step three.
A common pattern I have seen succeed: build the coordinator as a small library inside one repo, with its own tests, and let one team's saga be the proving ground. Once it is stable for a few weeks, extract it. Premature library extraction tends to produce abstractions that fit nobody's actual needs; one concrete saga in production tells you exactly which knobs deserve to be public API.
Saga design lacks the visible glamour of model selection or prompt engineering. But once an agent is running real side effects in production, this layer is what determines how often you wake up at 2 a.m. The careful, unglamorous discipline is exactly what makes the user-visible parts feel reliable.
Your next step
Pick one tool in your current Agent SDK workflow that fires two or more external side effects in sequence. Write the compensate function for each step, even just as a one-liner each. Two of those will surface compensation logic you had not yet written. From there, the rest of the pattern lands quickly. The first small saga is the only hard one.
Thank you for reading. I hope a piece of this helps the workflows you are designing right now.