Shadow Mode with Claude Agent SDK — Measuring Agent Accuracy on Live Traffic Without Touching Users

Minimal implementation — running the agent in parallel

The mental model I use when building this wrapper is that the existing function is a trusted black box that already works, and my job is to add a side channel that does not disturb it. Anything that changes the return type, throws new exceptions, or introduces new timeout behavior is out of scope. The wrapper should be additive only.

Let's start from working code. The snippet below is a minimum viable shadow runner for a support ticket classifier. The existing classifier (legacyClassify) remains the source of truth, and a Claude Agent SDK call runs alongside with its result only being logged.

// shadow-runner.ts
// Goal: keep legacyClassify as the source of truth while running an
//       Agent SDK classifier in parallel and logging only its decision.
import Anthropic from "@anthropic-ai/sdk";
import { recordShadowResult } from "./shadow-logger";
 
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
 
type Ticket = { id: string; subject: string; body: string };
type Category = "billing" | "technical" | "account" | "other";
 
export async function classifyWithShadow(
  ticket: Ticket,
  legacyClassify: (t: Ticket) => Promise<Category>,
): Promise<Category> {
  // 1. Legacy path always wins. Never block it on the shadow.
  const legacyPromise = legacyClassify(ticket);
 
  // 2. Fire-and-forget the shadow. Do NOT Promise.allSettled —
  //    that would put shadow latency on the critical path.
  runShadow(ticket, legacyPromise).catch((err) => {
    console.warn("[shadow] suppressed error:", err?.message);
  });
 
  return legacyPromise;
}
 
async function runShadow(ticket: Ticket, legacyPromise: Promise<Category>) {
  const startedAt = Date.now();
  try {
    const response = await client.messages.create({
      model: "claude-haiku-4-5-20251001",
      max_tokens: 200,
      system:
        "You are a ticket classifier. " +
        "Reply with exactly one word: billing, technical, account, or other.",
      messages: [
        {
          role: "user",
          content: `Subject: ${ticket.subject}\nBody: ${ticket.body}`,
        },
      ],
    });
    const shadow = normalizeCategory(
      response.content[0].type === "text" ? response.content[0].text : "",
    );
    const legacy = await legacyPromise;
 
    await recordShadowResult({
      ticketId: ticket.id,
      legacy,
      shadow,
      match: legacy === shadow,
      latencyMs: Date.now() - startedAt,
      usage: response.usage,
    });
  } catch (err) {
    await recordShadowResult({
      ticketId: ticket.id,
      legacy: await legacyPromise.catch(() => "other"),
      shadow: null,
      match: null,
      latencyMs: Date.now() - startedAt,
      error: normalizeError(err),
    });
  }
}
 
function normalizeCategory(raw: string): Category {
  const lower = raw.trim().toLowerCase();
  if (["billing", "technical", "account", "other"].includes(lower)) {
    return lower as Category;
  }
  // Bucketing surprises into "other" is a safe default, but you also
  // want to count format violations separately so you can tune the prompt.
  return "other";
}
 
function normalizeError(err: unknown) {
  if (err instanceof Error) {
    return { name: err.name, message: err.message };
  }
  return { name: "Unknown", message: String(err) };
}

The key decision here is fire-and-forget. Using await Promise.all([legacy, shadow]) would pull shadow latency (especially if Extended Thinking is on — that can add seconds) onto the user-facing path. The iron rule of shadow mode is: the production path must not lose a single millisecond to the shadow.

What you should observe after deploying this: classifyWithShadow returns the legacy result at the same latency as before, and shadow results trickle into your log sink a few seconds later, untouched by anyone waiting.

What to log, and how to compare

Before we get into the schema, a note on PII. Customer support and email workflows almost always contain names, contact details, or financial identifiers. Hashing or tokenizing the raw input before it lands in your shadow table is non-negotiable. I have seen shadow projects paused for months because a privacy review found plaintext customer emails in the analytics warehouse. Bake the redaction step into the write path from day one — it is much harder to retrofit afterward.

The quality of your downstream analysis depends entirely on what you record. I recommend logging at least these six fields.

1. A hash of the input (avoid raw PII — dereference when needed)
2. The legacy / source-of-truth decision
3. The agent's decision (including format violations and errors)
4. Match/mismatch flag
5. Latency and token usage (for cost forecasts)
6. The agent version (prompt hash or template tag)

Number 6 is the one that bites you later. If you update the prompt but the old logs sit in the same table, your accuracy charts start lying to you. I always include prompt_hash: sha256(system + templateVersion) in every row.

Comparing is straightforward for discrete tasks like classification, but for generative tasks (summaries, replies) exact match is useless. You need a bundle of signals instead.

# shadow_compare.py
# Goal: compare an agent's generated output against the legacy output
#       with a cluster of signals, not a single similarity score.
from dataclasses import dataclass
from typing import Optional
import difflib
 
@dataclass
class ComparisonResult:
    text_similarity: float           # 0.0-1.0
    length_ratio: float              # 1.0 means identical length
    sentiment_match: Optional[bool]  # only set when sentiment was analyzed
    structural_match: bool           # headings and bullet counts align
    flags: list[str]                 # reasons human review should be triggered
 
def compare_shadow_vs_legacy(shadow: str, legacy: str) -> ComparisonResult:
    """Multi-axis diff so reviewers can localize *where* the agent drifted."""
    flags: list[str] = []
 
    text_sim = difflib.SequenceMatcher(None, shadow, legacy).ratio()
    if text_sim < 0.3:
        flags.append("very_different_text")
 
    shadow_len = max(len(shadow), 1)
    legacy_len = max(len(legacy), 1)
    length_ratio = shadow_len / legacy_len
    if length_ratio < 0.3 or length_ratio > 3.0:
        flags.append("length_anomaly")
 
    structural_match = (
        shadow.count("\n-") == legacy.count("\n-")
        and shadow.count("\n#") == legacy.count("\n#")
    )
    if not structural_match:
        flags.append("structural_diff")
 
    sentiment_match = None  # Wire this up to your sentiment analyzer.
 
    return ComparisonResult(
        text_similarity=text_sim,
        length_ratio=length_ratio,
        sentiment_match=sentiment_match,
        structural_match=structural_match,
        flags=flags,
    )
 
if __name__ == "__main__":
    # Example output:
    # ComparisonResult(text_similarity=0.72, length_ratio=1.1, ...)
    r = compare_shadow_vs_legacy(
        shadow="## Response\nWe will refund the amount in full.\n- Refund confirmed",
        legacy="## Response\nWe will issue a full refund as requested.\n- Refund processed",
    )
    print(r)

Resist the urge to collapse these into a single score. A 70% similarity that also happens to be half the length probably lost something important. Keeping the axes separate lets you later distinguish "drifted but still on-topic" from "went off the rails."

Four pitfalls you will hit

A brief note on testing: before you roll shadow into production, simulate all four of these pitfalls in staging. Force the shadow path to throw, force it to be slow, point it at a broken API key, set the sampling rate to 100%. Watching how your production path behaves under each failure is the fastest way to make sure the isolation is real, not theoretical. I have seen teams discover the "silent latency leak" bug in production because nobody had actually tried saturating the shadow runner locally.

These are the ones I (and peers I trust) have actually stepped on when rolling out shadow mode.

The ordering matters — the earlier ones are cheaper to fix after the fact; the later ones can silently corrupt weeks of data if you get them wrong. Read all four before you write a single line of production-bound shadow code.

1. Cost blowouts — never shadow 100% of traffic

The first failure mode is exactly this: shadow everything, watch the API bill double, panic. If you skip the estimate, you learn the order of magnitude from the invoice. I went through that phase myself.

The fix is sampling. Drive a SHADOW_SAMPLE_RATE=0.05 (5%) knob from config and fire probabilistically. Even at 5%, a 10M request system produces 500k shadow decisions per month, which is more than enough for statistically meaningful accuracy estimates.

const sampleRate = Number(process.env.SHADOW_SAMPLE_RATE ?? "0.05");
if (Math.random() < sampleRate) {
  runShadow(ticket, legacyPromise).catch(() => {});
}

Before turning anything on, I estimate the monthly cost by hand. This script lays out the bill per sample rate. The unit prices are placeholders, so please swap in the current numbers from the official pricing page.

# shadow_budget.py — estimate the monthly cost of shadowing before you turn it on
# Prices are placeholders; check the official pricing page and swap in current values.
def monthly_cost_usd(requests_per_month, sample_rate, in_tokens, out_tokens,
                     usd_per_mtok_in, usd_per_mtok_out):
    n = requests_per_month * sample_rate
    return n * (in_tokens * usd_per_mtok_in + out_tokens * usd_per_mtok_out) / 1_000_000
 
if __name__ == "__main__":
    for rate in (1.0, 0.25, 0.05):
        c = monthly_cost_usd(10_000_000, rate, 800, 20, 1.0, 5.0)  # placeholder $1 / $5
        print(f"sample_rate={rate:>4}: ${c:>9,.0f} / month ({10_000_000*rate:,.0f} requests)")

Running it gives this (800 input and 20 output tokens per request, placeholder prices of $1 / $5 per million tokens):

sample_rate= 1.0: $    9,000 / month (10,000,000 requests)
sample_rate=0.25: $    2,250 / month (2,500,000 requests)
sample_rate=0.05: $      450 / month (500,000 requests)

Cut the rate to a twentieth and the bill drops to a twentieth. It sounds obvious, but putting this table in front of the team made "we don't need all of it" an easy agreement.

Layer stratified sampling on top of that. If new users and returning users have very different category distributions, sample both populations evenly so neither dominates the dataset.

2. Log pollution — don't contaminate your production alerts

If shadow errors and format violations flow into the same ERROR stream as production, the operations team will start investigating real incidents that aren't real. Trust erodes fast.

Segregate shadow telemetry into a dedicated namespace or stream. On Datadog, give shadow its own service name. On OpenTelemetry, tag the resource with environment: shadow. Make sure no alert on the production error rate fires from shadow activity.

3. Non-determinism of the model

Even at temperature 0, Claude is not bit-for-bit reproducible. The same input can produce slightly different outputs across weeks. Don't panic when yesterday's 95% match rate is 92% today.

Always compare shadow against the current legacy result for the same input in the same request. Do not compare across time. Also, annotate your charts with the dates you changed the prompt — the 2–3 days after a change are inherently noisy and not the right window to judge a rollback.

4. Silent latency leaks

Even with fire-and-forget, your shadow agent can run out of HTTP connections and back-pressure the production path. In Node.js, use a dedicated Agent instance, or — better — move shadow execution into a separate worker entirely.

My rule: shadow never runs in the same process as production. A queue-based dispatch (Cloudflare Queues, SQS, Kafka) to a dedicated consumer isolates resource usage completely.

// producer.ts — inside the production service: only enqueue.
await env.SHADOW_QUEUE.send({
  ticket,
  legacyResult,
  enqueuedAt: Date.now(),
});
 
// consumer.ts — in a dedicated worker.
export default {
  async queue(batch: MessageBatch, env: Env) {
    for (const message of batch.messages) {
      try {
        // legacyResult is already settled; runShadow accepts a value or a Promise
        await runShadow(message.body.ticket, Promise.resolve(message.body.legacyResult));
        message.ack();
      } catch (err) {
        message.retry({ delaySeconds: 60 });
      }
    }
  },
};

Queues can redeliver the same message, so key the stored row on ticketId + prompt_hash and upsert. Otherwise every retry inflates the denominator of your match rate.

Two months in, when production traffic doubles, you will be very glad the shadow workload cannot touch the production path's CPU, memory, or socket budget.

How many samples before you can trust "92%"

Once I wrote "92% match rate" into the promotion criteria, the denominator was the first thing that bothered me. A 94% on 50 tickets might be a true 94%, or it might be a lucky 50.

So I compute the Wilson lower bound of the observed rate and ask how many samples it takes before I can say the true rate is at least 92%.

# shadow_sample_size.py — how many samples before "92% match rate" is believable (Wilson lower bound)
from math import sqrt
 
def wilson_lower(matches, n, z=1.96):
    if n == 0:
        return 0.0
    p = matches / n
    denom = 1 + z * z / n
    centre = p + z * z / (2 * n)
    margin = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (centre - margin) / denom
 
if __name__ == "__main__":
    # Given an observed 94% match rate, what can we claim about the true rate at n samples?
    for n in (50, 200, 1000, 5000):
        m = round(n * 0.94)
        print(f"n={n:>5}  observed 94%  95% lower bound = {wilson_lower(m, n):.1%}")
    # Smallest n whose lower bound clears 92% (observed rate fixed at 94%)
    n = 50
    while wilson_lower(round(n * 0.94), n) < 0.92:
        n += 10
    print("minimum n for lower bound > 92% ≈", n)

The output:

n=   50  observed 94%  95% lower bound = 83.8%
n=  200  observed 94%  95% lower bound = 89.8%
n= 1000  observed 94%  95% lower bound = 92.4%
n= 5000  observed 94%  95% lower bound = 93.3%
minimum n for lower bound > 92% ≈ 690

Even at an observed 94%, 200 samples only guarantee 89.8%. My promotion deck now shows the lower bound and the count next to the match rate. A number only becomes evidence once it travels with its denominator. I repeat that sentence to myself before every promotion meeting.

For reference, 40,000 tickets a month at 10% sampling gives about 4,000 samples in a month — enough to see which categories are weak.

The promotion ladder — shadow → canary → full

A common question at this stage is whether to combine shadow with feature flags. The answer is yes, and it matters. The cleanest rollout I have seen uses three flags: shadow_enabled (on/off), shadow_sample_rate (0.0–1.0), and canary_percent (0–100). All three are server-side flags readable without a redeploy. This separation means you can quickly throttle shadow for a cost spike without touching canary, or freeze canary at a specific percentage while collecting more shadow data.

Before we talk about promotion, one reminder: every team I have worked with has been tempted to skip shadow and go straight to canary once offline eval "looks good enough." The temptation is real — shadow costs money and produces no direct business value. In my experience, every time we skipped shadow we paid for it later in an incident postmortem. The one-time cost of shadow has always been smaller than the compound cost of "we shipped an agent that was wrong on a predictable pattern our eval set didn't cover."

Once shadow has accumulated enough data, promotion is staged. The thresholds I actually use:

• Shadow (1–4 weeks): 92% match rate (or human-reviewed pass rate) stable for two consecutive weeks
• Canary (1–2 weeks): step traffic 1% → 5% → 10%, monitoring CSAT and complaint rate for regressions
• Full rollout: if, at 10%, operational metrics (resolution time, reopen rate) are at least on par with the legacy baseline

These numbers are not universal. For a B2C high-volume product, 92% is fine. For finance or healthcare, don't promote under 98% — the cost of an incorrect decision is too high. Work backward from "cost of failure × number of failures expected" to pick your bar.

One underrated rule: never stop shadow during canary. The 99% of traffic that the canary didn't take should still be shadowed at the same rate. Otherwise, you lose the ability to spot post-promotion sampling bias.

Final promotion should involve human sign-off, not auto-promotion. The decision usually needs CS, legal, and leadership to agree, and those conversations are rarely captured in a single number. The numbers trigger the meeting; they don't replace the meeting.

Metrics and alerts worth wiring up

Shadow mode without discipline becomes a data graveyard. Decide up front who owns the dashboards, who reviews the Friday samples, and who has authority to pause traffic promotion. These responsibilities should not rotate weekly — accuracy intuition compounds, and you want the same person watching the trendlines across the whole rollout.

Shadow data without dashboards is wasted. The four views I always put up:

1. Time-series match rate (daily moving average) with vertical lines marking prompt and model changes. You want "why did accuracy move?" answered at a glance.
2. Per-category drift. Category-level match rates usually vary wildly; you want to know which slice is weak and focus prompt or few-shot work there.
3. Cost-versus-value. In shadow, this is pure burn. After canary, compare against operator-hours saved. $3k/month of shadow cost saving 500 agent-hours is a trivially good trade; surface that explicitly.
4. Format violation rate. This is the "agent returned something we didn't expect" counter. If it exceeds 5%, the prompt's output spec needs tightening.

For alerts, I set three: three-day rolling match-rate drop below threshold, format-violation rate spike, and cost over 120% of budget. One-day spikes fire too often to be useful; always smooth over a window.

Case study — what a month of shadow actually looks like

This section describes one deployment, so treat the specific numbers as illustrative rather than universal. The shape of the curve, however, has been consistent across every team I have helped do this.

Theory is cheap; here's what actually happened on a one-month run. Monthly ticket volume was about 40,000, shadow sampling at 10%.

Week 1's match rate was 76%. Lower than we hoped, and the team's initial reaction was "maybe this doesn't work." But when we hand-reviewed about 100 mismatches, roughly 35% of them were cases where the agent was actually more accurate than the legacy rule-based classifier. The rule-based system was dumping many tickets into other that the agent was correctly classifying as technical or billing. That finding flipped our metric: we started tracking "human-judged correctness" in addition to raw match rate.

From week 2 we iterated on the prompt, and watched a weekly dashboard with four panels: overall match rate, per-category match rate, 20 hand-reviewed mismatches every Friday, and weekly cost on a Haiku 4.5 basis.

The Friday 20-sample review turned out to be the most valuable piece. Patterns invisible in aggregate stats — extremely short subjects, multi-lingual tickets — jumped out once humans read them. Feeding those into the prompt as few-shot examples lifted match rate to 88% and human-judged accuracy to 95% by week 3. What helped first, I realised later, was the hour spent reading twenty tickets with my own eyes, more than any number on the dashboard.

In week 4 we convened the promotion meeting. The numbers cleared the bar, but legal and CS flagged concerns about escalation paths for incorrect classifications, and we extended shadow for another two weeks. I count that as a win, not a failure. Shadow mode provides the ground for the debate, not the debate itself.

Implementation checklist and next steps

If you are working with a team rather than alone, I recommend putting this article (or the ideas in it) in front of whoever owns the rollout before you start coding. Shadow mode is a small design decision with large organizational implications: it changes how release decisions are made, how incidents are investigated, and how cost is budgeted. The sooner those implications are shared, the less likely it is that the pattern gets pushed back on during the first canary meeting.

A minimum practical path to stand shadow mode up in about a week:

• Identify the existing decision function or endpoint, and type both its input and output.
• Stand up a Claude Agent SDK implementation that already passes 80%+ on your eval set.
• Wrap it in a fire-and-forget shadow runner, and confirm production latency is unchanged.
• Pick a sample rate (start at 5%), give shadow its own log namespace, and estimate the monthly cost with shadow_budget.py.
• Create a comparison store (BigQuery, ClickHouse, or a dedicated table) and log the prompt version with every row.
• Make a weekly habit of reviewing match rate, cost, and the top 10 mismatches.
• After 2–4 weeks, schedule the promotion meeting once the lower bound of your match rate clears 92%.

The cheapest place to start is the "look at match rate every morning" habit. That single daily glance will change how your team thinks about the agent's real behavior in production — without any customer ever noticing.

Once shadow mode is in place, pair it with cost optimization patterns for Claude API in production and idempotency design for the Claude Agent SDK. Getting idempotency right before promoting the canary makes incident recovery dramatically easier once the agent is live.