Managing Claude API Prompts as Code: Registry, Versioning, and A/B Testing in Production

Defining the prompt schema with Zod

Each registry entry has the following shape, designed to feed directly into messages.create from the Anthropic SDK.

// src/prompts/types.ts
import { z } from "zod";
 
export const PromptVariantSchema = z.object({
  id: z.string().regex(/^v\d+(_[a-z0-9-]+)?$/), // e.g. v3, v3_a, v3_b
  weight: z.number().int().min(0).max(100),     // total of all non-deprecated variants must equal 100
  status: z.enum(["draft", "canary", "stable", "deprecated"]),
  systemPrompt: z.string().min(1),
  messages: z.array(z.object({
    role: z.enum(["user", "assistant"]),
    content: z.string(),
  })),
  model: z.enum([
    "claude-opus-5",
    "claude-sonnet-5",
    "claude-opus-4-6",
    "claude-sonnet-4-6",
    "claude-haiku-4-5-20251001",
  ]),
  maxTokens: z.number().int().positive().max(64000),
  temperature: z.number().min(0).max(1).default(0.7),
  inputSchema: z.record(z.any()).default({}),
  // Do not put .default() inside and .default({}) outside — see the measured section below
  slo: z.object({
    p95LatencyMs: z.number().positive(),
    minSuccessRate: z.number().min(0).max(1),
    maxCostPerCallUsd: z.number().positive(),
  }).default({ p95LatencyMs: 8000, minSuccessRate: 0.97, maxCostPerCallUsd: 0.05 }),
  createdAt: z.string().datetime(),
  createdBy: z.string(),
});
 
export const PromptEntrySchema = z.object({
  promptId: z.string().regex(/^[a-z][a-z0-9-]*$/),
  description: z.string(),
  variants: z.array(PromptVariantSchema).min(1),
});
 
export type PromptEntry = z.infer<typeof PromptEntrySchema>;
export type PromptVariant = z.infer<typeof PromptVariantSchema>;

The key design decision is separating weight from status. weight controls traffic share, status controls the lifecycle stage. A canary variant typically starts at weight: 5. Once metrics stay green long enough, you promote it to stable and demote the previous stable to deprecated. Keeping these axes orthogonal is what makes staged rollouts coherent.

One more contract to settle before writing any selection logic: of the four statuses, only canary and stable may ever receive live traffic. Put that rule in exactly one place. If you spread it across the registry loader and the selector, the two copies will drift, and a half-finished draft variant will end up in front of real users. Mine did.

// src/prompts/types.ts (appended)
export const SERVABLE_STATUSES = ["canary", "stable"] as const;
export const isServable = (v: PromptVariant) =>
  (SERVABLE_STATUSES as readonly string[]).includes(v.status);

Implementing the registry

You can keep all prompts in a single file or one file per prompt. I strongly recommend the latter. Pull request reviews stay small, blame is precise, and rebases on shared changes are painless.

// prompts/article-summarizer.json
{
  "promptId": "article-summarizer",
  "description": "Summarize the opening of a blog article in three lines",
  "variants": [
    {
      "id": "v2",
      "weight": 95,
      "status": "stable",
      "model": "claude-haiku-4-5-20251001",
      "maxTokens": 256,
      "temperature": 0.3,
      "systemPrompt": "You are a senior editor. Produce a 3-line summary that helps the reader decide whether to open the article.",
      "messages": [
        { "role": "user", "content": "Summarize the following article in three lines:\n\n{{ article }}" }
      ],
      "inputSchema": { "article": "string" },
      "slo": { "p95LatencyMs": 4000, "minSuccessRate": 0.98, "maxCostPerCallUsd": 0.005 },
      "createdAt": "2026-03-12T09:00:00Z",
      "createdBy": "ops@example.com"
    },
    {
      "id": "v3_canary",
      "weight": 5,
      "status": "canary",
      "model": "claude-haiku-4-5-20251001",
      "maxTokens": 256,
      "temperature": 0.2,
      "systemPrompt": "You are a senior editor. Produce a 3-line summary whose first line is the conclusion that hooks the reader.",
      "messages": [
        { "role": "user", "content": "Summarize the following article in three lines. The first line must state the conclusion:\n\n{{ article }}" }
      ],
      "inputSchema": { "article": "string" },
      "slo": { "p95LatencyMs": 4000, "minSuccessRate": 0.98, "maxCostPerCallUsd": 0.005 },
      "createdAt": "2026-04-25T09:00:00Z",
      "createdBy": "ops@example.com"
    }
  ]
}

Load the registry at module init time and validate it eagerly.

// src/prompts/registry.ts
import fs from "node:fs";
import path from "node:path";
import { PromptEntrySchema, isServable, type PromptEntry } from "./types";
 
const PROMPTS_DIR = path.join(process.cwd(), "prompts");
 
export function loadRegistry(): Map<string, PromptEntry> {
  const registry = new Map<string, PromptEntry>();
  const files = fs.readdirSync(PROMPTS_DIR).filter((f) => f.endsWith(".json"));
 
  for (const file of files) {
    const raw = JSON.parse(fs.readFileSync(path.join(PROMPTS_DIR, file), "utf8"));
    const parsed = PromptEntrySchema.parse(raw); // throws on schema violation
    // Only servable statuses (canary / stable) count toward 100. Never include draft.
    const totalWeight = parsed.variants
      .filter(isServable)
      .reduce((acc, v) => acc + v.weight, 0);
    if (totalWeight !== 100) {
      throw new Error(`[${parsed.promptId}] servable weights sum to ${totalWeight}, expected 100`);
    }
    const strays = parsed.variants.filter((v) => !isServable(v) && v.weight !== 0);
    if (strays.length > 0) {
      throw new Error(
        `[${parsed.promptId}] non-servable variants still carry weight: ${strays.map((v) => v.id).join(", ")}`,
      );
    }
    registry.set(parsed.promptId, parsed);
  }
  return registry;
}

Refusing to start when the active weights do not sum to 100 is the single most useful invariant in this system. Without it you eventually ship a configuration where 5% of users silently fall into a "no variant chosen" hole, and you spend days hunting for a ghost bug.

Note that the filter is isServable, not status !== "deprecated". The looser version lets a draft variant's weight count as part of the 100, which produces the worst possible combination: the build passes because the sum is correct, and an unreviewed prompt is live.

The Selector — sticky bucketing on a stable identity

The worst possible A/B implementation is to roll a fresh random number on every request. The same user sees a different variant on consecutive turns, the conversation becomes incoherent, and the experiment's data is meaningless. Use a stable identity — userId, account id, or device id — and bucket it deterministically.

// src/prompts/selector.ts
import crypto from "node:crypto";
import { isServable, type PromptEntry, type PromptVariant } from "./types";
 
function stableHash(input: string): number {
  // Returns 0..99 deterministically
  const hash = crypto.createHash("sha256").update(input).digest();
  return hash.readUInt32BE(0) % 100;
}
 
export function selectVariant(
  entry: PromptEntry,
  identityKey: string,
  options?: { forceVariantId?: string; disabled?: ReadonlySet<string> },
): PromptVariant {
  if (options?.forceVariantId) {
    const forced = entry.variants.find((v) => v.id === options.forceVariantId);
    if (forced) return forced;
  }
 
  // Servable statuses only, minus anything the Guardian has killed at runtime
  const candidates = entry.variants.filter(
    (v) => isServable(v) && !options?.disabled?.has(`${entry.promptId}:${v.id}`),
  );
  if (candidates.length === 0) {
    throw new Error(`[${entry.promptId}] no eligible variants`);
  }
 
  // A runtime exclusion drops the sum below 100, so project the bucket onto what remains
  const totalWeight = candidates.reduce((acc, v) => acc + v.weight, 0);
  const scaled = (stableHash(`${entry.promptId}::${identityKey}`) / 100) * totalWeight;
  let cumulative = 0;
  for (const v of candidates) {
    cumulative += v.weight;
    if (scaled < cumulative) return v;
  }
  return candidates[candidates.length - 1]!;
}

A practical rule: build the identity key as ${userId}:${experimentName}, not just ${userId}. Otherwise unrelated experiments on the same user end up correlated, which contaminates your A/B analysis.

The important structural choice is that disabled lives inside the selection predicate rather than being applied by the caller afterwards. A Guardian that sets a kill switch the Selector never reads stops exactly zero requests. I shipped that version. There are measurements further down.

Renderer and API call

Use any templating engine you like — Mustache or a small regex helper both work — but always throw on unbound variables. A leftover {{ article }} reaching the user is a hard failure mode worth crashing for.

// src/prompts/renderer.ts
export function render(template: string, vars: Record<string, string>): string {
  return template.replace(/\{\{\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*\}\}/g, (_, key) => {
    if (!(key in vars)) {
      throw new Error(`Unbound template variable: ${key}`);
    }
    return vars[key]!;
  });
}

For the actual call, the Anthropic SDK is enough. The non-obvious requirement is that every response must be persisted alongside the variantId that produced it. Without that join key, you lose the ability to attribute regressions later.

// src/prompts/invoke.ts
import Anthropic from "@anthropic-ai/sdk";
import { loadRegistry } from "./registry";
import { selectVariant } from "./selector";
import { render } from "./renderer";
import { recordTelemetry } from "./telemetry";
 
const client = new Anthropic();
const registry = loadRegistry();
 
export async function invokePrompt(params: {
  promptId: string;
  identityKey: string;
  vars: Record<string, string>;
  forceVariantId?: string;
}) {
  const entry = registry.get(params.promptId);
  if (!entry) throw new Error(`Unknown promptId: ${params.promptId}`);
 
  const variant = selectVariant(entry, params.identityKey, {
    forceVariantId: params.forceVariantId,
  });
 
  const renderedMessages = variant.messages.map((m) => ({
    role: m.role,
    content: render(m.content, params.vars),
  }));
 
  const startedAt = Date.now();
  let usage: Anthropic.Messages.Usage | undefined;
  let success = false;
 
  try {
    const res = await client.messages.create({
      model: variant.model,
      max_tokens: variant.maxTokens,
      temperature: variant.temperature,
      system: variant.systemPrompt,
      messages: renderedMessages,
    });
    usage = res.usage;
    success = true;
    return { promptId: entry.promptId, variantId: variant.id, output: res.content, usage };
  } catch (err) {
    success = false;
    throw err;
  } finally {
    await recordTelemetry({
      promptId: entry.promptId,
      variantId: variant.id,
      model: variant.model,
      identityKey: params.identityKey,
      latencyMs: Date.now() - startedAt,
      success,
      usage,
    });
  }
}

The telemetry call lives in finally deliberately. It is precisely the failed calls that the Guardian needs to see — if you only log success, your detector blinds itself to the cases that matter most.

Telemetry on Cloudflare Workers + D1

In my own production setup the telemetry sink is a Cloudflare D1 table queried from a Worker. D1 is small, fast, and supports enough SQL to run real analytics. You do not need a separate analytics warehouse to start.

// src/prompts/telemetry.ts
import { env } from "cloudflare:workers"; // DB and PROMPT_FLAGS are bound in wrangler.toml
import type { Anthropic } from "@anthropic-ai/sdk";
 
interface TelemetryRecord {
  promptId: string;
  variantId: string;
  model: string;        // required to price the call — pass variant.model straight through
  identityKey: string;
  latencyMs: number;
  success: boolean;
  usage?: Anthropic.Messages.Usage;
  feedbackScore?: number;
}
 
export async function recordTelemetry(rec: TelemetryRecord) {
  const inputTokens = rec.usage?.input_tokens ?? 0;
  const outputTokens = rec.usage?.output_tokens ?? 0;
  const costUsd = estimateCostUsd(rec.model, inputTokens, outputTokens);
 
  await env.DB.prepare(
    `INSERT INTO prompt_invocations
     (prompt_id, variant_id, model, identity_key, latency_ms, success, input_tokens, output_tokens, cost_usd, created_at)
     VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
  )
    .bind(
      rec.promptId,
      rec.variantId,
      rec.model,
      rec.identityKey,
      rec.latencyMs,
      rec.success ? 1 : 0,
      inputTokens,
      outputTokens,
      costUsd,
      new Date().toISOString(),
    )
    .run();
}
 
// Published list prices, USD per million tokens. Adding a model means editing this table only.
const PRICE_USD_PER_MTOK: Record<string, { input: number; output: number }> = {
  "claude-opus-5": { input: 5, output: 25 },
  "claude-sonnet-5": { input: 2, output: 10 },
  "claude-opus-4-6": { input: 5, output: 25 },
  "claude-sonnet-4-6": { input: 3, output: 15 },
  "claude-haiku-4-5-20251001": { input: 1, output: 5 },
};
 
function estimateCostUsd(model: string, inputTokens: number, outputTokens: number) {
  const price = PRICE_USD_PER_MTOK[model];
  // Never quietly return 0 — an unknown model should be loud
  if (!price) throw new Error(`No price entry for model: ${model}`);
  return (inputTokens * price.input + outputTokens * price.output) / 1_000_000;
}

import { env } from "cloudflare:workers" landed in Workers in March 2025. Before that, env arrived only as a handler argument, so reaching a binding from a deeply nested function meant threading it through every call in between. One caveat worth knowing: you cannot call KV or D1 methods in top-level scope. Import the object at module level, but keep the actual prepare() and put() calls inside a request — which is what the code above does.

Storing identity_key may feel optional today; six months from now you will be glad you did. Per-user regression detection and repeat-usage cohorts both depend on it.

The Guardian — automated rollback when SLOs break

The final piece is the cron-driven Guardian. It aggregates each canary variant's metrics over a 15-minute window and flips a kill-switch flag if the SLO is breached.

// src/prompts/guardian.ts
import { env } from "cloudflare:workers";
import { loadRegistry } from "./registry";
 
interface VariantStats {
  count: number;
  successRate: number;
  p95LatencyMs: number;
  avgCostUsd: number;
}
 
async function fetchStats(promptId: string, variantId: string, sinceIso: string): Promise<VariantStats> {
  const result = await env.DB.prepare(
    `SELECT
       COUNT(*) AS count,
       AVG(CASE WHEN success = 1 THEN 1.0 ELSE 0.0 END) AS success_rate,
       AVG(cost_usd) AS avg_cost
     FROM prompt_invocations
     WHERE prompt_id = ? AND variant_id = ? AND created_at >= ?`
  )
    .bind(promptId, variantId, sinceIso)
    .first<{ count: number; success_rate: number; avg_cost: number }>();
 
  // D1 has no PERCENTILE function, so approximate p95 with ORDER BY + OFFSET
  const p95Row = await env.DB.prepare(
    `SELECT latency_ms FROM prompt_invocations
     WHERE prompt_id = ? AND variant_id = ? AND created_at >= ?
     ORDER BY latency_ms ASC
     LIMIT 1 OFFSET CAST((SELECT COUNT(*) FROM prompt_invocations
       WHERE prompt_id = ? AND variant_id = ? AND created_at >= ?) * 0.95 AS INTEGER)`
  )
    .bind(promptId, variantId, sinceIso, promptId, variantId, sinceIso)
    .first<{ latency_ms: number }>();
 
  return {
    count: result?.count ?? 0,
    successRate: result?.success_rate ?? 1,
    p95LatencyMs: p95Row?.latency_ms ?? 0,
    avgCostUsd: result?.avg_cost ?? 0,
  };
}
 
export async function evaluateAndRollback() {
  const registry = loadRegistry();
  const sinceIso = new Date(Date.now() - 15 * 60 * 1000).toISOString();
 
  for (const entry of registry.values()) {
    const canary = entry.variants.find((v) => v.status === "canary");
    if (!canary) continue;
 
    const stats = await fetchStats(entry.promptId, canary.id, sinceIso);
    if (stats.count < 50) continue; // not enough data yet
 
    const slo = canary.slo;
    const violations: string[] = [];
    if (stats.successRate < slo.minSuccessRate) {
      violations.push(`successRate ${stats.successRate.toFixed(3)} < ${slo.minSuccessRate}`);
    }
    if (stats.p95LatencyMs > slo.p95LatencyMs) {
      violations.push(`p95 ${stats.p95LatencyMs}ms > ${slo.p95LatencyMs}ms`);
    }
    if (stats.avgCostUsd > slo.maxCostPerCallUsd) {
      violations.push(`avgCost $${stats.avgCostUsd.toFixed(4)} > $${slo.maxCostPerCallUsd}`);
    }
 
    if (violations.length > 0) {
      console.warn(`[GUARDIAN] ${entry.promptId}/${canary.id} SLO violation: ${violations.join(", ")}`);
      await rollback(entry.promptId, canary.id);
    }
  }
}
 
async function rollback(promptId: string, variantId: string) {
  // Set a "disabled" flag in KV
  await env.PROMPT_FLAGS.put(`disabled:${promptId}:${variantId}`, "1", { expirationTtl: 86400 });
}
 
// --- The read side. Without this, nothing ever consults the flag ---
export async function loadDisabledVariants(): Promise<ReadonlySet<string>> {
  const listed = await env.PROMPT_FLAGS.list({ prefix: "disabled:" });
  return new Set<string>(
    listed.keys.map((k: { name: string }) => k.name.slice("disabled:".length)),
  );
}

On the request path, fetch loadDisabledVariants() once per request and hand it to the Selector. A KV list costs tens of milliseconds, so on Workers it is worth holding the result in memory for ~30 seconds and refreshing it via ctx.waitUntil.

// src/prompts/invoke.ts (excerpt)
let cache: { at: number; set: ReadonlySet<string> } | null = null;
 
async function disabledSet(): Promise<ReadonlySet<string>> {
  if (cache && Date.now() - cache.at < 30_000) return cache.set;
  const set = await loadDisabledVariants();
  cache = { at: Date.now(), set };
  return set;
}
 
const variant = selectVariant(entry, identityKey, { disabled: await disabledSet() });

The non-obvious decision here is not editing the JSON weight directly during a rollback. I wanted Git and prod to stay in sync, but writing a Git commit at 3 a.m. while CI is restarting only widens the blast radius. Treat rollback as immediate damage control via a runtime flag, then file a follow-up Pull Request the next morning to reflect the change in the registry.

Four silent failures I found by running this code again

I reinstalled the code from the original version of this article on Node.js 22.23.2 and put real traffic through it. Four defects surfaced, and every one of them disabled the Guardian without raising an exception or writing a single log line. The blocks above are already corrected; here is why each one happened, because a fix you do not understand comes back.

1. .default({}) does not apply inner defaults in Zod 4

The slo field was originally written like this.

slo: z.object({
  p95LatencyMs: z.number().positive().default(8000),
  minSuccessRate: z.number().min(0).max(1).default(0.97),
  maxCostPerCallUsd: z.number().positive().default(0.05),
}).default({}),

Parsing a variant that omits slo produces different results on Zod 3 and Zod 4.

In Zod 4 the value passed to an outer .default(value) is returned as-is and never re-parsed through the inner schema. A variant missing slo therefore ends up with slo.p95LatencyMs === undefined, and all three Guardian comparisons evaluate to false. The SLO breach is not "missed" — the comparison never happens at all, silently.

What makes this genuinely nasty is that the TypeScript types stay clean. z.infer treats defaulted fields as non-optional, so tsc never says a word. The places where you trust your types the most are exactly where this hides.

Two fixes work: spell out every value in the outer .default() (what the code above now does), or use Zod 4's .prefault() when you specifically want the old behavior. I went with the former, because having the SLO defaults sitting in one visible place makes them reviewable.

2. draft variants were being served to production traffic

The candidate filter was status !== "deprecated", which leaves draft in. Here is what 100,000 requests looked like against a registry holding stable 80 / canary 5 / draft 15 — a single unfinished variant left in the file.

An unreviewed prompt reached 15% of real users. And because loadRegistry validated the total with the same status !== "deprecated" predicate, the weights summed to exactly 100 and the build passed without complaint. The safeguard shared the bug it was supposed to catch. That is the part that stung.

The small shift in canary share (5.11% → 5.03%) comes from projecting the bucket onto the post-exclusion total. The gap to the designed 5% is 0.03 points.

3. The Guardian's kill switch was written but never read

rollback() wrote disabled:{promptId}:{variantId} into KV, and the comment claimed "the Selector consults this on every request." No such code existed anywhere in the Selector.

So the Guardian would detect the breach, log [GUARDIAN] ... SLO violation correctly, and write to KV successfully. Read the logs and automated rollback looks healthy. Not one request stops. A convincing log line is the only artifact this class of bug leaves behind.

After the fix, killing the canary and replaying 100,000 requests routed 0 requests to it — and 0 users outside the canary bucket changed variants. Sticky bucketing survives the exclusion. An emergency stop has to be judged on both counts: that it stops, and that it does not drag bystanders with it. Measure both before you trust it in production.

Working as an indie developer, I am the only reviewer this layer ever gets. That is precisely why the "looks healthy" state has to be challenged by the code itself rather than by a second pair of eyes.

4. The cost estimator ignored the model, so the cost guard never fired

The last one I noticed while simply re-reading the signature. estimateCostUsd accepted _promptId and _variantId and used neither. The body was two hardcoded constants for a single model — and those constants had fallen a generation behind the published price list.

Haiku 4.5 lists at $1.00 input / $5.00 output per million tokens. The old code's $0.25 / $1.25 is a quarter of that. Recomputed at 2,400 input and 220 output tokens per call — an unremarkable load for a summarizer — the picture looks like this:

Point a canary at a larger model and the real cost per call jumps fivefold. The recorded value does not move at all, because it is computed from one model's outdated rate no matter what variant.model says. maxCostPerCallUsd is one of the Guardian's three checks, and in this state it was a condition that could never fire from the day it was written. Runaway cost shows up in neither success rate nor latency. The invoice is the first notification you get.

The fix routes variant.model through to Telemetry and looks the price up in a Record<string, {input, output}>. An unknown model throws rather than returning 0. I added a model column to prompt_invocations for the same reason: I wanted to be able to recompute a row later and know which generation of pricing produced it.

A function that quietly returns zero opens a hole on the monitoring side. Whether to return a default or to throw comes down to who reads the value — this one changed my mind about that.

The common thread is that all four failures are quiet. Exceptions get noticed. ERROR lines get noticed. A safety mechanism that disables itself in silence is only findable if you deliberately go looking. So when you build the Guardian, ship a variant with an intentionally broken SLO on the same day and confirm end to end that traffic actually stops. Fold that step into Day 5-6 of the plan above.

Common pitfalls

1. Logging success/failure but not variantId

If the only thing you log is whether a call succeeded, you cannot tell which variant caused the regression. Aggregate metrics become useless and your rollback signal evaporates.

2. Bumping both prompt and model in the same canary

When you ship a new prompt it is tempting to also try a more capable model "while you're at it". Don't. You will not be able to attribute the change in metrics to one or the other. Keep model fixed during canary evaluation; ship a model migration as its own experiment.

3. Starting at 50/50

A new prompt feels exciting and you want it widely tested. The flip side is that 50% of users absorb any regression. Cap canary at 5–10% until a full evaluation window passes.

4. Forgetting to escape user input

Raw user input rendered into {{ article }} is a prompt injection vector. Wrap it in [USER_INPUT_BEGIN]…[USER_INPUT_END] blocks and have the system prompt explicitly state that anything between those markers is data, not instructions. See Prompt Injection Defense Production Patterns for the full playbook.

5. Promoting canary without demoting the old stable

I have shipped this bug. Two stable variants summing to weight 100 each will silently break the Selector. Enforce "exactly one stable per prompt" as a Zod refinement that fails CI, not as a checklist item in the PR template.

Metrics that actually drive decisions

The metric panel I rely on, in roughly the order you should add them:

• SLO basics: success rate, p95 latency, USD per call.
• Quality signals: thumbs-up/down feedback, day-2 retention, repeat-question rate within a session.
• Business signals: paid conversion rate, drop-off correlated with response time.

Repeat-question rate is the most underrated of the bunch. When a prompt regresses, it climbs cleanly. Combine it with prompt caching analysis and you get an honest picture of cost-per-quality. For caching specifically, see the Prompt Caching Cost Guide.

Migrating from inline prompts — a 7-day playbook

Most codebases I see keep prompts as template literals scattered across handlers. The path from there to a registry is shorter than it looks if you do it in stages.

Day 1–2 — extract and freeze. Find every messages.create(...) call and replace the inline strings with invokePrompt({ promptId: "..." }). Move the strings into JSON files under prompts/. At this point only one variant exists per prompt — v1, weight: 100, status: stable. No behavior changes; the goal is to make the next week's changes reviewable.

Day 3 — wire telemetry. Stand up the D1 (or Postgres, if you're outside Cloudflare) table. Run for 24 hours and confirm rows appear. You will already see useful signals: which prompts dominate cost, which have absurd p95 latency.

Day 4 — pick the lowest-risk experiment. Choose the prompt with the lowest user-visible impact and add a v2_canary variant with weight: 5. Watch the metrics for two days. The point is to exercise the pipeline, not to prove a hypothesis.

Day 5–6 — wire the Guardian cron. With one canary in flight, you can verify the rollback path works end-to-end. Force a rollback manually (set the kill-switch flag yourself) and confirm the Selector excludes the variant within seconds.

Day 7 — write the runbook. "Promotion" and "rollback" should each be a one-page document any engineer on the team can follow. The system is only as good as the runbook that explains it.

What to chart and what to alert on

Once telemetry is flowing, the first dashboard you should build has just five panels.

• Calls per minute, broken down by promptId × variantId. You will instantly see if the canary is getting the share you expect.
• Success rate, last 1 hour, broken down by variant. Sets the bar for the SLO threshold.
• p95 latency, last 1 hour, broken down by variant. A latency regression often precedes a quality regression by a few hours.
• USD spend per hour, broken down by variant. Make cost a first-class observability signal, not an afterthought you discover at month end.
• Error rate by error class. Group by 429, 529, 400, 5xx, plus a "timeout" bucket. A spike in 400s usually means the new prompt produced output the downstream system rejected.

For alerts, three rules cover most situations:

• Critical: success rate of any variant drops below slo.minSuccessRate for 10 consecutive minutes with at least 100 samples. Page someone.
• Warning: p95 latency exceeds slo.p95LatencyMs * 1.5 for 30 minutes. Slack notification, no page.
• Cost guard: hourly spend on any single variant exceeds 3× the 7-day rolling median. Slack notification.

Alerts that page on absolute thresholds get noisy fast. Pair every threshold with a sample-size guard so you do not wake up over a 10-call sample where one timeout looked catastrophic.

Handling failure modes that the Guardian cannot see

There is a category of regression the cron-based Guardian cannot catch: silent quality drops where success rate, latency, and cost all look fine, but the model is producing subtly worse content. For those, you need a human-in-the-loop review channel.

The lightest-weight version that works: every Nth response from a canary variant is fanned out to a Slack channel where you (or a designated reviewer) can react with 👍 or 👎. Store the reaction back in prompt_invocations.feedback_score and join it into the Guardian query. Once you have ~200 labeled samples, you can train the Guardian to also reject canaries whose human-rated quality is significantly below the stable variant's baseline.

This is also where I would warn against premature automation. You can build a complex LLM-as-judge evaluator that grades outputs automatically — and you should, eventually. But for the first three months of running a prompt registry, the cheapest signal that actually reflects "is this prompt better?" is a human looking at twenty samples a day.

Versioning strategy and the discipline of small variants

A natural temptation, once a registry exists, is to package every change into "the next big variant." Resist this. The variants that succeed in canary tend to be small, single-axis changes: one phrasing tweak, one structural rewrite, one new constraint. Variants that bundle five changes at once almost always regress on at least one dimension, and the post-mortem becomes guesswork.

A practical naming convention I follow:

• v3 — production stable.
• v3_canary_phrasing — testing only a wording change.
• v3_canary_structure — testing only a reordering of sections.
• v4 — promoted from a canary that won.

Yes, you can run multiple canaries simultaneously as long as their combined weight stays small (e.g. two canaries at 5% each, leaving stable at 90%). This works because of the sticky bucketing: each user is deterministically routed to one variant, never two, and the Guardian evaluates each variant independently. What you must not do is overlap canaries on the same promptId × identityKey axis without realizing it — that breaks the independence assumption your statistics rely on.

Reproducibility — replaying a regression six months later

The reason you persist variantId alongside every output is that, six months from now, someone will report "the summarizer produced something weird in this article" and you will want to reproduce it exactly. Three pieces of information let you do that.

• The prompt body that ran (recoverable from the registry's Git history at the timestamp of the call).
• The model and parameters (model, temperature, maxTokens — recoverable from the same Git snapshot).
• The exact rendered prompt (worth storing in prompt_invocations.rendered_prompt if your privacy policy permits).

I have gone back and forth on whether to store the rendered prompt. The argument for: replays become trivial and you can build a high-quality regression test set from real production calls. The argument against: the rendered prompt may contain user data and creates a longer retention surface. The middle ground that worked for me is hashing the rendered prompt and only storing the hash by default, with full text retention behind a feature flag for staging environments and short-lived production debug windows.

Coordinating prompt rollouts with model upgrades

Anthropic ships new models on its own cadence. The day a new Claude model becomes available, you do not want to ship "the new prompt and the new model" as one bundle. The clean sequence is:

1. Phase A — model migration only. Keep the prompt fixed. Add a canary variant whose only difference from stable is model: claude-opus-4-7. Watch metrics for a week. Promote when stable.
2. Phase B — prompt experiment. With the model now stable, layer a prompt-only canary on top.

This sequencing is not glamorous but it is the difference between knowing what changed and guessing. I have skipped Phase A under deadline pressure exactly once, and the resulting two-day investigation cost more than the deadline saved.

Two registry fields Phase A has to revisit

Swapping a model tempts you to edit the model string and stop there. But Claude 4.7 and later use a newer tokenizer that produces roughly 30% more tokens for the same text (Sonnet 4.6 and earlier use the previous one). Two fields in the registry schema are denominated in tokens.

The first is maxTokens. The article-summarizer variant sets it to 256. Under the newer tokenizer that same three-line summary may no longer fit. What comes back when it doesn't is not an exception — it's a truncated completion with stop_reason: "max_tokens". Success rate holds. The Guardian passes. A summary missing its third line keeps going out, and nothing flags it.

The second is slo.maxCostPerCallUsd. On list price alone, Sonnet 4.6 ($3 / $15) to Sonnet 5 ($2 / $10) reads as a 33% cut. Fold in the token increase and the real saving is smaller.

It is still a price cut. But a forecast built from the rate card alone leaves you unable to explain the gap when the invoice arrives. Comparing unit prices only holds if the token count is the same on both sides. In Phase A I now query prompt_invocations for the change in input_tokens on identical inputs before I look at cost at all — which is a single query once the model column exists.

Should you adopt a hosted SaaS instead?

A reasonable counter-question: "Is this worth building when Promptlayer, LangSmith, and Helicone exist?" My honest answer is that hosted tools are great for small teams that do not want to own this layer, and overkill for teams of one or two who are still finding product-market fit. The crossover point in my experience is around 100k API calls per month. Below that, a self-hosted registry on Cloudflare Workers + D1 costs nearly nothing and keeps your prompt history in your own repo. Above that, the dashboards and team features of a SaaS start to earn their license fee.

Either way, the architecture in this article maps cleanly onto the SaaS world: most hosted tools are essentially a Registry + Selector + Telemetry stack with a hosted UI on top. Understanding the layers first means you will not be surprised when you switch tools.

Closing — what to do this week

Thank you for reading this far. You do not have to build all six layers at once. The single highest-ROI step is extracting the prompt strings from your application code into a JSON file in your repo. That alone unlocks Git diffs, blame, and review. The Selector, Telemetry, and Guardian can land incrementally over the following weeks.

Hand-tweaking prompts works fine until your service grows. Past that point, it's a slow-burning incident waiting to happen. Treating prompts like code is what lets you experiment aggressively without flinching.