Managing Claude API Prompts as Code: Registry, Versioning, and A/B Testing in Production

Defining the prompt schema with Zod

Each registry entry has the following shape, designed to feed directly into messages.create from the Anthropic SDK.

// src/prompts/types.ts
import { z } from "zod";
 
export const PromptVariantSchema = z.object({
  id: z.string().regex(/^v\d+(_[a-z0-9-]+)?$/), // e.g. v3, v3_a, v3_b
  weight: z.number().int().min(0).max(100),     // total of all non-deprecated variants must equal 100
  status: z.enum(["draft", "canary", "stable", "deprecated"]),
  systemPrompt: z.string().min(1),
  messages: z.array(z.object({
    role: z.enum(["user", "assistant"]),
    content: z.string(),
  })),
  model: z.enum(["claude-opus-4-6", "claude-sonnet-4-6", "claude-haiku-4-5-20251001"]),
  maxTokens: z.number().int().positive().max(64000),
  temperature: z.number().min(0).max(1).default(0.7),
  inputSchema: z.record(z.any()).default({}),
  slo: z.object({
    p95LatencyMs: z.number().positive().default(8000),
    minSuccessRate: z.number().min(0).max(1).default(0.97),
    maxCostPerCallUsd: z.number().positive().default(0.05),
  }).default({}),
  createdAt: z.string().datetime(),
  createdBy: z.string(),
});
 
export const PromptEntrySchema = z.object({
  promptId: z.string().regex(/^[a-z][a-z0-9-]*$/),
  description: z.string(),
  variants: z.array(PromptVariantSchema).min(1),
});
 
export type PromptEntry = z.infer<typeof PromptEntrySchema>;
export type PromptVariant = z.infer<typeof PromptVariantSchema>;

The key design decision is separating weight from status. weight controls traffic share, status controls the lifecycle stage. A canary variant typically starts at weight: 5. Once metrics stay green long enough, you promote it to stable and demote the previous stable to deprecated. Keeping these axes orthogonal is what makes staged rollouts coherent.

Implementing the registry

You can keep all prompts in a single file or one file per prompt. I strongly recommend the latter. Pull request reviews stay small, blame is precise, and rebases on shared changes are painless.

// prompts/article-summarizer.json
{
  "promptId": "article-summarizer",
  "description": "Summarize the opening of a blog article in three lines",
  "variants": [
    {
      "id": "v2",
      "weight": 95,
      "status": "stable",
      "model": "claude-haiku-4-5-20251001",
      "maxTokens": 256,
      "temperature": 0.3,
      "systemPrompt": "You are a senior editor. Produce a 3-line summary that helps the reader decide whether to open the article.",
      "messages": [
        { "role": "user", "content": "Summarize the following article in three lines:\n\n{{ article }}" }
      ],
      "inputSchema": { "article": "string" },
      "slo": { "p95LatencyMs": 4000, "minSuccessRate": 0.98, "maxCostPerCallUsd": 0.005 },
      "createdAt": "2026-03-12T09:00:00Z",
      "createdBy": "masaki@dolice.design"
    },
    {
      "id": "v3_canary",
      "weight": 5,
      "status": "canary",
      "model": "claude-haiku-4-5-20251001",
      "maxTokens": 256,
      "temperature": 0.2,
      "systemPrompt": "You are a senior editor. Produce a 3-line summary whose first line is the conclusion that hooks the reader.",
      "messages": [
        { "role": "user", "content": "Summarize the following article in three lines. The first line must state the conclusion:\n\n{{ article }}" }
      ],
      "inputSchema": { "article": "string" },
      "slo": { "p95LatencyMs": 4000, "minSuccessRate": 0.98, "maxCostPerCallUsd": 0.005 },
      "createdAt": "2026-04-25T09:00:00Z",
      "createdBy": "masaki@dolice.design"
    }
  ]
}

Load the registry at module init time and validate it eagerly.

// src/prompts/registry.ts
import fs from "node:fs";
import path from "node:path";
import { PromptEntrySchema, type PromptEntry } from "./types";
 
const PROMPTS_DIR = path.join(process.cwd(), "prompts");
 
export function loadRegistry(): Map<string, PromptEntry> {
  const registry = new Map<string, PromptEntry>();
  const files = fs.readdirSync(PROMPTS_DIR).filter((f) => f.endsWith(".json"));
 
  for (const file of files) {
    const raw = JSON.parse(fs.readFileSync(path.join(PROMPTS_DIR, file), "utf8"));
    const parsed = PromptEntrySchema.parse(raw); // throws on schema violation
    const totalWeight = parsed.variants
      .filter((v) => v.status !== "deprecated")
      .reduce((acc, v) => acc + v.weight, 0);
    if (totalWeight !== 100) {
      throw new Error(`[${parsed.promptId}] active weights sum to ${totalWeight}, expected 100`);
    }
    registry.set(parsed.promptId, parsed);
  }
  return registry;
}

Refusing to start when the active weights do not sum to 100 is the single most useful invariant in this system. Without it you eventually ship a configuration where 5% of users silently fall into a "no variant chosen" hole, and you spend days hunting for a ghost bug.

The Selector — sticky bucketing on a stable identity

The worst possible A/B implementation is to roll a fresh random number on every request. The same user sees a different variant on consecutive turns, the conversation becomes incoherent, and the experiment's data is meaningless. Use a stable identity — userId, account id, or device id — and bucket it deterministically.

// src/prompts/selector.ts
import crypto from "node:crypto";
import type { PromptEntry, PromptVariant } from "./types";
 
function stableHash(input: string): number {
  // Returns 0..99 deterministically
  const hash = crypto.createHash("sha256").update(input).digest();
  return hash.readUInt32BE(0) % 100;
}
 
export function selectVariant(
  entry: PromptEntry,
  identityKey: string,
  options?: { forceVariantId?: string },
): PromptVariant {
  if (options?.forceVariantId) {
    const forced = entry.variants.find((v) => v.id === options.forceVariantId);
    if (forced) return forced;
  }
 
  const candidates = entry.variants.filter((v) => v.status !== "deprecated");
  if (candidates.length === 0) {
    throw new Error(`[${entry.promptId}] no eligible variants`);
  }
 
  const bucket = stableHash(`${entry.promptId}::${identityKey}`);
  let cumulative = 0;
  for (const v of candidates) {
    cumulative += v.weight;
    if (bucket < cumulative) return v;
  }
  return candidates[candidates.length - 1]!;
}

A practical rule: build the identity key as ${userId}:${experimentName}, not just ${userId}. Otherwise unrelated experiments on the same user end up correlated, which contaminates your A/B analysis.

Renderer and API call

Use any templating engine you like — Mustache or a small regex helper both work — but always throw on unbound variables. A leftover {{ article }} reaching the user is a hard failure mode worth crashing for.

// src/prompts/renderer.ts
export function render(template: string, vars: Record<string, string>): string {
  return template.replace(/\{\{\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*\}\}/g, (_, key) => {
    if (!(key in vars)) {
      throw new Error(`Unbound template variable: ${key}`);
    }
    return vars[key]!;
  });
}

For the actual call, the Anthropic SDK is enough. The non-obvious requirement is that every response must be persisted alongside the variantId that produced it. Without that join key, you lose the ability to attribute regressions later.

// src/prompts/invoke.ts
import Anthropic from "@anthropic-ai/sdk";
import { loadRegistry } from "./registry";
import { selectVariant } from "./selector";
import { render } from "./renderer";
import { recordTelemetry } from "./telemetry";
 
const client = new Anthropic();
const registry = loadRegistry();
 
export async function invokePrompt(params: {
  promptId: string;
  identityKey: string;
  vars: Record<string, string>;
  forceVariantId?: string;
}) {
  const entry = registry.get(params.promptId);
  if (!entry) throw new Error(`Unknown promptId: ${params.promptId}`);
 
  const variant = selectVariant(entry, params.identityKey, {
    forceVariantId: params.forceVariantId,
  });
 
  const renderedMessages = variant.messages.map((m) => ({
    role: m.role,
    content: render(m.content, params.vars),
  }));
 
  const startedAt = Date.now();
  let usage: Anthropic.Messages.Usage | undefined;
  let success = false;
 
  try {
    const res = await client.messages.create({
      model: variant.model,
      max_tokens: variant.maxTokens,
      temperature: variant.temperature,
      system: variant.systemPrompt,
      messages: renderedMessages,
    });
    usage = res.usage;
    success = true;
    return { promptId: entry.promptId, variantId: variant.id, output: res.content, usage };
  } catch (err) {
    success = false;
    throw err;
  } finally {
    await recordTelemetry({
      promptId: entry.promptId,
      variantId: variant.id,
      identityKey: params.identityKey,
      latencyMs: Date.now() - startedAt,
      success,
      usage,
    });
  }
}

The telemetry call lives in finally deliberately. It is precisely the failed calls that the Guardian needs to see — if you only log success, your detector blinds itself to the cases that matter most.

Telemetry on Cloudflare Workers + D1

In my own production setup the telemetry sink is a Cloudflare D1 table queried from a Worker. D1 is small, fast, and supports enough SQL to run real analytics. You do not need a separate analytics warehouse to start.

// src/prompts/telemetry.ts
import type { Anthropic } from "@anthropic-ai/sdk";
 
interface TelemetryRecord {
  promptId: string;
  variantId: string;
  identityKey: string;
  latencyMs: number;
  success: boolean;
  usage?: Anthropic.Messages.Usage;
  feedbackScore?: number;
}
 
export async function recordTelemetry(rec: TelemetryRecord) {
  const inputTokens = rec.usage?.input_tokens ?? 0;
  const outputTokens = rec.usage?.output_tokens ?? 0;
  const costUsd = estimateCostUsd(rec.promptId, rec.variantId, inputTokens, outputTokens);
 
  // env.DB is the D1 binding from wrangler.toml
  // @ts-expect-error - global env in Workers
  await env.DB.prepare(
    `INSERT INTO prompt_invocations
     (prompt_id, variant_id, identity_key, latency_ms, success, input_tokens, output_tokens, cost_usd, created_at)
     VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`
  )
    .bind(
      rec.promptId,
      rec.variantId,
      rec.identityKey,
      rec.latencyMs,
      rec.success ? 1 : 0,
      inputTokens,
      outputTokens,
      costUsd,
      new Date().toISOString(),
    )
    .run();
}
 
function estimateCostUsd(_promptId: string, _variantId: string, inputTokens: number, outputTokens: number) {
  // Centralize per-model pricing here (Haiku 4.5 example)
  const inputUsdPerMTok = 0.25;
  const outputUsdPerMTok = 1.25;
  return (inputTokens * inputUsdPerMTok + outputTokens * outputUsdPerMTok) / 1_000_000;
}

Storing identity_key may feel optional today; six months from now you will be glad you did. Per-user regression detection and repeat-usage cohorts both depend on it.

The Guardian — automated rollback when SLOs break

The final piece is the cron-driven Guardian. It aggregates each canary variant's metrics over a 15-minute window and flips a kill-switch flag if the SLO is breached.

// src/prompts/guardian.ts
import { loadRegistry } from "./registry";
 
interface VariantStats {
  count: number;
  successRate: number;
  p95LatencyMs: number;
  avgCostUsd: number;
}
 
async function fetchStats(promptId: string, variantId: string, sinceIso: string): Promise<VariantStats> {
  // @ts-expect-error - global env in Workers
  const result = await env.DB.prepare(
    `SELECT
       COUNT(*) AS count,
       AVG(CASE WHEN success = 1 THEN 1.0 ELSE 0.0 END) AS success_rate,
       AVG(cost_usd) AS avg_cost
     FROM prompt_invocations
     WHERE prompt_id = ? AND variant_id = ? AND created_at >= ?`
  )
    .bind(promptId, variantId, sinceIso)
    .first<{ count: number; success_rate: number; avg_cost: number }>();
 
  // D1 has no PERCENTILE function, so approximate p95 with ORDER BY + OFFSET
  // @ts-expect-error
  const p95Row = await env.DB.prepare(
    `SELECT latency_ms FROM prompt_invocations
     WHERE prompt_id = ? AND variant_id = ? AND created_at >= ?
     ORDER BY latency_ms ASC
     LIMIT 1 OFFSET CAST((SELECT COUNT(*) FROM prompt_invocations
       WHERE prompt_id = ? AND variant_id = ? AND created_at >= ?) * 0.95 AS INTEGER)`
  )
    .bind(promptId, variantId, sinceIso, promptId, variantId, sinceIso)
    .first<{ latency_ms: number }>();
 
  return {
    count: result?.count ?? 0,
    successRate: result?.success_rate ?? 1,
    p95LatencyMs: p95Row?.latency_ms ?? 0,
    avgCostUsd: result?.avg_cost ?? 0,
  };
}
 
export async function evaluateAndRollback() {
  const registry = loadRegistry();
  const sinceIso = new Date(Date.now() - 15 * 60 * 1000).toISOString();
 
  for (const entry of registry.values()) {
    const canary = entry.variants.find((v) => v.status === "canary");
    if (!canary) continue;
 
    const stats = await fetchStats(entry.promptId, canary.id, sinceIso);
    if (stats.count < 50) continue; // not enough data yet
 
    const slo = canary.slo;
    const violations: string[] = [];
    if (stats.successRate < slo.minSuccessRate) {
      violations.push(`successRate ${stats.successRate.toFixed(3)} < ${slo.minSuccessRate}`);
    }
    if (stats.p95LatencyMs > slo.p95LatencyMs) {
      violations.push(`p95 ${stats.p95LatencyMs}ms > ${slo.p95LatencyMs}ms`);
    }
    if (stats.avgCostUsd > slo.maxCostPerCallUsd) {
      violations.push(`avgCost $${stats.avgCostUsd.toFixed(4)} > $${slo.maxCostPerCallUsd}`);
    }
 
    if (violations.length > 0) {
      console.warn(`[GUARDIAN] ${entry.promptId}/${canary.id} SLO violation: ${violations.join(", ")}`);
      await rollback(entry.promptId, canary.id);
    }
  }
}
 
async function rollback(promptId: string, variantId: string) {
  // Set a "disabled" flag in KV. The Selector consults this on every request.
  // @ts-expect-error
  await env.PROMPT_FLAGS.put(`disabled:${promptId}:${variantId}`, "1", { expirationTtl: 86400 });
}

The non-obvious decision here is not editing the JSON weight directly during a rollback. I wanted Git and prod to stay in sync, but writing a Git commit at 3 a.m. while CI is restarting only widens the blast radius. Treat rollback as immediate damage control via a runtime flag, then file a follow-up Pull Request the next morning to reflect the change in the registry.

Common pitfalls

1. Logging success/failure but not variantId

If the only thing you log is whether a call succeeded, you cannot tell which variant caused the regression. Aggregate metrics become useless and your rollback signal evaporates.

2. Bumping both prompt and model in the same canary

When you ship a new prompt it is tempting to also try a more capable model "while you're at it". Don't. You will not be able to attribute the change in metrics to one or the other. Keep model fixed during canary evaluation; ship a model migration as its own experiment.

3. Starting at 50/50

A new prompt feels exciting and you want it widely tested. The flip side is that 50% of users absorb any regression. Cap canary at 5–10% until a full evaluation window passes.

4. Forgetting to escape user input

Raw user input rendered into {{ article }} is a prompt injection vector. Wrap it in [USER_INPUT_BEGIN]…[USER_INPUT_END] blocks and have the system prompt explicitly state that anything between those markers is data, not instructions. See Prompt Injection Defense Production Patterns for the full playbook.

5. Promoting canary without demoting the old stable

I have shipped this bug. Two stable variants summing to weight 100 each will silently break the Selector. Enforce "exactly one stable per prompt" as a Zod refinement that fails CI, not as a checklist item in the PR template.

Metrics that actually drive decisions

The metric panel I rely on, in roughly the order you should add them:

• SLO basics: success rate, p95 latency, USD per call.
• Quality signals: thumbs-up/down feedback, day-2 retention, repeat-question rate within a session.
• Business signals: paid conversion rate, drop-off correlated with response time.

Repeat-question rate is the most underrated of the bunch. When a prompt regresses, it climbs cleanly. Combine it with prompt caching analysis and you get an honest picture of cost-per-quality. For caching specifically, see the Prompt Caching Cost Guide.

Migrating from inline prompts — a 7-day playbook

Most codebases I see keep prompts as template literals scattered across handlers. The path from there to a registry is shorter than it looks if you do it in stages.

Day 1–2 — extract and freeze. Find every messages.create(...) call and replace the inline strings with invokePrompt({ promptId: "..." }). Move the strings into JSON files under prompts/. At this point only one variant exists per prompt — v1, weight: 100, status: stable. No behavior changes; the goal is to make the next week's changes reviewable.

Day 3 — wire telemetry. Stand up the D1 (or Postgres, if you're outside Cloudflare) table. Run for 24 hours and confirm rows appear. You will already see useful signals: which prompts dominate cost, which have absurd p95 latency.

Day 4 — pick the lowest-risk experiment. Choose the prompt with the lowest user-visible impact and add a v2_canary variant with weight: 5. Watch the metrics for two days. The point is to exercise the pipeline, not to prove a hypothesis.

Day 5–6 — wire the Guardian cron. With one canary in flight, you can verify the rollback path works end-to-end. Force a rollback manually (set the kill-switch flag yourself) and confirm the Selector excludes the variant within seconds.

Day 7 — write the runbook. "Promotion" and "rollback" should each be a one-page document any engineer on the team can follow. The system is only as good as the runbook that explains it.

What to chart and what to alert on

Once telemetry is flowing, the first dashboard you should build has just five panels.

• Calls per minute, broken down by promptId × variantId. You will instantly see if the canary is getting the share you expect.
• Success rate, last 1 hour, broken down by variant. Sets the bar for the SLO threshold.
• p95 latency, last 1 hour, broken down by variant. A latency regression often precedes a quality regression by a few hours.
• USD spend per hour, broken down by variant. Make cost a first-class observability signal, not an afterthought you discover at month end.
• Error rate by error class. Group by 429, 529, 400, 5xx, plus a "timeout" bucket. A spike in 400s usually means the new prompt produced output the downstream system rejected.

For alerts, three rules cover most situations:

• Critical: success rate of any variant drops below slo.minSuccessRate for 10 consecutive minutes with at least 100 samples. Page someone.
• Warning: p95 latency exceeds slo.p95LatencyMs * 1.5 for 30 minutes. Slack notification, no page.
• Cost guard: hourly spend on any single variant exceeds 3× the 7-day rolling median. Slack notification.

Alerts that page on absolute thresholds get noisy fast. Pair every threshold with a sample-size guard so you do not wake up over a 10-call sample where one timeout looked catastrophic.

Handling failure modes that the Guardian cannot see

There is a category of regression the cron-based Guardian cannot catch: silent quality drops where success rate, latency, and cost all look fine, but the model is producing subtly worse content. For those, you need a human-in-the-loop review channel.

The lightest-weight version that works: every Nth response from a canary variant is fanned out to a Slack channel where you (or a designated reviewer) can react with 👍 or 👎. Store the reaction back in prompt_invocations.feedback_score and join it into the Guardian query. Once you have ~200 labeled samples, you can train the Guardian to also reject canaries whose human-rated quality is significantly below the stable variant's baseline.

This is also where I would warn against premature automation. You can build a complex LLM-as-judge evaluator that grades outputs automatically — and you should, eventually. But for the first three months of running a prompt registry, the cheapest signal that actually reflects "is this prompt better?" is a human looking at twenty samples a day.

Versioning strategy and the discipline of small variants

A natural temptation, once a registry exists, is to package every change into "the next big variant." Resist this. The variants that succeed in canary tend to be small, single-axis changes: one phrasing tweak, one structural rewrite, one new constraint. Variants that bundle five changes at once almost always regress on at least one dimension, and the post-mortem becomes guesswork.

A practical naming convention I follow:

• v3 — production stable.
• v3_canary_phrasing — testing only a wording change.
• v3_canary_structure — testing only a reordering of sections.
• v4 — promoted from a canary that won.

Yes, you can run multiple canaries simultaneously as long as their combined weight stays small (e.g. two canaries at 5% each, leaving stable at 90%). This works because of the sticky bucketing: each user is deterministically routed to one variant, never two, and the Guardian evaluates each variant independently. What you must not do is overlap canaries on the same promptId × identityKey axis without realizing it — that breaks the independence assumption your statistics rely on.

Reproducibility — replaying a regression six months later

The reason you persist variantId alongside every output is that, six months from now, someone will report "the summarizer produced something weird in this article" and you will want to reproduce it exactly. Three pieces of information let you do that.

• The prompt body that ran (recoverable from the registry's Git history at the timestamp of the call).
• The model and parameters (model, temperature, maxTokens — recoverable from the same Git snapshot).
• The exact rendered prompt (worth storing in prompt_invocations.rendered_prompt if your privacy policy permits).

I have gone back and forth on whether to store the rendered prompt. The argument for: replays become trivial and you can build a high-quality regression test set from real production calls. The argument against: the rendered prompt may contain user data and creates a longer retention surface. The middle ground that worked for me is hashing the rendered prompt and only storing the hash by default, with full text retention behind a feature flag for staging environments and short-lived production debug windows.

Coordinating prompt rollouts with model upgrades

Anthropic ships new models on its own cadence. The day a new Claude model becomes available, you do not want to ship "the new prompt and the new model" as one bundle. The clean sequence is:

1. Phase A — model migration only. Keep the prompt fixed. Add a canary variant whose only difference from stable is model: claude-opus-4-7. Watch metrics for a week. Promote when stable.
2. Phase B — prompt experiment. With the model now stable, layer a prompt-only canary on top.

This sequencing is not glamorous but it is the difference between knowing what changed and guessing. I have skipped Phase A under deadline pressure exactly once, and the resulting two-day investigation cost more than the deadline saved.

Should you adopt a hosted SaaS instead?

A reasonable counter-question: "Is this worth building when Promptlayer, LangSmith, and Helicone exist?" My honest answer is that hosted tools are great for small teams that do not want to own this layer, and overkill for teams of one or two who are still finding product-market fit. The crossover point in my experience is around 100k API calls per month. Below that, a self-hosted registry on Cloudflare Workers + D1 costs nearly nothing and keeps your prompt history in your own repo. Above that, the dashboards and team features of a SaaS start to earn their license fee.

Either way, the architecture in this article maps cleanly onto the SaaS world: most hosted tools are essentially a Registry + Selector + Telemetry stack with a hosted UI on top. Understanding the layers first means you will not be surprised when you switch tools.

Closing — what to do this week

Thank you for reading this far. You do not have to build all six layers at once. The single highest-ROI step is extracting the prompt strings from your application code into a JSON file in your repo. That alone unlocks Git diffs, blame, and review. The Selector, Telemetry, and Guardian can land incrementally over the following weeks.

Hand-tweaking prompts works fine until your service grows. Past that point, it's a slow-burning incident waiting to happen. Treating prompts like code is what lets you experiment aggressively without flinching.