Auditing pinned model IDs before claude-sonnet-4 and claude-opus-4 retire from the API

A parity harness that measures output before retirement

Whether the replacement is safe is something you can measure without waiting for retirement. Take a handful of prompts you actually run in production, send the same input to both the old and the new model, and line up the output, token counts, and latency side by side.

import json
import time
from anthropic import Anthropic
 
client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment
 
OLD = "claude-opus-4"      # retiring
NEW = "claude-opus-4-8"    # target (confirm the latest ID in the docs)
 
# Drop in 5-10 prompts you actually send in production.
PROMPTS = [
    "Classify this ticket as high/normal/low and give a one-sentence reason: ...",
    "Point out any bug in this diff, or return NONE if there is none: ...",
]
 
 
def run(model: str, prompt: str) -> dict:
    t0 = time.time()
    msg = client.messages.create(
        model=model,
        max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    text = "".join(b.text for b in msg.content if b.type == "text")
    return {
        "text": text,
        "out_tokens": msg.usage.output_tokens,
        "latency_ms": round((time.time() - t0) * 1000),
    }
 
 
for prompt in PROMPTS:
    a = run(OLD, prompt)
    b = run(NEW, prompt)
    print(json.dumps({
        "prompt": prompt[:40],
        "identical": a["text"].strip() == b["text"].strip(),
        "out_tokens": [a["out_tokens"], b["out_tokens"]],
        "latency_ms": [a["latency_ms"], b["latency_ms"]],
    }))

If identical comes back mostly false, do not panic. Word-for-word identical output is the exception. What matters is whether the downstream breaks. If you json.loads the output, check that the new model's output still passes schema validation. If formatting shifts, any regex that slices the output will drift. If a classification task expects only a high/normal/low label, check that the new model does not add a preamble.

In these comparisons I have repeatedly found that the output itself improved, while a brittle downstream parser could not absorb the new formatting and broke. The thing to fix is usually the parser, not the model. Knowing this before retirement turns the cutover from "deploy and pray" into "deploy what you already verified."

An alias layer that makes the next retirement a one-liner

Once verified, do not sprinkle the target ID across the code. Collapse it into a single layer that resolves a logical name to a concrete ID. Naming by role is the trick.

// models.ts — an alias layer that confines model IDs to one place
export type ModelRole = "reasoning" | "balanced" | "fast";
 
const RESOLVE: Record<ModelRole, string> = {
  reasoning: "claude-opus-4-8",
  balanced: "claude-sonnet-4-6",
  fast: "claude-haiku-4-5",
};
 
export function modelFor(role: ModelRole): string {
  return RESOLVE[role];
}

Callers never know the concrete ID.

import { modelFor } from "./models";
 
const msg = await client.messages.create({
  model: modelFor("reasoning"),
  max_tokens: 1024,
  messages,
});

Whichever model retires next, the only change is one line in RESOLVE. Caller code stays untouched. That is the practical payoff of an anti-corruption layer (a boundary that keeps external concerns from leaking into your code): you move from full-text-searching on every retirement notice to editing one line in a config.

A safety net after retirement — do not swallow model_not_found

Even with an alias layer, a slow deploy or a cached copy of old code can produce model_not_found for a few minutes right after retirement. Leave that unguarded and those few minutes become an outage.

import Anthropic from "@anthropic-ai/sdk";
 
const client = new Anthropic();
 
// Temporary landing spot if a retired ID is somehow still called.
const FALLBACK: Record<string, string> = {
  "claude-opus-4": "claude-opus-4-8",
  "claude-sonnet-4": "claude-sonnet-4-6",
};
 
export async function createWithFallback(
  params: Anthropic.MessageCreateParams,
) {
  try {
    return await client.messages.create(params);
  } catch (err) {
    if (err instanceof Anthropic.NotFoundError && FALLBACK[params.model]) {
      const next = FALLBACK[params.model];
      console.warn(`model ${params.model} unavailable -> falling back to ${next}`);
      metrics.increment("model_fallback", { from: params.model, to: next });
      return client.messages.create({ ...params, model: next });
    }
    throw err;
  }
}

But a fallback is one step from a cover-up. Always emit the console.warn and a metric. "The fallback keeps firing" equals "a code path that still references a retired ID survives." If you silently absorb it, code pointing at the retired ID lingers indefinitely. The right mental temperature: a safety net exists to detect that the migration is done and make itself unnecessary.

Do not flip it all at once — watch a canary

Finally, stage the cutover itself. You do need everything on the new model by the deadline, but flipping 100% at once means any unexpected behavior gap hits every user.

In practice I always go in this order. First, point only my own and internal traffic at the new model for a day and watch output quality and downstream error rate. If that is clean, widen to about 10% and put three numbers side by side on a dashboard: average output_tokens, downstream parser error rate, and p95 latency. Only after confirming no regression in the numbers do I move the rest.

// Stably route ~10% to the new model by hashing the user ID.
function useNewModel(userId: string): boolean {
  return hash(userId) % 100 < 10;
}

The nice thing about a canary is that rollback is "change the number from 10 back to 0." Get this in place with room to spare before retirement, and on the day you just set the percentage to 100 and the migration is done.

If you are racing the same migration, I hope this gives you a usable order of operations. Start by running the scanner above once at your production repo root. If even one [RETIRING 6/15] shows up, that is where model_not_found will land on the day.