Read the Catalog Your CLI Already Ships Before Bulk-Replacing Model IDs

Extracted strings include IDs that do not exist

This is where reality diverged hardest from my expectations. What you extract is strings present in the binary, not valid model identifiers. Inside a binary, strings get concatenated and sit next to unrelated tokens.

Here is exactly what turned up mixed into my results:

That first row is the dangerous one. claude-haiku-3-55 cleanly satisfies family plus generation plus minor, so a regex shape check waves it through. The only tell is a two-digit minor version, and if you miss it you walk away believing a generation exists that never did.

Shape checking is necessary but not sufficient. I added two more constraints: the minor version must be a single digit, and the family must be one of the four known names. Even then, "it was in the binary" does not prove "it is valid," so the final judgment splits along these lines:

• An extraction is a candidate, useful for narrowing mechanically
• A match is a sufficient reason to stop a replacement — the ID is still known, so there is no urgency to delete it
• A non-match is not a reason to delete or replace anything (more on this below)

That asymmetry turned out to be the single most useful idea here.

The same file gives you the per-provider spellings

The binary also carries the Amazon Bedrock and Vertex AI spellings. Here is real data pulled from my copy:

Anything carrying -v1 or :0 falls outside the bare-ID shape. Flag those mechanically as invalid and you will report a perfectly correct Bedrock configuration value as an error. Separate the three families — bare, vendor-prefixed, and @-dated — before you classify anything.

The underlying problem of one model wearing different names per destination is best handled by putting a logical name in front of it in configuration. For where that switch should live, the ordering work in curating the /model picker with modelPicker and the precedence rules in ANTHROPIC_MODEL versus ANTHROPIC_DEFAULT_MODEL are the groundwork.

Sort your own files into three buckets

Once you have a catalog, reconcile rather than replace. This is the audit script I use verbatim. It targets MDX here, but source and config files work the same way.

#!/usr/bin/env python3
"""audit_model_ids.py — reconcile model IDs in your files against a catalog
 
usage:
    ./model_catalog.sh > catalog.txt
    python3 audit_model_ids.py catalog.txt ./content
 
buckets:
    A matched          … present in the catalog. Leave alone
    B date mismatch    … family and generation known, date suffix not in catalog
    C needs fixing     … malformed, or an unknown family/generation
"""
import collections
import os
import re
import sys
 
# Bare ID shape, no provider prefix or suffix
SHAPE = re.compile(r"^claude-(opus|sonnet|haiku|fable)-\d+(-\d+)?(-20\d{6})?$")
# Pick IDs out of prose. Skip anything embedded in a longer token
IN_TEXT = re.compile(
    r"(?<![\w-])claude-(?:opus|sonnet|haiku|fable)-\d+(?:-\d+)?(?:-20\d{6})?(?![\w-])"
)
# Catch missing-hyphen typos such as claude-sonnet-46 separately
MALFORMED = re.compile(r"(?<![\w-])claude-(?:opus|sonnet|haiku|fable)-?\d{2,}(?![\w-])")
# Markdown link targets contain slugs, so strip them before counting
LINK = re.compile(r"\]\([^)]*\)")
 
 
def load_catalog(path):
    """Read the catalog, drop implausible spellings, return IDs and generations."""
    ids = set()
    for line in open(path, encoding="utf-8"):
        mid = line.strip()
        if not SHAPE.match(mid):
            continue
        parts = mid.split("-")
        # A two-digit minor is a concatenation artifact (guards claude-haiku-3-55)
        if len(parts) >= 4 and not parts[3].startswith("20") and len(parts[3]) > 1:
            continue
        ids.add(mid)
    generations = set()
    for mid in ids:
        p = mid.split("-")
        has_minor = len(p) >= 4 and not p[3].startswith("20")
        generations.add("-".join(p[:4]) if has_minor else "-".join(p[:3]))
    return ids, generations
 
 
def classify(mid, ids, generations):
    if mid in ids:
        return "A"
    if not SHAPE.match(mid):
        return "C"
    p = mid.split("-")
    has_minor = len(p) >= 4 and not p[3].startswith("20")
    base = "-".join(p[:4]) if has_minor else "-".join(p[:3])
    return "B" if base in generations else "C"
 
 
def main(catalog_path, root):
    ids, generations = load_catalog(catalog_path)
    buckets = {k: collections.Counter() for k in "ABC"}
    where = collections.defaultdict(set)
 
    for dirpath, _, filenames in os.walk(root):
        for name in filenames:
            if not name.endswith((".mdx", ".md", ".py", ".ts", ".json", ".yaml")):
                continue
            path = os.path.join(dirpath, name)
            text = LINK.sub("", open(path, encoding="utf-8", errors="replace").read())
            found = set(IN_TEXT.findall(text))
            for mid in IN_TEXT.findall(text):
                bucket = classify(mid, ids, generations)
                buckets[bucket][mid] += 1
                if bucket != "A":
                    where[mid].add(path)
            for mid in MALFORMED.findall(text):
                if mid not in found:  # do not double-count valid IDs
                    buckets["C"][mid] += 1
                    where[mid].add(path)
 
    for key, label in (("A", "matched"), ("B", "date mismatch"), ("C", "needs fixing")):
        counter = buckets[key]
        print(f"{key} {label}: {len(counter)} distinct / {sum(counter.values())} total")
    print()
    for key in ("B", "C"):
        if not buckets[key]:
            continue
        print(f"--- {key} ---")
        for mid, count in buckets[key].most_common():
            print(f"  {count:>4}  {mid}  ({len(where[mid])} files)")
    # Fail only while C is non-empty. B is meant for a human to read.
    return 1 if buckets["C"] else 0
 
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Run across 837 Japanese articles (excluding this one), the result was:

Out of 1,392 references, 63 warranted attention — about 4.5 percent. The claude-sonnet-4-6 replacement I had been about to run would have swept up 579 of the matched ones.

Before pointing it at real files, I test it on a tiny sample

Whether the buckets can be trusted depends on whether the script sorts correctly in the first place. So before aiming it at my real files, I ran it on a small sample with mistakes mixed in on purpose.

# catalog.txt (sample: real shapes mixed with a broken one)
claude-opus-4
claude-opus-4-6
claude-sonnet-4-6
claude-sonnet-4-20250514
claude-haiku-3-55   <- stands in for a concatenation fragment
claude-opus-5
 
# a.md (file to scan)
claude-sonnet-4-6 / claude-sonnet-4-6-20250514 / claude-sonnet-46 / claude-opus-4-6 / claude-haiku-3-55 / claude-opus-9

The output was:

A matched: 2 distinct / 2 total
B date mismatch: 1 distinct / 1 total
C needs fixing: 3 distinct / 3 total
--- B ---
     1  claude-sonnet-4-6-20250514  (1 files)
--- C ---
     1  claude-sonnet-46  (1 files)
     1  claude-haiku-3-55  (1 files)
     1  claude-opus-9  (1 files)
exit=1

Three things stand out. claude-haiku-3-55 is dropped from the catalog side as a two-digit minor, so it lands in C the moment it appears in a file. The hyphen-less claude-sonnet-46 is C as well. And claude-sonnet-4-6-20250514 has a known family and generation, so it goes to B rather than C and stays in the pile a human checks against primary sources. The exit code is 1 only while something remains in C.

The date suffix goes wrong before the family name does

Reading the 13 distinct entries in bucket B one by one produced the most useful finding of the exercise.

claude-sonnet-4-6-20250514 appeared 14 times. Known family, known generation, valid shape, nothing obviously wrong. Except 20250514 is Sonnet 4's date. A generation-date mismatch had been sitting in 14 places wearing a perfectly plausible face.

Any check that only inspects the family name passes this. It surfaces only when you reconcile the generation and the date together. Put differently: if you never write the date suffix, the contradiction cannot occur.

I settled on this split for my own material:

Absence from the catalog does not mean invalid

Getting this backwards turns the audit itself into the next incident.

The catalog you extracted is the set that build knows about. IDs introduced after that build are obviously missing. So absence carries two meanings at once:

1. The ID may not exist, or may have been retired
2. The ID may have arrived after the version you happen to be running

Nothing mechanical separates those. That is why B and C are split, and why only C fails the gate. B is the pile a human works through against primary sources.

My extraction did include spellings I had never seen. Unfamiliar is not the same as wrong, so the rule is to confirm against the official model list before treating any of them as usable.

Which brings us back to the original question. claude-opus-4-6 and claude-sonnet-4-6 were both in the catalog on my disk. That is reason enough to stop a replacement. Match means stop; absence means investigate. Deleting or replacing waits for a retirement announcement, which is primary information.

A small gate in front of bulk replacement

Finally, take the judgment out of memory. What I use runs the audit only on commits that touch model IDs.

#!/usr/bin/env bash
# .git/hooks/pre-commit — audit only when a change touches model IDs
set -euo pipefail
 
STAGED="$(git diff --cached --name-only --diff-filter=ACM)"
[ -z "$STAGED" ] && exit 0
 
# No model IDs in the staged diff means no audit needed
if ! git diff --cached -U0 -- $STAGED \
     | grep -qE '^[+-].*claude-(opus|sonnet|haiku|fable)-[0-9]'; then
  exit 0
fi
 
CATALOG="$(mktemp)"
trap 'rm -f "$CATALOG"' EXIT
 
if ! ./tools/model_catalog.sh > "$CATALOG" 2>/dev/null || [ ! -s "$CATALOG" ]; then
  # No catalog means no basis for judgment. Do not let that pass as success.
  echo "Could not read the model catalog. Verify manually before committing." >&2
  exit 1
fi
 
# Check whether the IDs on removed lines are still in the catalog
REMOVED="$(git diff --cached -U0 -- $STAGED \
  | grep -E '^-' \
  | grep -oE 'claude-(opus|sonnet|haiku|fable)-[0-9]+(-[0-9]+)?' \
  | sort -u || true)"
 
STILL_KNOWN=""
for mid in $REMOVED; do
  if grep -qx "$mid" "$CATALOG"; then
    STILL_KNOWN="$STILL_KNOWN $mid"
  fi
done
 
if [ -n "$STILL_KNOWN" ]; then
  echo "You are removing IDs that are still in the catalog:$STILL_KNOWN" >&2
  echo "Confirm the retirement notice, then pass --no-verify deliberately." >&2
  exit 1
fi
 
python3 tools/audit_model_ids.py "$CATALOG" ./content

The hook earns its keep by inspecting removed lines rather than added ones. Migration accidents rarely come from mistyping a new ID; they come from deleting one that still works. That is precisely the step where I caught myself.

The --no-verify escape hatch stays open so a legitimate, verified migration is not obstructed. The gate exists to ask whether you decided, not to decide for you.

Three things that tripped me up running this

It did not work on the first pass. Here is what I actually hit and fixed.

Assuming links were the cause of the false positives

My first run reported 28 distinct entries needing fixes. Opening them showed article slugs such as claude-sonnet-46-complete-mastery-guide. I assumed link targets were to blame, stripped Markdown links, and recounted.

Three entries disappeared. Twenty-eight became twenty-five, and the pile stayed essentially intact.

What actually worked was a different change. Adding negative word-boundary lookarounds and restricting the family name to the four known values dropped non-ID detections to zero — same files, same corpus.

Stripping links is not useless, but it was not the fix. Guessing at the cause of a false positive means paying for a remedy that does not work while believing the problem is handled. I nearly skipped measuring before and after here.

Reporting -v1 as an error

Check against the bare-ID shape alone and Bedrock's anthropic.claude-opus-4-6-v1 lands in the needs-fixing bucket. Reporting a correct configuration value as a defect erodes trust in the report itself, and a gate nobody trusts is a gate nobody reads.

I now separate the three families — bare, vendor-prefixed, and @-dated — before classifying anything, and I keep that order fixed.

Passing silently when the catalog cannot be read

My first hook exited 0 when strings failed. That treated "no basis for judgment" as success, which is the hardest failure mode to detect in production. If the binary ships stripped in some future release, the gate would quietly stop working.

It now fails explicitly on an empty catalog. Stopping beats passing in silence.

Wrapping up

If you'd like to try this, run strings once and drop your build's catalog into a file. It takes under a minute. The moment you reconcile that file against your repository, "what to replace" and "what to leave alone" separate into two clearly different piles.

Had I run that sed command, I would have deleted still-working IDs from 56 articles without noticing. What stopped my hand was asking, right after finishing the count, whether those IDs were really invalid.

Look it up before you replace it. I try to keep that order most of all on the nights when I'm in a hurry.