A Silent Drop to a Weaker Model Is Scarier Than an Error: Designing a Capability Floor for Claude API Fallback

Cut the chain at the floor, not at a fallback table

Sort candidate models by capability and keep only the ones that satisfy the contract. For allow_downgrade=False tasks, discard everything but the primary (the top candidate).

def build_chain(catalog: list[ModelSpec], contract: TaskContract) -> list[ModelSpec]:
    """Return a fallback chain of contract-satisfying models, most capable first."""
    eligible = [m for m in catalog if contract.is_satisfied_by(m)]
    eligible.sort(key=lambda m: m.tier, reverse=True)
 
    if not eligible:
        raise RuntimeError(
            f"No model satisfies the floor (min_tier={contract.min_tier.name}) "
            f"for task '{contract.name}'. Review the catalog or the contract."
        )
 
    if not contract.allow_downgrade:
        # No-downgrade tasks get exactly one model. Stop rather than drop
        return eligible[:1]
    return eligible

What matters here is that build_chain raises before it can return empty. "Silently moving on with zero candidates" was exactly the quiet incident from the opening. If no model meets the floor, that's a configuration anomaly and it should ring loudly.

A degradation budget: stop "stuck on weaker" by time and count

Dropping one tier inside the chain might be acceptable, but doing so for hours is a different problem. What you assumed was a brief absence can actually be a long primary outage. So give it a budget for how much degradation you'll tolerate.

import time
 
 
class DegradationBudget:
    """A cap on downgrades within a window. Past it, forbid downgrades and fail."""
 
    def __init__(self, max_events: int = 5, window_seconds: int = 3600):
        self.max_events = max_events
        self.window_seconds = window_seconds
        self._events: list[float] = []
 
    def _prune(self, now: float) -> None:
        cutoff = now - self.window_seconds
        self._events = [t for t in self._events if t >= cutoff]
 
    def can_degrade(self) -> bool:
        now = time.time()
        self._prune(now)
        return len(self._events) < self.max_events
 
    def record(self) -> None:
        self._events.append(time.time())

Once the budget is spent, tip subsequent downgrades to "not allowed." In other words, if the primary is unreachable, record that run as a failure and defer to the next one. In unattended operation, this is the last line of defense against the worst case: mass-producing thin artifacts without noticing. "Five per hour" is a number I tuned by feel; adjust it to your run frequency.

Separate temporary absence from permanent removal

Treating every failure with the same "fallback" leads to bad decisions. At least these three are different things.

Permanent removal really happens. Even as new models ship, access to a specific model can suddenly halt for regional or policy reasons, and older models get retired on notice. If you keep retrying these as temporary absences, you'll keep paying wasted latency and cost. That's exactly why permanent removal — and only that — should "remember the switch."

class ModelHealth:
    """Record permanently-gone models and exclude them from the catalog thereafter."""
 
    PERMANENT_HINTS = ("model_not_found", "not_found",
                       "deprecated", "decommissioned", "unsupported_region")
 
    def __init__(self) -> None:
        self._dead: set[str] = set()
 
    def classify(self, status_code: int, error_text: str) -> str:
        text = error_text.lower()
        if status_code in (401, 400):
            return "caller_error"
        if any(h in text for h in self.PERMANENT_HINTS):
            return "permanent"
        # 529, "currently unavailable," and timeouts fall through to here
        return "transient"
 
    def mark_dead(self, model_id: str) -> None:
        self._dead.add(model_id)
 
    def filter_catalog(self, catalog: list[ModelSpec]) -> list[ModelSpec]:
        return [m for m in catalog if m.model_id not in self._dead]

In an unattended process, don't just keep _dead in memory — write it somewhere that survives restarts, like a KV store or a small file. Otherwise, every time the process dies and comes back, it will go bang into the already-gone model all over again.

Run chain, budget, and health together in one place

With these parts in place, the call site becomes very plain.

def call_with_floor(client, contract, catalog, health, budget, build_payload):
    """Call while honoring the contract's floor. Raise rather than drop below it."""
    live = health.filter_catalog(catalog)
    chain = build_chain(live, contract)  # raises here if zero candidates
 
    last_error = None
    for index, spec in enumerate(chain):
        is_downgrade = index > 0
        if is_downgrade and not budget.can_degrade():
            raise RuntimeError(
                f"'{contract.name}': degradation budget exhausted. "
                f"Treating this run as a failure until the primary recovers."
            )
        try:
            resp = client.messages.create(**build_payload(spec))
            if is_downgrade:
                budget.record()  # a successful downgrade spends budget — worth ringing
                log_degradation(contract.name, spec.model_id)
            return resp
        except APIError as e:
            kind = health.classify(e.status_code, str(e))
            last_error = e
            if kind == "caller_error":
                raise  # dropping won't fix it. stop now
            if kind == "permanent":
                health.mark_dead(spec.model_id)  # gone from this chain next time
            # transient: give up on this model, move to the next candidate
            continue
 
    raise RuntimeError(f"'{contract.name}': chain exhausted.") from last_error

build_payload(spec) assembles the per-model messages.create arguments, injecting spec.model_id, max_tokens, and whether thinking is on (real IDs are injected from env via placeholders like YOUR_MODEL_ID). log_degradation explicitly records the fact that a downgrade happened — somewhere separate from the success logs. The philosophy is right there in the code: a downgrade is not a "success," it's a "degradation worth recording."

For a final-artifact task, just set the contract to allow_downgrade=False and the chain shrinks to a single primary. If the primary is temporarily absent, you exhaust candidates on transient and finally stop with an exception. That's how you structurally prevent the "quiet drop to a weaker model" incident.

Surface the fact that you dropped

The finishing touch for unattended operation is notification. How many downgrades happened, which models you dropped as permanently gone, how much degradation budget remains. Just putting these three on a channel separate from the success logs — a daily summary, in my case — nearly eliminated "only noticing the next morning."

After years of running automation as an indie developer, here's what I keep coming back to: the value of automation lives in the hours nobody is watching — so its counterpart is always recording what degraded during those hours. The quieter the success, the more you should doubt it. Fallback is where you bake that doubt into the code.

If you want one concrete step, open the fallback code in a pipeline you're running right now and check just one thing: is there a path that silently moves on when candidates hit zero? That's where the quiet incident hides.