A Fail-Closed Model Pricing Registry So New Models Don't Quietly Break Your Cost Math

Compute Real Cost From the usage Block

The response usage carries the exact token breakdown your bill is based on. Run it through the registry and you get measured cost, not a guess.

// cost.ts — turn usage plus rates into the cost of one request
import { RATES, ModelRate } from "./pricing";
 
interface Usage {
  input_tokens: number;
  output_tokens: number;
  cache_creation_input_tokens?: number; // cache writes
  cache_read_input_tokens?: number;     // cache reads
}
 
const PER = 1_000_000; // divide to per-token
 
export function costOf(model: string, usage: Usage, cacheTtl: "5m" | "1h" = "5m"): number {
  const r: ModelRate | undefined = RATES[model];
  if (!r) {
    // The crux: do not score unknown models as 0 (implemented next).
    throw new UnknownModelError(model);
  }
  const write = cacheTtl === "1h" ? r.cacheWrite1h : r.cacheWrite5m;
  return (
    (usage.input_tokens * r.input +
      usage.output_tokens * r.output +
      (usage.cache_creation_input_tokens ?? 0) * write +
      (usage.cache_read_input_tokens ?? 0) * r.cacheRead) /
    PER
  );
}

Pricing cache_creation_input_tokens and cache_read_input_tokens at the input rate inflates the error the more caching you do. Reads are about a tenth of input, so splitting them out cleanly was enough to visibly tighten my rollup.

Don't Let Unknown Model IDs Pass at Zero

This was the heart of the first incident: make an unknown model an exception, not zero dollars. Fail closed. Push the cost to the safe side — the side you can detect, not the side that hides.

export class UnknownModelError extends Error {
  constructor(public readonly model: string) {
    super(`Unpriced model: ${model} — add it to pricing.ts`);
    this.name = "UnknownModelError";
  }
}
 
// How the aggregation loop absorbs it: one unknown model must not halt
// the whole job, but it must always surface (never be swallowed).
export function aggregate(records: { model: string; usage: Usage }[]) {
  let total = 0;
  const unpriced = new Set<string>();
  for (const rec of records) {
    try {
      total += costOf(rec.model, rec.usage);
    } catch (e) {
      if (e instanceof UnknownModelError) {
        unpriced.add(e.model); // not passed through as 0 — visible later
        continue;
      }
      throw e;
    }
  }
  return { total, unpriced: [...unpriced] };
}

What I care about here is landing between "halt" and "swallow." Crashing an overnight job over a single unknown model is too much. But passing it through at zero takes you right back to the original incident. Returning unpriced alongside the total keeps the state I want: the numbers come out, and the gaps come out with them.

A Test That Turns a New Model Into a Red Build

Once design prevents the failure, build the state where the next new model gets noticed. Collect model IDs from real logs and keep one light test that fails if any of them is missing from the registry.

// pricing.test.ts — every model seen in real logs must be priced
import { RATES } from "./pricing";
 
// Model IDs pulled from recent request logs (auto-refreshed in ops)
import { observedModels } from "./fixtures/observed-models";
 
test("every observed model exists in the pricing registry", () => {
  const missing = observedModels.filter((m) => !(m in RATES));
  expect(missing).toEqual([]); // one unpriced model turns CI red
});
 
test("rates are positive and output is at least input", () => {
  for (const [model, r] of Object.entries(RATES)) {
    expect(r.input, model).toBeGreaterThan(0);
    expect(r.output, model).toBeGreaterThanOrEqual(r.input);
  }
});

The second test looks trivial but earns its keep. It is how I caught the haiku-4-5 row I had left at 0 under a TODO. The plain invariant "output should not be cheaper than input" reliably catches copy-paste misses during a rate update.

How Far It Drifts, in Numbers

Abstractions get deprioritized, so here is the order of magnitude. Say one overnight job uses 500k input and 80k output tokens, and 30% of that runs on a newly added, unpriced model. If the unpriced portion slides through at zero, that run reports roughly 30% under its true cost.

In my case, what hurt was less the dollar figure than the broken day-over-day comparison. On a day I tried a new model, usage actually went up, yet the chart showed it going down. When the sign of the change flips, people stop noticing the anomaly. Once I failed closed and surfaced unpriced models, a non-empty unpriced became the alert itself, and I could go straight to the cause instead of second-guessing the numbers.

Small Calls That Paid Off

A few decisions around pricing don't show up in any number but made operations easier.

One is keeping aliases (moving names like claude-3-5-haiku-latest) out of aggregation. The billed response returns a resolved, concrete model ID, so I made that the source of truth. The rule is simple: reconcile against the ID the response declares, not what you asked for at request time.

The other is treating a rate update as a code change with history. Rates move for external reasons, and if the change leaves no record, a later recomputation of past cost will not reconcile. I commit pricing.ts edits with an effective-date comment and compute historical traffic at the rate in force then.

Your Next Step

If your rates are inlined all over the code today, start by creating a single pricing.ts and centralizing on the response's model field as the key. Just routing everything through one costOf that throws on unknown models means that the next morning a model appears, CI goes red and tells you, instead of the chart drifting in silence.

I am still fixing this as I run it, but the one principle that always holds for cost math is this: if it has to break, let it break visibly rather than quietly. Thank you for reading.