A 40% Lower Price Doesn't Mean a 40% Lower Bill — Measuring the Opus 4.8 to Sonnet 5 Migration by Cost per Completed Task

A minimal layer for recording consumption vectors

Everything you need is already in response.usage; the only missing piece is aggregation per task. Accumulate each turn's usage under a task ID:

// consumption.ts — accumulate a per-task consumption vector
export interface ConsumptionVector {
  taskId: string;
  model: string;
  turns: number;            // API calls, including tool round trips
  inputTokens: number;      // non-cached input
  outputTokens: number;
  cacheReadTokens: number;  // cache_read_input_tokens
  cacheWriteTokens: number; // cache_creation_input_tokens
  retries: number;          // failed attempts that were re-run
  completed: boolean;       // passed the success gate
}
 
export function emptyVector(taskId: string, model: string): ConsumptionVector {
  return { taskId, model, turns: 0, inputTokens: 0, outputTokens: 0,
           cacheReadTokens: 0, cacheWriteTokens: 0, retries: 0, completed: false };
}
 
// Add one turn's usage. If a response is missing usage, mark the task
// as "consumption unknown" so downstream aggregation can exclude it.
export function addTurn(v: ConsumptionVector, usage: {
  input_tokens: number; output_tokens: number;
  cache_read_input_tokens?: number; cache_creation_input_tokens?: number;
}): ConsumptionVector {
  return {
    ...v,
    turns: v.turns + 1,
    inputTokens: v.inputTokens + usage.input_tokens,
    outputTokens: v.outputTokens + usage.output_tokens,
    cacheReadTokens: v.cacheReadTokens + (usage.cache_read_input_tokens ?? 0),
    cacheWriteTokens: v.cacheWriteTokens + (usage.cache_creation_input_tokens ?? 0),
  };
}

One operational habit worth adopting: retries accrue to the same taskId, not to a new one. A failed attempt is money you spent to get that task done. Book retries separately and you end up with the upside-down conclusion that failure-prone models are cheap.

A paired-run harness for old and new models

Next, run the identical task spec on both models and collect consumption vectors in pairs. The essential piece is a success gate — a predicate that decides whether the task actually completed. Fast and cheap but unfinished is a cost, not a result.

// paired-run.ts — side-by-side measurement across models
import { ConsumptionVector, emptyVector, addTurn } from "./consumption";
 
interface TaskSpec {
  id: string;
  run: (model: string, onUsage: (u: any) => void) => Promise<unknown>;
  succeeded: (result: unknown) => boolean; // success gate
  maxRetries: number;
}
 
export async function measureOnce(
  spec: TaskSpec, model: string
): Promise<ConsumptionVector> {
  let v = emptyVector(spec.id, model);
  for (let attempt = 0; attempt <= spec.maxRetries; attempt++) {
    if (attempt > 0) v = { ...v, retries: v.retries + 1 };
    try {
      const result = await spec.run(model, (usage) => { v = addTurn(v, usage); });
      if (spec.succeeded(result)) return { ...v, completed: true };
    } catch { /* count the failure as a retry and keep going */ }
  }
  return v; // returns with completed: false; aggregate it separately
}
 
// Run the same spec N times per model, interleaved to avoid time-of-day bias
export async function pairedRun(
  spec: TaskSpec, models: [string, string], n: number
): Promise<ConsumptionVector[]> {
  const out: ConsumptionVector[] = [];
  for (let i = 0; i < n; i++) {
    for (const m of models) out.push(await measureOnce(spec, m));
  }
  return out;
}

Ten runs per task family is enough to see the shape. I sampled from the real overnight jobs behind the Dolice Labs sites rather than writing synthetic benchmarks — measuring your own production tasks is the whole point. Skip that and you get the classic outcome: winning the benchmark and losing the invoice.

Solve for the break-even turn count in advance

While the paired runs accumulate, you can also draw the theoretical defense line. "How many turns can the new model afford before it stops being cheaper?" depends only on the task shape (S, d, output per turn) and both price vectors:

// break-even.ts — how many extra turns a migration can tolerate
interface Price { inPerMTok: number; outPerMTok: number; }
interface Shape { base: number; growthPerTurn: number; outPerTurn: number; }
 
function taskCost(p: Price, s: Shape, turns: number): number {
  const input = turns * s.base + s.growthPerTurn * (turns * (turns - 1)) / 2;
  const output = turns * s.outPerTurn;
  return (input * p.inPerMTok + output * p.outPerMTok) / 1e6;
}
 
// Find the turn count where the new model matches the old model's cost
export function breakEvenTurns(
  oldP: Price, newP: Price, s: Shape, baselineTurns: number
): number {
  const ceiling = taskCost(oldP, s, baselineTurns);
  let lo = baselineTurns, hi = baselineTurns * 4;
  for (let i = 0; i < 40; i++) {
    const mid = (lo + hi) / 2;
    taskCost(newP, s, mid) < ceiling ? (lo = mid) : (hi = mid);
  }
  return lo;
}
 
const shape = { base: 3000, growthPerTurn: 1200, outPerTurn: 400 };
const opus = { inPerMTok: 5, outPerMTok: 25 };
console.log(breakEvenTurns(opus, { inPerMTok: 2, outPerMTok: 10 }, shape, 4)); // ≈ 7.6
console.log(breakEvenTurns(opus, { inPerMTok: 3, outPerMTok: 15 }, shape, 4)); // ≈ 5.7

For this shape, the tolerance is about 7.6 turns while intro pricing lasts and about 5.7 turns at standard pricing. Put differently: any task family whose median exceeds 6 turns at standard rates loses money on this migration despite the 40% price cut. Intro pricing ends August 31, so every permanent decision should also be computed at standard rates — the same discipline as in the effective-dated cost forecast for the Sonnet 5 intro-price expiry.

A week of paired runs: the answer split by task family

Here is what a week of side-by-side runs produced across three families. Costs are per completed task, retries included, incomplete runs excluded and tracked separately.

The single-shot families dropped almost exactly by the price ratio: when the consumption vector holds still, the discount passes straight through. The link checker was the interesting one — Sonnet 5 tended to decompose the verification into finer steps, adding turns. Writing the ceiling into the spec ("verify up to three links per response, five round trips maximum") brought the median back to 5 turns and improved the per-task cost to -41%. My main takeaway: as the model gets better at planning, you have to state your loop budget explicitly, or it will spend the extra diligence on your dime.

Pitfalls to defuse before you trust the numbers

The aggregation design trips people up more than the code does. In the order I hit them:

1. Never average incomplete tasks in. Consumption with completed: false belongs to a separate metric (wasted-shot rate). Mixing it in makes failure-prone models look cheap.
2. Keep cache components out of the input column. Caches are cold right after a switch, so the first week's effective cost reads high versus steady state. Holding cacheReadTokens separately lets you recompute once things warm up.
3. Align max_tokens and retry policy across both models. Otherwise you are measuring configuration differences, not model differences.
4. Do not let a price change straddle your measurement window. With the intro price ending August 31, a window that crosses that date mixes consumption changes with price changes. Cut windows where the price is constant.

Wrap-up — the first step is a turn-count histogram

Before any migration decision, pull the median and distribution of turn counts per task family from the usage logs you already have. That single query separates the families where the sticker discount applies as-is (one or two turns) from the ones where the consumption profile can flip the sign (five turns and up). The paired-run harness can come after. Since adopting this order, a model migration decision takes me half a day instead of a week of second-guessing.