⬡ API & SDK/2026-07-13Advanced

When Extended Thinking Flattened My Accuracy but Doubled the Bill — Field Notes on Measuring the Marginal Utility of Thinking Tokens

You pinned budget_tokens high 'to be safe,' accuracy barely moved, and the bill kept climbing. These field notes show how to ledger real thinking-token usage by p50/p95/hit-rate and measure how much accuracy each extra 1,000 thinking tokens actually buys.

claude⁸ extended-thinking⁷ budget-tokens cost-optimization²⁷ observability²⁰

✦ Premium Article

One morning, a nightly ticket-classification job on an app I run as an indie developer was pushing my bill to nearly double the prior month. Tracing it back, the cause was a change I had made almost absent-mindedly: while moving to Opus 4.8, I pinned budget_tokens at 32,000 "to leave headroom." Accuracy had barely moved. I was paying for deeper reasoning that never reached the answer — most of the money spent on thinking was landing nowhere.

These are the field notes from getting that budget back down to a defensible number. The whole thing rests on one idea: budget is neither "safer when larger" nor "cheaper when smaller." Once you measure how much accuracy one more step of budget actually buys, the number you should pay for is nearly self-selecting.

budget_tokens is a ceiling, not a fixed cost

Clear up the common misreading first. budget_tokens caps the tokens the model may spend on internal reasoning; it does not spend that amount every call. Easy inputs wrap up in a few hundred tokens, and only hard tasks push toward the ceiling.

Setting	The misreading	Actual behavior
budget=32,000	Every call is billed for 32,000	Only reaches deep on hard tasks; plain inputs stop in the hundreds
budget=4,000	Always cheap	May truncate reasoning on hard tasks and return a shallow answer

Raising the budget hands the model room to think deeply when it hits a hard task. Miss that, and "the bill scares me, so cut everything" turns a model that could have solved the problem into one that can't. And going the other way — "the bill scares me, so make it big," which is exactly what I did — wastes the headroom on easy tasks and compounds quietly on the hard ones. Either fixed value misses without measurement.

from anthropic import Anthropic
 
client = Anthropic()
 
resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=8000,
    thinking={"type": "enabled", "budget_tokens": 10000},  # ceiling for reasoning
    messages=[{"role": "user", "content": "Optimize this SQL: ..."}],
)
 
# Always record how many thinking tokens were actually spent
usage = resp.usage
print(usage.thinking_tokens, usage.output_tokens)

response.usage.thinking_tokens is the real spend. Whether you log it on every request is what separates later tuning that is measurement from tuning that is guesswork.

Ledger the thinking tokens first

Before touching the budget, ledger production usage. Averages hide the long tail, so keep three things: p50, p95, and hit-rate — the share of requests where thinking_tokens reached at least 95% of the budget. A high hit-rate hints that reasoning is being cut off.

import json, time, statistics
from pathlib import Path
 
LEDGER = Path("thinking_ledger.jsonl")
 
def call_and_log(client, *, model, budget, messages, task_type):
    resp = client.messages.create(
        model=model, max_tokens=8000,
        thinking={"type": "enabled", "budget_tokens": budget},
        messages=messages,
    )
    u = resp.usage
    rec = {
        "ts": time.time(), "task_type": task_type, "budget": budget,
        "thinking_tokens": u.thinking_tokens,
        "output_tokens": u.output_tokens,
        "hit": u.thinking_tokens >= budget * 0.95,  # pinned to the ceiling?
    }
    with LEDGER.open("a") as f:
        f.write(json.dumps(rec) + "\n")
    return resp
 
 
def summarize(task_type):
    rows = [json.loads(l) for l in LEDGER.read_text().splitlines()]
    rows = [r for r in rows if r["task_type"] == task_type]
    tt = sorted(r["thinking_tokens"] for r in rows)
    if not tt:
        return None
    p50 = statistics.median(tt)
    p95 = tt[min(len(tt) - 1, int(len(tt) * 0.95))]
    hit_rate = sum(r["hit"] for r in rows) / len(rows)
    return {"n": len(rows), "p50": p50, "p95": p95,
            "max": tt[-1], "hit_rate": round(hit_rate, 3)}
 
# e.g. {"n": 214, "p50": 1180, "p95": 5200, "max": 9900, "hit_rate": 0.02}

Past 100 records or so, the shape per task type emerges. Finding budget=32,000 on classification while p95 sits at 5,200 is a common sight. The other 26,800 is just headroom you may never touch, and plain inputs are not billed for it. The real problem is the case where hard tasks do reach deep and that depth still doesn't move the answer. That is what we measure next.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦Instrumentation code that ledgers real thinking-token spend by p50/p95 and budget hit-rate, so you set budget from data instead of gut feel

✦An eval harness that measures the accuracy gained per 1,000 extra thinking tokens as marginal utility

✦A rule for telling under-budget apart from an overloaded task when the hit-rate is high but accuracy stalls

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Measure the marginal utility of one more budget step

Distribution and hit-rate alone can't tell you whether deeper thinking produced a better answer — only accuracy can. So fix an eval set, sweep the budget in steps, run the same task at each, and report how many accuracy points you gained per 1,000 thinking tokens.

def evaluate(client, dataset, *, model, budget, judge):
    """dataset: [{"messages": [...], "gold": ...}], judge(pred, gold)->bool"""
    correct = 0
    thinking_sum = 0
    for ex in dataset:
        resp = client.messages.create(
            model=model, max_tokens=4000,
            thinking={"type": "enabled", "budget_tokens": budget},
            messages=ex["messages"],
        )
        pred = resp.content[-1].text
        thinking_sum += resp.usage.thinking_tokens
        correct += judge(pred, ex["gold"])
    n = len(dataset)
    return {"budget": budget, "acc": correct / n, "avg_thinking": thinking_sum / n}
 
 
def marginal_utility(results):
    """From budget-ascending results, accuracy gained per 1,000 thinking tokens."""
    out = []
    for prev, cur in zip(results, results[1:]):
        d_acc = cur["acc"] - prev["acc"]
        d_think = max(1.0, cur["avg_thinking"] - prev["avg_thinking"])
        out.append({
            "from": prev["budget"], "to": cur["budget"],
            "d_acc_pts": round(d_acc * 100, 2),
            "per_1k": round(d_acc / d_think * 1000 * 100, 3),  # pts / 1k think tok
        })
    return out
 
grid = [2000, 4000, 8000, 16000, 32000]
results = [evaluate(client, dataset, model="claude-opus-4-8", budget=b, judge=judge)
           for b in grid]
for step in marginal_utility(results):
    print(step)

On my classification workload the numbers came out roughly like this.

Step	Accuracy gain	Per 1,000 thinking tokens
2,000 → 4,000	+4.1 pts	+2.3 pts
4,000 → 8,000	+1.2 pts	+0.5 pts
8,000 → 16,000	+0.3 pts	+0.08 pts
16,000 → 32,000	-0.1 pts	~0

The curve flattens sharply past 4,000, and everything above 16,000 was inside the noise. The thinking tokens I was paying for at 32,000 returned nothing in accuracy. Once you can see this, choosing a budget stops being a judgment call and becomes a reading. Take the budget one step before marginal utility drops below the floor you can live with — I set mine at "+0.2 pts per 1,000 thinking tokens" — and you avoid both overpaying and over-cutting. That floor pointed at 4,000.

A small scorer that recommends a budget from logs

The eval is reusable once built, but for day-to-day jobs it helps to derive a candidate mechanically from the ledger: combine the distribution (p95) and hit-rate, then let the marginal-utility result cap it.

def recommend_budget(summary, mu_floor_budget=None):
    """summary: output of summarize(); mu_floor_budget: cap from the marginal-utility eval"""
    p95 = summary["p95"]
    hit = summary["hit_rate"]
 
    base = p95 * 1.3                  # default: 1.3x p95
    if hit > 0.10:
        base = p95 * 1.8              # frequent truncation -> add headroom
 
    rec = int(round(base / 1000) * 1000)
    if mu_floor_budget is not None:
        rec = min(rec, mu_floor_budget)   # cap where utility flattens
    return max(1000, rec)
 
# e.g. summarize->{"p95":5200,"hit_rate":0.02}, mu_floor_budget=4000 -> recommends 4000

When the distribution recommendation (p95 x 1.3 ~ 6,800) collides with the marginal-utility cap (4,000), the cap wins. The distribution only tells you how far reasoning can stretch; it says nothing about whether that reasoning reached the answer. The one exception is workloads with a hit-rate above 10%: truncation may be eating accuracy there, so re-measure marginal utility one step higher before capping.

When the hit-rate is high but accuracy won't move

The tricky case is a high hit-rate — reasoning pinned to the ceiling — where raising the budget still doesn't lift accuracy. That is often not under-budget but a task too hard for the model. I separated the two like this:

Double the budget and rerun the eval set.
Accuracy rises significantly -> under-budget. Raise the recommendation.
Accuracy flat and hit-rate still high -> overloaded task. Suspect the input design, not the budget.

For case 3, no amount of budget helps. Move to a different lever: structure the input, spell out the choices, trim long context to the relevant span, or switch to a stronger model. Raising the reasoning ceiling only helps problems that have room to think — an obvious point that the pairing of hit-rate and accuracy makes concrete.

Cost lands only on hard tasks — attribute it per request

An over-large budget's cost is not fixed; it appears only when a hard task shows up. That is exactly why "average thinking tokens x price" misestimates reality. Attribute cost per request from real spend, and you see which task type is driving the bill.

def request_cost(usage, *, in_rate, out_rate):
    """rate = price per 1M tokens (substitute your contract's values).
    Note: thinking tokens are billed on the output side."""
    input_cost = usage.input_tokens / 1_000_000 * in_rate
    output_cost = (usage.output_tokens + usage.thinking_tokens) / 1_000_000 * out_rate
    return round(input_cost + output_cost, 6)
 
# At 100k req/month, going budget 32,000 -> 4,000 cut avg thinking by 3,000 tokens.
# With an assumed output rate of $75/Mtok:
#   100k * 3,000 / 1e6 * 75 = $22,500/month saved
# Weigh that against the accuracy lost in the marginal-utility eval (here, effectively zero).

Rates vary by contract and model, so substitute your own. What matters is not the absolute figure but the shape: weigh the savings against the accuracy the marginal-utility eval says you would lose. On my classification job the loss sat inside measurement error, so the savings dropped straight to the bottom line.

If you do raise the budget, pair it with timeouts

When marginal utility clearly justifies a higher budget, design the latency alongside it. Deep reasoning can take tens of seconds and will trip a default HTTP client timeout. Extend the client timeout past 90 seconds and stream long reasoning. For large offline batches, pairing this with the Batch API cost-optimization guide keeps real cost down even at a higher budget. Anything without a real-time requirement is worth pushing onto Batch.

Where to start

Next time you reach for budget_tokens, before moving the value, collect 100 production thinking_tokens and compute p50, p95, and hit-rate. Then sweep the budget over a fixed eval set and put the accuracy gained per 1,000 thinking tokens into a single table. With those two in hand, budget stops being something you decide and becomes something you read. Until I built this, I kept paying "big to be safe." Only after measuring did I see that most of the thinking I paid for never reached the answer.

If you are caught between the bill and the accuracy, I hope this gives you a place to start with your hands on the numbers. Thank you for reading.

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.