A Two-Tier Setup — Haiku 4.5 Orchestrator with Opus 4.6 Worker for Balancing Cost and Quality

What Goes to the Worker — Five Routing Criteria

"Escalate to Opus when it feels hard" is not a reproducible heuristic. In production I put five explicit criteria in the router's system prompt and ask it to return them as JSON:

• output_long: the response is likely to exceed 2,000 tokens
• multi_hop: the task requires 5+ logical hops
• domain_voice: it requires strong domain-specific context or terminology
• differentiation: it must clearly differentiate from similar existing output
• architecture: it concerns long-lived architectural decisions

If two or more are true, escalate to Opus. Otherwise, Haiku handles it inline. The core of the implementation:

import os
import json
from anthropic import Anthropic
 
client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
 
ROUTER_SYSTEM = """You are an orchestrator. Given a task spec,
decide whether to delegate to Opus 4.6 and return JSON only.
 
Criteria:
- output_long: response likely exceeds 2000 tokens
- multi_hop: 5+ logical hops required
- domain_voice: requires personal voice / first-person experience
- differentiation: must differentiate from similar existing output
- architecture: involves long-lived architectural decisions
 
If two or more are true, escalate=true.
 
Schema:
{
  "output_long": bool,
  "multi_hop": bool,
  "domain_voice": bool,
  "differentiation": bool,
  "architecture": bool,
  "escalate": bool,
  "reason": "short justification"
}
"""
 
def route(task_spec: dict) -> dict:
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=400,
        system=ROUTER_SYSTEM,
        messages=[{
            "role": "user",
            "content": json.dumps(task_spec, ensure_ascii=False)
        }]
    )
    return json.loads(first_text(response))
 
 
def first_text(response) -> str:
    for block in response.content:
        if block.type == "text":
            return block.text
    raise ValueError("text block not found in response")

The one thing I won't shorten is first_text(). Writing response.content[0].text works right up until you point the router at a model where thinking is on by default — Sonnet 5 and Opus 5, for instance — at which point content[0] is a thinking block and .text raises AttributeError. Swapping a model generation breaks the pipeline somewhere entirely unrelated to cost. It's the quietest version of the "leaning on defaults" problem I come back to later.

The router caps max_tokens at 400 and demands JSON-only output. Haiku 4.5 is fast and stable when constrained this way; the system prompt does the work of keeping it from adding chatty preambles. In my pipeline the average routing decision lands at about 1.2 seconds and roughly $0.0006 per call.

Invoking the Worker and Carrying State Forward

When the router returns escalate=true, I call Opus 4.6. The detail that matters more than I expected: feed the router's reason into Opus alongside the task itself, as a two-layer context. Opus then knows why it was called, and the response leans harder into the dimension the router cared about.

def delegate(task_spec: dict, routing: dict) -> str:
    if not routing["escalate"]:
        return run_haiku_inline(task_spec)
    response = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=8000,
        system=(
            "You are a senior writer. The upstream router escalated this "
            f"task to you because: {routing['reason']}\n"
            "Lean into long-form depth, logical chaining, and personal "
            "experience as the reason suggests."
        ),
        messages=[{
            "role": "user",
            "content": build_prompt(task_spec)
        }]
    )
    return first_text(response)

I didn't include the router's reason in the early version. Opus produced perfectly fine prose, but it would occasionally drift from what the router had flagged, and my downstream quality gate (a small script that mechanically inspects each output) would bounce it back. After I started piping the reason through, the first-pass quality gate acceptance rate moved from 62% to 84%. That's not a vanity metric — every bounced job translates into a regeneration that costs API tokens again.

Streaming vs Blocking — Where Each Belongs

The router returns short structured JSON, so streaming buys almost nothing there. Trimming 1.2s of latency is less valuable than the clean error-handling you get from waiting for the full response and then parsing JSON.

Opus, on the other hand, is a strong fit for streaming. Long-form output (5,000–8,000 tokens) gives the early bytes time to be inspected. I run a lightweight keyword scan on the first few hundred tokens and abort early if obvious template phrasing, banned words, or hype-style headlines slip through.

def stream_with_early_abort(prompt: str, banned: list[str]) -> str:
    parts: list[str] = []
    window = ""
    keep = max((len(w) for w in banned), default=1) - 1
    seen = 0
    intro_checked = False
 
    with client.messages.stream(
        model="claude-opus-4-6",
        max_tokens=8000,
        messages=[{"role": "user", "content": prompt}],
    ) as stream:
        for text in stream.text_stream:
            parts.append(text)
            seen += len(text)
 
            window += text
            for word in banned:
                if word in window:
                    stream.close()
                    raise EarlyAbort(word)
            window = window[-keep:] if keep > 0 else ""
 
            if not intro_checked and seen > 600:
                intro_checked = True
                if looks_like_template_intro("".join(parts)):
                    stream.close()
                    raise EarlyAbort("template_intro")
 
    return "".join(parts)

I rewrote this twice before it settled. The first version rebuilt the entire transcript with "".join(buffer) on every chunk, then scanned that whole string once per banned word. Across an 8,000-token response the per-chunk work keeps growing — a guard meant to save money became most expensive precisely when it never fired.

Now I keep only a tail window as long as the longest banned word, so a phrase split across two chunks still matches. The template-intro check is a one-time judgment about how the piece opens, so it runs once, the first time the response passes 600 characters. stream.close() is explicit because raising out of the loop should not leave the connection to chance.

Beyond improving the first-pass acceptance rate, the early abort saves real tokens. On average, an aborted run saves about 3,500 output tokens, which translates to 5–8% additional monthly savings on top of the routing savings.

Three Pitfalls I Hit Running the Hybrid

The theory cleans up nicely on a whiteboard. Real operation surfaces stranger problems. Three for the record.

1. The router collapses to "always escalate"

For the first two weeks, the Haiku router escalated nearly everything. The cause was a single line in the system prompt: "when in doubt, escalate=true." Haiku is conscientious; given that nudge, it doubts most things. I changed it to "when in doubt, escalate=false — Haiku can handle this itself," and shifted the rule to a numeric threshold of two-or-more true criteria. The escalation rate dropped from 92% to roughly 30%, and the two-tier shape started to actually function.

2. Cache-key churn destroys latency

I was running prompt caching on the Opus side, but I had been inserting the router's reason into the system prompt. The cache key changed with every request, the hit rate fell to near-zero, and average end-to-end latency degraded from 4.8s to 11.3s.

The fix was to keep the system prompt fixed and pass the router's reason through messages instead, then place an ephemeral cache breakpoint at the end of the system prompt with cache_control. Cache hit rate climbed back to ~78%, and latency settled around 5.2s. If you want to push caching further up the stack, the similarity thresholds I worked out in putting a semantic cache for the Claude API into production are the natural next joint.

3. Two-layer logging blows up storage

Storing the router's full response next to Opus's full response increased my monthly log volume by 2.4x. Cloudflare R2 storage costs scale modestly, but at that ratio they start to be noticeable.

I now only persist escalate and a truncated reason from the router and keep the Opus output verbatim. When I need the raw router response for debugging, I write it to an ephemeral KV with a short TTL. Log volume returned to prior levels.

Monthly Cost Math — Running 16 Jobs a Day

Abstract per-token prices are easier to internalize when grounded in your own workload. My setup runs about 16 long-form jobs a day, so roughly 480 jobs per month.

Average token consumption per job looks like this: router input 1,500, router output 300, Opus input 3,000, Opus output 5,000. Assume 30% escalation:

• Haiku router input: 480 × 1,500 = 720,000 tokens
• Haiku router output: 480 × 300 = 144,000 tokens
• Opus input (30% escalation): 144 × 3,000 = 432,000 tokens
• Opus output (30% escalation): 144 × 5,000 = 720,000 tokens
• Haiku body work (70%): 336 × 4,500 = 1,512,000 tokens

The 4,500 tokens of Haiku body work break down as 1,500 input and 3,000 output. Applied to published rates, the bill lines up like this:

The same 480 jobs run Opus-4.6-only (3,000 in, 5,000 out) come to $7.20 of input and $60.00 of output: $67.20. The gap is $40.06 a month, roughly $480 a year, a 59.6% reduction.

Building that table is what finally made the mechanism clear to me. Opus output alone — $18.00 — is 66% of the bill. Moving the orchestrator from Opus to Haiku saves less than a dollar a month. The entire result comes from one place: cutting the output tokens that reach Opus by 70%. Everything else I tuned sat inside the rounding error.

What you're cutting isn't call volume. It's output tokens from the expensive model. If you're weighing whether this shape is worth the plumbing, count what share of your bill is output tokens before you build anything. Below about half, the effort is unlikely to pay you back.

Escalation rate isn't a constant, either. In November–December, when I run more promotional long-form work, it climbs to 45% and the same arithmetic gives about $36 a month. Months dominated by short, formulaic output sit near 18% and land around $20. Tracking this number monthly is the rhythm that makes the routing prompt actually improve over time.

Logging Design — Squeeze a Second Use From Router Output

It feels wasteful to use the router's JSON only for the escalation decision. I now persist it in Cloudflare D1 as a structured log for downstream analysis.

import datetime
import sqlite3
from contextlib import closing
 
def log_routing(task_id, task_spec, routing, escalated, db_path="routing.db"):
    ts = datetime.datetime.now(datetime.timezone.utc).isoformat()
    with closing(sqlite3.connect(db_path)) as conn, conn:
        conn.execute(
            """
        INSERT INTO routing_log
        (task_id, ts, topic, output_long, multi_hop, domain_voice,
         differentiation, architecture, escalate, reason)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
        """,
        (
            task_id,
            ts,
            task_spec.get("topic", ""),
            int(routing["output_long"]),
            int(routing["multi_hop"]),
            int(routing["domain_voice"]),
            int(routing["differentiation"]),
            int(routing["architecture"]),
            int(escalated),
                routing["reason"][:200],
            ),
        )

Two things here I got wrong the first time. datetime.datetime.utcnow() has been deprecated since Python 3.12, and it returns a naive datetime — a string you meant as UTC but which carries no evidence of it. Since the monthly review slices "the last 30 days" out of this table, that ambiguity is not something I want sitting in the timestamp column. datetime.now(datetime.timezone.utc) keeps the trailing +00:00.

The connection handling was the other one. conn.commit() alone leaves the connection open if anything raises on the way there. closing() guarantees the close and with conn: handles commit and rollback — the two-layer form is the one that stops leaking in SQLite.

With this log, end-of-month reviews can ask: which criterion contributes most to escalation? In my data, when domain_voice is true, escalation is effectively 100%, while output_long alone only triggers escalation about 32% of the time. Once you can measure that, tuning the router prompt becomes a quantitative exercise rather than a guess.

The multi_hop criterion is the hardest for Haiku to judge consistently. I shore it up by adding a small handful of summaries from past jobs that escalated on multi_hop=true directly into the router's system prompt as few-shot examples. Subjectively, the rate of agreement improved by maybe 20%.

Three Approaches I Tried and Discarded

Before I landed on this two-tier shape, I tried a few alternatives. The lessons sit in what I gave up.

The first was "Opus only, with lower temperature." The idea was that shorter outputs save tokens. Dropping temperature to 0.2 did cut output length by about 20%, but it also flattened the prose. The texture went out of the output. Unit costs fell slightly, but the time I spent reworking results by hand went up, so total cost moved the wrong way.

The second was "Sonnet only." Cheaper than Opus, smarter than Haiku — a tempting middle. I ran it for about six months. For complex long-form work, Sonnet's depth didn't quite match Opus, and quality-gate bounces became more frequent. The cost of regenerating bounced jobs eventually exceeded the cost of just using Opus selectively for those cases.

The third was "isolate the router as a separate Cloudflare Workers app." Routing decisions are stateless, so they ought to run at the edge. Router-only latency did drop to 0.4s. But the Workers CPU/memory ceilings and the Anthropic SDK's dependency tree made the operational overhead worse than the latency win. Today the router happily co-locates with the Python runtime running everything else.

The shared lesson across these failed paths: optimizing only the per-token unit cost narrows your field of vision. Regeneration cost, operational load, and downstream rework time all need to be included in the same equation. The slightly more elaborate two-tier ended up being cheaper in total terms.

A Three-Tier Variant — Where Sonnet 4.6 Fits

Once a two-tier pipeline is running smoothly, it's worth thinking about a three-tier variant with Sonnet 4.6 in the middle. As the batch grew, I found a band of tasks that didn't quite need Opus but felt under-served by Haiku alone.

The three-tier shape treats output_long as a graded threshold: under ~1,200 tokens stays on Haiku, 1,200–3,500 routes to Sonnet, 3,500+ goes to Opus. Sonnet's ~$3.00 per 1M is a sweet spot for that band, and once it stabilized I saw an additional 10–15% monthly savings on top of the two-tier baseline.

The trade-off is operational complexity: more prompt versions, more quality gates, more dashboards. My recommendation is to live with the two-tier shape for six months until your metrics feel stable, then expand to three.

Metrics to Watch — Keeping the Orchestrator Healthy

"It's working" isn't a metric. I review these on a Grafana board every day:

• Escalation rate (target band: 20–40%)
• Average router decision latency
• First-pass acceptance rate from the quality gate on Opus output
• Prompt cache hit rate on Opus input
• Total cost per job (Haiku + Opus combined)

Escalation rate is the most important health indicator. Below 20% suggests Haiku alone might be sufficient for more tasks than you realized. Above 40% suggests over-reliance on Opus. My monthly cadence is to read those numbers, tweak the router system prompt, and watch the rate drift back into the band.

Revisiting the Criteria Monthly — Revising From Router Logs

The weak point of a two-tier setup is that the routing criteria freeze at whatever you assumed the day you wrote them. Task mix shifts within weeks. In my case, adding a single new job type pushed the output_long distribution to the right, and the escalation rate climbed from 30% to 44% before I noticed it — on the month-end invoice, not in a dashboard.

So I now set aside time once a month to reread the router log. There are only three things to do:

1. Break down the last 30 days of escalation rate by task type
2. Pull cases that were escalated to Opus but produced a short output (i.e. the escalation may not have been necessary)
3. Pull cases that Haiku handled but the downstream quality gate rejected (i.e. they should have been escalated)

Items 2 and 3 are your false positives and false negatives. Collapsing router accuracy into a single number makes it easy to draw the wrong conclusion, so I count these two directions separately.

Add just two columns to the routing_log table from the previous section — the actual output token count and the quality gate result — and the whole review becomes a single SQL statement.

import sqlite3
 
REVIEW_SQL = """
SELECT
  CASE
    WHEN escalate = 1 AND actual_output_tokens < 1200 THEN 'false_positive'
    WHEN escalate = 0 AND gate_rejected = 1           THEN 'false_negative'
    ELSE 'ok'
  END AS verdict,
  COUNT(*)                         AS n,
  ROUND(AVG(actual_output_tokens)) AS avg_tokens
FROM routing_log
WHERE ts >= date('now', '-30 day')
GROUP BY verdict
ORDER BY n DESC
"""
 
def monthly_review(db_path: str = "routing.db"):
    conn = sqlite3.connect(db_path)
    rows = conn.execute(REVIEW_SQL).fetchall()
    total = sum(r[1] for r in rows) or 1
    for verdict, n, avg_tokens in rows:
        print(f"{verdict:<15} {n:>4} ({n / total:.1%})  avg output {avg_tokens} tok")
    return rows

The actual_output_tokens < 1200 threshold is deliberately the same number the router uses for its output_long flag. If the criterion you route on and the criterion you audit with are different numbers, you lose the ability to tell whether the threshold is wrong or the classification is wrong. Pinning both sides to one value is the one decision worth making up front.

This script does not decide anything. It only assembles candidates. Whether I actually change a threshold, I decide after reading a handful of the flagged cases myself. The month I raised output_long from 1,200 to 1,600, false positives dropped from 18 to 5 and quality gate rejections rose by only 2. That trade I was willing to take.

Write the criteria down, hold them against what actually happened, and revise when the gap is real. Running that loop once a month is what lets you catch a slowly degrading cost structure before the invoice does.

The Increase That Never Came, and What Actually Moved

Everything above was measured on the May 2026 configuration. Several assumptions have shifted since, so here is what I went back and checked.

I'll start with my own mistake. I had been working from the premise that Sonnet 5's promotional rate of $2 / $10 per MTok ran through August 31st and returned to $3 / $15 per MTok on September 1st, and I narrowed the middle-tier band accordingly. If the middle tier is about to get expensive, pull work back out of it.

That increase never happened. The $2 / $10 rate became the standard price on August 10th, and Anthropic's pricing page states plainly that the September 1st change will not occur. I spent two weeks starving a middle tier in preparation for a price that was never charged. The bill didn't drop; downstream rejections ticked up slightly.

A scheduled price and a charged price are not the same thing. Opening the pricing page before moving the design would have cost me a minute. Skipping it cost two weeks of the wrong shape.

What actually moved was model generations, not rates

Checking the dates again, the thing I had filed away as "the August change" wasn't pricing at all. Sonnet 5 (claude-sonnet-5) shipped on June 30th; Opus 5 (claude-opus-5) shipped on July 24th at $5 / $25 per MTok, the same rate as Opus 4.8.

What makes generation changes awkward for a two-tier setup is their shape: without touching a single setting, the orchestrator and the worker can swap underneath you at the same time. Escalation rate depends on the router's judgment tendencies and the worker's capability. When one side moves, the number gets noisy. When both move together, the 20–40% band you have been monitoring stops meaning what it used to mean.

The fix is unglamorous: pin model as an explicit string on both the router and the worker. An implementation that leans on defaults is one you find out about from an invoice. For a second line of defense on spend, the circuit breaker in enforcing a hard budget ceiling on Claude API in production is the piece that stops the bleeding while you investigate.

When the tokenizer changes, so does the ruler

This is the part the price table never shows. Claude 4.7 and later models — Sonnet 5, Opus 5 — use a newer tokenizer that produces roughly 30% more tokens for the same text. Sonnet 4.6 and earlier, including Haiku 4.5, use the previous one.

That lands on a two-tier setup twice.

The first is cost. Moving the worker from Opus 4.6 to Opus 5 holds the rate at $5 / $25, but the same manuscript now counts as about 30% more tokens, so the invoice rises by roughly that much. "Same rate, so switching is cost-neutral" does not hold here.

The second is thresholds. The router in this article judges output_long at 2,000 tokens and the monthly review validates against 1,200 — both denominated in tokens. Leave the orchestrator on Haiku 4.5 (old tokenizer) and move only the worker to a new generation, and the ruler used for deciding stops matching the ruler used for measuring. The same manuscript reads 1,900 and "short" at decision time, 2,500 and "long" after the fact. That's the worst kind of mismatch: it removes your ability to tell whether the criterion is wrong or the judgment is.

If you migrate, restate the thresholds in characters, or re-measure them on the new generation first. I've pinned the worker to Opus 4.6 for now and put the threshold audit ahead of the move.

What a Sonnet 5 middle tier actually costs

Take the monthly model from the previous section — 480 jobs, 30% escalation — and move half of the 144 escalated jobs down into a middle tier. Keeping the 3,000-in / 5,000-out profile, the middle tier handles 0.216M input and 0.36M output tokens.

The second row is the one worth sitting with. On the rate card, Sonnet 5 looks 33% cheaper than Sonnet 4.6. Fold in the token increase and the gap narrows to 13%. Still the better middle tier — but worth less than half of what the rate card suggests.

I'm also withdrawing a figure I published here earlier: "running the same volume back through Opus costs $8.64." That came from the blended $15 per 1M approximation. Split into input and output at published rates, it's $10.08. On an output-dominated workload, a blended rate understates the expensive model by more than 10%.

For an overnight batch, the Batch API was the real lever

Having stacked all of this up, the biggest saving in my setup turned out not to be the middle-tier band at all. What this pipeline runs is a daily batch; nothing needs to come back immediately. The Batch API discounts input and output by 50%, so routing the 144 Opus jobs through it drops the $27.14 monthly total to $17.06 — a 74.6% reduction against the $67.20 Opus-only baseline. No amount of careful threshold tuning got close to that.

The only thing traded away is synchronous latency. Router decisions stay synchronous at 1.2 seconds each; only the worker's long-form generation moves to batch. That split held up in practice, and it depends entirely on having sorted your jobs beforehand into the ones that need an answer now and the ones that need an answer by morning.

One more line worth budgeting for: inference_geo: "us", which pins inference to the United States, applies a 1.1x multiplier across every category on Claude 4.6 and later. If you have a data residency requirement, fold it into the monthly estimate rather than discovering it later.

Prices, availability, and defaults change. I'd rather you check the current values on the pricing page before applying any of these numbers to your own setup.

Where to Start

If you're thinking of trying this two-tier shape, I'd suggest staging it like this rather than swapping the whole pipeline at once:

1. Pick one stable classification task (tagging, categorization, metadata extraction) and move it to Haiku-only. Confirm quality holds for two weeks.
2. Identify a task with variable output quality (long-form prose, code generation) and put a router in front of it.
3. Tune the router system prompt until the escalation rate settles in the 20–40% band.
4. Add prompt caching to the Opus side. Keep the system prompt static so the cache breakpoint can actually hit.
5. Build a small dashboard for escalation rate, cache hit rate, and per-job cost. Review weekly for the first month, monthly afterward.

I didn't follow this order myself. I jumped in, hit each of the three pitfalls in turn, and learned them the slow way. If you take a calmer entry, you'll spend less time refunding yourself in the form of API regenerations.

The real value of a two-tier setup, I've come to think, isn't the cost cut. It's that splitting judgment from production work surfaces the quality of your judgment as something you can measure separately.

The one thing worth doing today: pick a single job you already have running and check whether its model is pinned as a literal string. If it isn't, that's where I'd start. Measuring escalation rate can wait until after.