A Two-Tier Setup — Haiku 4.5 Orchestrator with Opus 4.6 Worker for Balancing Cost and Quality

What Goes to the Worker — Five Routing Criteria

"Escalate to Opus when it feels hard" is not a reproducible heuristic. In production I put five explicit criteria in the router's system prompt and ask it to return them as JSON:

• output_long: the response is likely to exceed 2,000 tokens
• multi_hop: the task requires 5+ logical hops
• domain_voice: it requires strong domain-specific context or terminology
• differentiation: it must clearly differentiate from similar existing output
• architecture: it concerns long-lived architectural decisions

If two or more are true, escalate to Opus. Otherwise, Haiku handles it inline. The core of the implementation:

import os
import json
from anthropic import Anthropic
 
client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
 
ROUTER_SYSTEM = """You are an orchestrator. Given a task spec,
decide whether to delegate to Opus 4.6 and return JSON only.
 
Criteria:
- output_long: response likely exceeds 2000 tokens
- multi_hop: 5+ logical hops required
- domain_voice: requires personal voice / first-person experience
- differentiation: must differentiate from similar existing output
- architecture: involves long-lived architectural decisions
 
If two or more are true, escalate=true.
 
Schema:
{
  "output_long": bool,
  "multi_hop": bool,
  "domain_voice": bool,
  "differentiation": bool,
  "architecture": bool,
  "escalate": bool,
  "reason": "short justification"
}
"""
 
def route(task_spec: dict) -> dict:
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=400,
        system=ROUTER_SYSTEM,
        messages=[{
            "role": "user",
            "content": json.dumps(task_spec, ensure_ascii=False)
        }]
    )
    return json.loads(response.content[0].text)

The router caps max_tokens at 400 and demands JSON-only output. Haiku 4.5 is fast and stable when constrained this way; the system prompt does the work of keeping it from adding chatty preambles. In my pipeline the average routing decision lands at about 1.2 seconds and roughly $0.0006 per call.

Invoking the Worker and Carrying State Forward

When the router returns escalate=true, I call Opus 4.6. The detail that matters more than I expected: feed the router's reason into Opus alongside the task itself, as a two-layer context. Opus then knows why it was called, and the response leans harder into the dimension the router cared about.

def delegate(task_spec: dict, routing: dict) -> str:
    if not routing["escalate"]:
        return run_haiku_inline(task_spec)
    response = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=8000,
        system=(
            "You are a senior writer. The upstream router escalated this "
            f"task to you because: {routing['reason']}\n"
            "Lean into long-form depth, logical chaining, and personal "
            "experience as the reason suggests."
        ),
        messages=[{
            "role": "user",
            "content": build_prompt(task_spec)
        }]
    )
    return response.content[0].text

I didn't include the router's reason in the early version. Opus produced perfectly fine prose, but it would occasionally drift from what the router had flagged, and my downstream quality gate (a small script that mechanically inspects each output) would bounce it back. After I started piping the reason through, the first-pass quality gate acceptance rate moved from 62% to 84%. That's not a vanity metric — every bounced job translates into a regeneration that costs API tokens again.

Streaming vs Blocking — Where Each Belongs

The router returns short structured JSON, so streaming buys almost nothing there. Trimming 1.2s of latency is less valuable than the clean error-handling you get from waiting for the full response and then parsing JSON.

Opus, on the other hand, is a strong fit for streaming. Long-form output (5,000–8,000 tokens) gives the early bytes time to be inspected. I run a lightweight keyword scan on the first few hundred tokens and abort early if obvious template phrasing, banned words, or hype-style headlines slip through.

def stream_with_early_abort(prompt: str, banned: list[str]) -> str:
    buffer = []
    with client.messages.stream(
        model="claude-opus-4-6",
        max_tokens=8000,
        messages=[{"role": "user", "content": prompt}],
    ) as stream:
        for text in stream.text_stream:
            buffer.append(text)
            joined = "".join(buffer)
            for word in banned:
                if word in joined:
                    raise EarlyAbort(word)
            if len(joined) > 600 and looks_like_template_intro(joined):
                raise EarlyAbort("template_intro")
        return "".join(buffer)

Beyond improving the first-pass acceptance rate, the early abort saves real tokens. On average, an aborted run saves about 3,500 output tokens, which translates to 5–8% additional monthly savings on top of the routing savings.

Three Pitfalls I Hit Running the Hybrid

The theory cleans up nicely on a whiteboard. Real operation surfaces stranger problems. Three for the record.

1. The router collapses to "always escalate"

For the first two weeks, the Haiku router escalated nearly everything. The cause was a single line in the system prompt: "when in doubt, escalate=true." Haiku is conscientious; given that nudge, it doubts most things. I changed it to "when in doubt, escalate=false — Haiku can handle this itself," and shifted the rule to a numeric threshold of two-or-more true criteria. The escalation rate dropped from 92% to roughly 30%, and the two-tier shape started to actually function.

2. Cache-key churn destroys latency

I was running prompt caching on the Opus side, but I had been inserting the router's reason into the system prompt. The cache key changed with every request, the hit rate fell to near-zero, and average end-to-end latency degraded from 4.8s to 11.3s.

The fix was to keep the system prompt fixed and pass the router's reason through messages instead, then place an ephemeral cache breakpoint at the end of the system prompt with cache_control. Cache hit rate climbed back to ~78%, and latency settled around 5.2s. If you want to push caching further up the stack, the similarity thresholds I worked out in putting a semantic cache for the Claude API into production are the natural next joint.

3. Two-layer logging blows up storage

Storing the router's full response next to Opus's full response increased my monthly log volume by 2.4x. Cloudflare R2 storage costs scale modestly, but at that ratio they start to be noticeable.

I now only persist escalate and a truncated reason from the router and keep the Opus output verbatim. When I need the raw router response for debugging, I write it to an ephemeral KV with a short TTL. Log volume returned to prior levels.

Monthly Cost Math — Running 16 Jobs a Day

Abstract per-token prices are easier to internalize when grounded in your own workload. My setup runs about 16 long-form jobs a day, so roughly 480 jobs per month.

Average token consumption per job looks like this: router input 1,500, router output 300, Opus input 3,000, Opus output 5,000. Assume 30% escalation:

• Haiku router input: 480 × 1,500 = 720,000 tokens
• Haiku router output: 480 × 300 = 144,000 tokens
• Opus input (30% escalation): 144 × 3,000 = 432,000 tokens
• Opus output (30% escalation): 144 × 5,000 = 720,000 tokens
• Haiku body work (70%): 336 × 4,500 = 1,512,000 tokens

At current reference prices, that lands around $25 a month. The year-prior Opus-only baseline at the same scale was about $90 a month — roughly $65 in monthly delta and $780 annually. At indie budget scale, that gap is the difference between running one more experiment a month and not running it.

One subtlety: escalation rate isn't a constant. In November–December, when I run more promotional long-form work, my escalation rate climbs to 45% and monthly cost jumps near $35. Months dominated by short, formulaic output sit around 18% and stay under $20. Tracking this number monthly is the rhythm that makes the routing prompt actually improve over time.

Logging Design — Squeeze a Second Use From Router Output

It feels wasteful to use the router's JSON only for the escalation decision. I now persist it in Cloudflare D1 as a structured log for downstream analysis.

import sqlite3
import datetime
 
def log_routing(task_id, task_spec, routing, escalated):
    conn = sqlite3.connect("routing.db")
    conn.execute(
        """
        INSERT INTO routing_log
        (task_id, ts, topic, output_long, multi_hop, domain_voice,
         differentiation, architecture, escalate, reason)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
        """,
        (
            task_id,
            datetime.datetime.utcnow().isoformat(),
            task_spec.get("topic", ""),
            int(routing["output_long"]),
            int(routing["multi_hop"]),
            int(routing["domain_voice"]),
            int(routing["differentiation"]),
            int(routing["architecture"]),
            int(escalated),
            routing["reason"][:200],
        ),
    )
    conn.commit()

With this log, end-of-month reviews can ask: which criterion contributes most to escalation? In my data, when domain_voice is true, escalation is effectively 100%, while output_long alone only triggers escalation about 32% of the time. Once you can measure that, tuning the router prompt becomes a quantitative exercise rather than a guess.

The multi_hop criterion is the hardest for Haiku to judge consistently. I shore it up by adding a small handful of summaries from past jobs that escalated on multi_hop=true directly into the router's system prompt as few-shot examples. Subjectively, the rate of agreement improved by maybe 20%.

Three Approaches I Tried and Discarded

Before I landed on this two-tier shape, I tried a few alternatives. The lessons sit in what I gave up.

The first was "Opus only, with lower temperature." The idea was that shorter outputs save tokens. Dropping temperature to 0.2 did cut output length by about 20%, but it also flattened the prose. The texture went out of the output. Unit costs fell slightly, but the time I spent reworking results by hand went up, so total cost moved the wrong way.

The second was "Sonnet only." Cheaper than Opus, smarter than Haiku — a tempting middle. I ran it for about six months. For complex long-form work, Sonnet's depth didn't quite match Opus, and quality-gate bounces became more frequent. The cost of regenerating bounced jobs eventually exceeded the cost of just using Opus selectively for those cases.

The third was "isolate the router as a separate Cloudflare Workers app." Routing decisions are stateless, so they ought to run at the edge. Router-only latency did drop to 0.4s. But the Workers CPU/memory ceilings and the Anthropic SDK's dependency tree made the operational overhead worse than the latency win. Today the router happily co-locates with the Python runtime running everything else.

The shared lesson across these failed paths: optimizing only the per-token unit cost narrows your field of vision. Regeneration cost, operational load, and downstream rework time all need to be included in the same equation. The slightly more elaborate two-tier ended up being cheaper in total terms.

A Three-Tier Variant — Where Sonnet 4.6 Fits

Once a two-tier pipeline is running smoothly, it's worth thinking about a three-tier variant with Sonnet 4.6 in the middle. As the batch grew, I found a band of tasks that didn't quite need Opus but felt under-served by Haiku alone.

The three-tier shape treats output_long as a graded threshold: under ~1,200 tokens stays on Haiku, 1,200–3,500 routes to Sonnet, 3,500+ goes to Opus. Sonnet's ~$3.00 per 1M is a sweet spot for that band, and once it stabilized I saw an additional 10–15% monthly savings on top of the two-tier baseline.

The trade-off is operational complexity: more prompt versions, more quality gates, more dashboards. My recommendation is to live with the two-tier shape for six months until your metrics feel stable, then expand to three.

Metrics to Watch — Keeping the Orchestrator Healthy

"It's working" isn't a metric. I review these on a Grafana board every day:

• Escalation rate (target band: 20–40%)
• Average router decision latency
• First-pass acceptance rate from the quality gate on Opus output
• Prompt cache hit rate on Opus input
• Total cost per job (Haiku + Opus combined)

Escalation rate is the most important health indicator. Below 20% suggests Haiku alone might be sufficient for more tasks than you realized. Above 40% suggests over-reliance on Opus. My monthly cadence is to read those numbers, tweak the router system prompt, and watch the rate drift back into the band.

Revisiting the Criteria Monthly — Revising From Router Logs

The weak point of a two-tier setup is that the routing criteria freeze at whatever you assumed the day you wrote them. Task mix shifts within weeks. In my case, adding a single new job type pushed the output_long distribution to the right, and the escalation rate climbed from 30% to 44% before I noticed it — on the month-end invoice, not in a dashboard.

So I now set aside time once a month to reread the router log. There are only three things to do:

1. Break down the last 30 days of escalation rate by task type
2. Pull cases that were escalated to Opus but produced a short output (i.e. the escalation may not have been necessary)
3. Pull cases that Haiku handled but the downstream quality gate rejected (i.e. they should have been escalated)

Items 2 and 3 are your false positives and false negatives. Collapsing router accuracy into a single number makes it easy to draw the wrong conclusion, so I count these two directions separately.

Add just two columns to the routing_log table from the previous section — the actual output token count and the quality gate result — and the whole review becomes a single SQL statement.

import sqlite3
 
REVIEW_SQL = """
SELECT
  CASE
    WHEN escalate = 1 AND actual_output_tokens < 1200 THEN 'false_positive'
    WHEN escalate = 0 AND gate_rejected = 1           THEN 'false_negative'
    ELSE 'ok'
  END AS verdict,
  COUNT(*)                         AS n,
  ROUND(AVG(actual_output_tokens)) AS avg_tokens
FROM routing_log
WHERE ts >= date('now', '-30 day')
GROUP BY verdict
ORDER BY n DESC
"""
 
def monthly_review(db_path: str = "routing.db"):
    conn = sqlite3.connect(db_path)
    rows = conn.execute(REVIEW_SQL).fetchall()
    total = sum(r[1] for r in rows) or 1
    for verdict, n, avg_tokens in rows:
        print(f"{verdict:<15} {n:>4} ({n / total:.1%})  avg output {avg_tokens} tok")
    return rows

The actual_output_tokens < 1200 threshold is deliberately the same number the router uses for its output_long flag. If the criterion you route on and the criterion you audit with are different numbers, you lose the ability to tell whether the threshold is wrong or the classification is wrong. Pinning both sides to one value is the one decision worth making up front.

This script does not decide anything. It only assembles candidates. Whether I actually change a threshold, I decide after reading a handful of the flagged cases myself. The month I raised output_long from 1,200 to 1,600, false positives dropped from 18 to 5 and quality gate rejections rose by only 2. That trade I was willing to take.

Write the criteria down, hold them against what actually happened, and revise when the gap is real. Running that loop once a month is what lets you catch a slowly degrading cost structure before the invoice does.

Folding the August 2026 Price Change Into the Middle Tier

Everything above was measured on the May 2026 configuration. Two things shifted in August, so here is what a running two-tier setup should revisit.

The first is the default model. On August 3rd, Sonnet 5 became the default on Pro, Team Standard, and Enterprise seats; on August 5th, Opus 5 shipped and became the default on Max. What makes this awkward for a two-tier setup is the shape of the change: without touching a single setting, the orchestrator and the worker can swap underneath you at the same time.

Escalation rate depends on both the router's judgment tendencies and the worker's actual capability. When one side moves, the number gets noisy. When both move together, the 20–40% band you have been monitoring stops meaning what it used to mean.

The fix is unglamorous: pin model as an explicit string on both the router and the worker. An implementation that leans on the default is one you find out about from an invoice. For a second line of defense on the spend side, the circuit breaker in enforcing a hard budget ceiling on Claude API in production is the piece that stops the bleeding while you investigate.

The second is pricing. Sonnet 5's promotional rate of $2 / $10 per Mtok runs through August 31st, 2026, and returns to $3 / $15 per Mtok on September 1st. If you have Sonnet sitting in the middle of a three-tier setup, that lands squarely on your bill.

Take the monthly model from the previous section — 480 jobs, 30% escalation — and move half of the 144 escalated jobs down into the middle tier. Keeping the same 3,000-in / 5,000-out profile, the middle tier handles 0.216M input tokens and 0.36M output tokens.

That is $2.02 more per month, a 50% increase. Against the $25 monthly total from the earlier section, the middle tier's share rises from 16.1% to 24.2%. Small in absolute terms, but proportionally the middle tier starts to carry real weight.

Running the same volume back through Opus (roughly $15 per 1M blended) would cost $8.64, so the saving from keeping a middle tier shrinks from $4.61 to $2.59. The middle tier still pays for itself after September 1st — it just pays about half as well.

The practical conclusion is that September 1st is a natural date to move the middle-tier threshold in one direction or the other. Which direction depends on one question: since moving work down to Sonnet, have downstream rejections gone up? If they haven't, widen the band and pull more work off Opus. If they have, the higher rate makes the middle tier a poor trade, so narrow the band and fall back toward two tiers.

I can answer that question straight out of the routing_log aggregation from the earlier section, with the quality gate result joined in. Not having to redesign anything when a price moves is the payoff for putting the deciding numbers in one place ahead of time.

One caveat worth stating plainly: prices, availability, and defaults change. Confirm current values from primary sources before applying any of these numbers to your own setup.

Where to Start

If you're thinking of trying this two-tier shape, I'd suggest staging it like this rather than swapping the whole pipeline at once:

1. Pick one stable classification task (tagging, categorization, metadata extraction) and move it to Haiku-only. Confirm quality holds for two weeks.
2. Identify a task with variable output quality (long-form prose, code generation) and put a router in front of it.
3. Tune the router system prompt until the escalation rate settles in the 20–40% band.
4. Add prompt caching to the Opus side. Keep the system prompt static so the cache breakpoint can actually hit.
5. Build a small dashboard for escalation rate, cache hit rate, and per-job cost. Review weekly for the first month, monthly afterward.

I didn't follow this order myself. I jumped in, hit each of the three pitfalls in turn, and learned them the slow way. If you take a calmer entry, you'll spend less time refunding yourself in the form of API regenerations.

The real value of a two-tier setup, I've come to think, isn't the cost cut. It's that splitting judgment from production work surfaces the quality of your judgment as something you can measure separately.

The one thing worth doing today: pick a single job you already have running and check whether its model is pinned as a literal string. If it isn't, fix that first. Measuring escalation rate can wait until after.