A Two-Tier Setup — Haiku 4.5 Orchestrator with Opus 4.6 Worker for Balancing Cost and Quality

What Goes to the Worker — Five Routing Criteria

"Escalate to Opus when it feels hard" is not a reproducible heuristic. In production I put five explicit criteria in the router's system prompt and ask it to return them as JSON:

• output_long: the response is likely to exceed 2,000 tokens
• multi_hop: the task requires 5+ logical hops
• author_voice: it requires strong personal voice or first-person experience
• differentiation: it must clearly differentiate from similar existing articles
• architecture: it concerns long-lived architectural decisions

If two or more are true, escalate to Opus. Otherwise, Haiku handles it inline. The core of the implementation:

import os
import json
from anthropic import Anthropic
 
client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
 
ROUTER_SYSTEM = """You are an orchestrator. Given a task spec,
decide whether to delegate to Opus 4.6 and return JSON only.
 
Criteria:
- output_long: response likely exceeds 2000 tokens
- multi_hop: 5+ logical hops required
- author_voice: requires personal voice / first-person experience
- differentiation: must differentiate from similar existing articles
- architecture: involves long-lived architectural decisions
 
If two or more are true, escalate=true.
 
Schema:
{
  "output_long": bool,
  "multi_hop": bool,
  "author_voice": bool,
  "differentiation": bool,
  "architecture": bool,
  "escalate": bool,
  "reason": "short justification"
}
"""
 
def route(task_spec: dict) -> dict:
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=400,
        system=ROUTER_SYSTEM,
        messages=[{
            "role": "user",
            "content": json.dumps(task_spec, ensure_ascii=False)
        }]
    )
    return json.loads(response.content[0].text)

The router caps max_tokens at 400 and demands JSON-only output. Haiku 4.5 is fast and stable when constrained this way; the system prompt does the work of keeping it from adding chatty preambles. In my pipeline the average routing decision lands at about 1.2 seconds and roughly $0.0006 per call.

Invoking the Worker and Carrying State Forward

When the router returns escalate=true, I call Opus 4.6. The detail that matters more than I expected: feed the router's reason into Opus alongside the task itself, as a two-layer context. Opus then knows why it was called, and the response leans harder into the dimension the router cared about.

def delegate(task_spec: dict, routing: dict) -> str:
    if not routing["escalate"]:
        return run_haiku_inline(task_spec)
    response = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=8000,
        system=(
            "You are a senior writer. The upstream router escalated this "
            f"task to you because: {routing['reason']}\n"
            "Lean into long-form depth, logical chaining, and personal "
            "experience as the reason suggests."
        ),
        messages=[{
            "role": "user",
            "content": build_prompt(task_spec)
        }]
    )
    return response.content[0].text

I didn't include the router's reason in the early version. Opus produced perfectly fine prose, but it would occasionally drift from what the router had flagged, and my downstream quality gate (the article_gate.py referenced elsewhere in our docs) would bounce it back. After I started piping the reason through, the first-pass quality gate acceptance rate moved from 62% to 84%. That's not a vanity metric — every bounced article translates into a regeneration that costs API tokens again.

Streaming vs Blocking — Where Each Belongs

The router returns short structured JSON, so streaming buys almost nothing there. Trimming 1.2s of latency is less valuable than the clean error-handling you get from waiting for the full response and then parsing JSON.

Opus, on the other hand, is a strong fit for streaming. Long-form output (5,000–8,000 tokens) gives the early bytes time to be inspected. I run a lightweight keyword scan on the first few hundred tokens and abort early if obvious template phrasing, banned words, or hype-style headlines slip through.

def stream_with_early_abort(prompt: str, banned: list[str]) -> str:
    buffer = []
    with client.messages.stream(
        model="claude-opus-4-6",
        max_tokens=8000,
        messages=[{"role": "user", "content": prompt}],
    ) as stream:
        for text in stream.text_stream:
            buffer.append(text)
            joined = "".join(buffer)
            for word in banned:
                if word in joined:
                    raise EarlyAbort(word)
            if len(joined) > 600 and looks_like_template_intro(joined):
                raise EarlyAbort("template_intro")
        return "".join(buffer)

Beyond improving the first-pass acceptance rate, the early abort saves real tokens. On average, an aborted run saves about 3,500 output tokens, which translates to 5–8% additional monthly savings on top of the routing savings.

Three Pitfalls I Hit Running the Hybrid

The theory cleans up nicely on a whiteboard. Real operation surfaces stranger problems. Three for the record.

1. The router collapses to "always escalate"

For the first two weeks, the Haiku router escalated nearly everything. The cause was a single line in the system prompt: "when in doubt, escalate=true." Haiku is conscientious; given that nudge, it doubts most things. I changed it to "when in doubt, escalate=false — Haiku can handle this itself," and shifted the rule to a numeric threshold of two-or-more true criteria. The escalation rate dropped from 92% to roughly 30%, and the two-tier shape started to actually function.

2. Cache-key churn destroys latency

I was running prompt caching on the Opus side, but I had been inserting the router's reason into the system prompt. The cache key changed with every request, the hit rate fell to near-zero, and average end-to-end latency degraded from 4.8s to 11.3s.

The fix was to keep the system prompt fixed and pass the router's reason through messages instead, then place an ephemeral cache breakpoint at the end of the system prompt with cache_control. Cache hit rate climbed back to ~78%, and latency settled around 5.2s.

3. Two-layer logging blows up storage

Storing the router's full response next to Opus's full response increased my monthly log volume by 2.4x. Cloudflare R2 storage costs scale modestly, but at that ratio they start to be noticeable.

I now only persist escalate and a truncated reason from the router and keep the Opus output verbatim. When I need the raw router response for debugging, I write it to an ephemeral KV with a short TTL. Log volume returned to prior levels.

Monthly Cost Math — Running 10 Articles a Day

Abstract per-token prices are easier to internalize when grounded in your own workload. My setup runs 4 articles a day across 4 sites, so roughly 480 articles per month.

Average token consumption per article looks like this: router input 1,500, router output 300, Opus input 3,000, Opus output 5,000. Assume 30% escalation:

• Haiku router input: 480 × 1,500 = 720,000 tokens
• Haiku router output: 480 × 300 = 144,000 tokens
• Opus input (30% escalation): 144 × 3,000 = 432,000 tokens
• Opus output (30% escalation): 144 × 5,000 = 720,000 tokens
• Haiku body work (70%): 336 × 4,500 = 1,512,000 tokens

At current reference prices, that lands around $25 a month. The year-prior Opus-only baseline at the same scale was about $90 a month — roughly $65 in monthly delta and $780 annually. Set against Stripe membership revenue (Pro at $5 / Premium at $15), API cost no longer materially compresses my margin.

One subtlety: escalation rate isn't a constant. In November–December, when I run more seasonal pieces, my escalation rate climbs to 45% and monthly cost jumps near $35. Months heavy on troubleshooting content sit around 18% and stay under $20. Tracking this number monthly is the rhythm that makes the routing prompt actually improve over time.

Logging Design — Squeeze a Second Use From Router Output

It feels wasteful to use the router's JSON only for the escalation decision. I now persist it in Cloudflare D1 as a structured log for downstream analysis.

import sqlite3
import datetime
 
def log_routing(task_id, task_spec, routing, escalated):
    conn = sqlite3.connect("routing.db")
    conn.execute(
        """
        INSERT INTO routing_log
        (task_id, ts, topic, output_long, multi_hop, author_voice,
         differentiation, architecture, escalate, reason)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
        """,
        (
            task_id,
            datetime.datetime.utcnow().isoformat(),
            task_spec.get("topic", ""),
            int(routing["output_long"]),
            int(routing["multi_hop"]),
            int(routing["author_voice"]),
            int(routing["differentiation"]),
            int(routing["architecture"]),
            int(escalated),
            routing["reason"][:200],
        ),
    )
    conn.commit()

With this log, end-of-month reviews can ask: which criterion contributes most to escalation? In my data, when author_voice is true, escalation is effectively 100%, while output_long alone only triggers escalation about 32% of the time. Once you can measure that, tuning the router prompt becomes a quantitative exercise rather than a guess.

The multi_hop criterion is the hardest for Haiku to judge consistently. I shore it up by adding a small handful of past article titles that escalated on multi_hop=true directly into the router's system prompt as few-shot examples. Subjectively, the rate of agreement improved by maybe 20%.

Three Approaches I Tried and Discarded

Before I landed on this two-tier shape, I tried a few alternatives. The lessons sit in what I gave up.

The first was "Opus only, with lower temperature." The idea was that shorter outputs save tokens. Dropping temperature to 0.2 did cut output length by about 20%, but it also flattened the prose. The texture readers were paying for went thin. Costs fell slightly, but churn climbed by about 4 points month-over-month and net margins worsened.

The second was "Sonnet only." Cheaper than Opus, smarter than Haiku — a tempting middle. I ran it for about six months. For complex long-form pieces, Sonnet's depth didn't quite match Opus, and quality-gate bounces became more frequent. The cost of regenerating bounced articles eventually exceeded the cost of just using Opus selectively for those pieces.

The third was "isolate the router as a separate Cloudflare Workers app." Routing decisions are stateless, so they ought to run at the edge. Router-only latency did drop to 0.4s. But the Workers CPU/memory ceilings and the Anthropic SDK's dependency tree made the operational overhead worse than the latency win. Today the router happily co-locates with the Python runtime running everything else.

The shared lesson across these failed paths: optimizing only the per-token unit cost narrows your field of vision. Regeneration cost, operational load, and churn rate all need to be included in the same equation. The slightly more elaborate two-tier ended up being cheaper in total terms.

A Three-Tier Variant — Where Sonnet 4.6 Fits

Once a two-tier pipeline is running smoothly, it's worth thinking about a three-tier variant with Sonnet 4.6 in the middle. As my premium catalog grew, I found a band of tasks that didn't quite need Opus but felt under-served by Haiku alone.

The three-tier shape treats output_long as a graded threshold: under ~1,200 tokens stays on Haiku, 1,200–3,500 routes to Sonnet, 3,500+ goes to Opus. Sonnet's ~$3.00 per 1M is a sweet spot for that band, and once it stabilized I saw an additional 10–15% monthly savings on top of the two-tier baseline.

The trade-off is operational complexity: more prompt versions, more quality gates, more dashboards. My recommendation is to live with the two-tier shape for six months until your metrics feel stable, then expand to three.

Metrics to Watch — Keeping the Orchestrator Healthy

"It's working" isn't a metric. I review these on a Grafana board every day:

• Escalation rate (target band: 20–40%)
• Average router decision latency
• First-pass acceptance rate from the quality gate on Opus output
• Prompt cache hit rate on Opus input
• Total cost per article (Haiku + Opus combined)

Escalation rate is the most important health indicator. Below 20% suggests Haiku alone might be sufficient for more tasks than you realized. Above 40% suggests over-reliance on Opus. My monthly cadence is to read those numbers, tweak the router system prompt, and watch the rate drift back into the band.

A Habit of Naming Your Judging Criteria

A short detour from engineering. Since 2019 I've been actively submitting to international art prizes and have received 17 awards along the way. The lesson that surprised me most was that "knowing which prizes to enter, and which to skip" mattered more than the act of winning.

Every open call has different criteria — some weight technical execution, some prize conceptual originality, some are watching for market appeal. When I submitted without reading those criteria, I learned nothing from rejections. Once I started naming the criteria explicitly and selecting where to apply, the pattern of acceptances finally began to compound.

Designing this router felt like the same exercise. Reading a task spec and deciding which model to invoke is the same muscle as reading a juror's brief and deciding what to submit. Writing the criteria down — and consulting them every time — is what splits the next six months into "predictable cost and quality" vs "a constant scramble."

The orchestrator-on-Haiku decision tends to be framed as a cost optimization story. To me, the deeper value is that it builds, into the pipeline itself, a habit of naming your own judging criteria.

Where to Start

If you're thinking of trying this two-tier shape, I'd suggest staging it like this rather than swapping the whole pipeline at once:

1. Pick one stable classification task (tagging, categorization, metadata extraction) and move it to Haiku-only. Confirm quality holds for two weeks.
2. Identify a task with variable output quality (long-form prose, code generation) and put a router in front of it.
3. Tune the router system prompt until the escalation rate settles in the 20–40% band.
4. Add prompt caching to the Opus side. Keep the system prompt static so the cache breakpoint can actually hit.
5. Build a small dashboard for escalation rate, cache hit rate, and per-article cost. Review weekly for the first month, monthly afterward.

I didn't follow this order myself. I jumped in, hit each of the three pitfalls in turn, and learned them the slow way. If you take a calmer entry, you'll spend less time refunding yourself in the form of API regenerations.

The real value of a two-tier setup, I've come to think, isn't the cost cut. It's that splitting judgment from production work surfaces the quality of your judgment as something you can measure separately. Before chasing that next billing alert, try giving the judgment job to Haiku and see how the rest of the pipeline starts to relax. Thank you for reading this far.