Rewiring Claude API Observability with OpenTelemetry GenAI Conventions — A Design Memo for Model Migrations and Cost Audits

I keep metrics in two layers: histograms plus a per-tenant counter

Span attributes alone make "what was tenant A's token usage this month?" a full trace scan at month-end. I run four metrics in parallel.

// src/lib/otel-metrics.ts
import { metrics } from "@opentelemetry/api";
 
const meter = metrics.getMeter("dolice.claude-api", "1.0.0");
 
export const claudeLatency = meter.createHistogram(
  "gen_ai.client.operation.duration",
  {
    description: "End-to-end latency of Claude API calls",
    unit: "s",
    advice: {
      explicitBucketBoundaries: [
        0.1, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64,
      ],
    },
  },
);
 
export const claudeTokens = meter.createHistogram(
  "gen_ai.client.token.usage",
  {
    description: "Token usage per request",
    unit: "{token}",
    advice: {
      explicitBucketBoundaries: [
        16, 64, 256, 1024, 4096, 16384, 65536, 200000,
      ],
    },
  },
);
 
export const claudeCostJpy = meter.createCounter("dolice.claude.cost_jpy", {
  description: "Estimated JPY cost per call (rolled up from token counts)",
});
 
export const claudeErrors = meter.createCounter(
  "gen_ai.client.error.count",
  { description: "Error count grouped by error type" },
);

The write side is rolled up at the end of tracedMessage from the previous section. I hold the unit pricing (Sonnet 4.6 as of 2026-05: $3 / 1M input tokens, $15 / 1M output, FX 150 JPY ≈ ¥450 / ¥2,250) in environment variables and make it hot-reloadable, so I can chase monthly invoice drift the moment it starts.

function recordUsage(model: string, res: Anthropic.Message, latencyMs: number) {
  const attrs = {
    "gen_ai.system": "anthropic",
    "gen_ai.request.model": model,
    "gen_ai.response.model": res.model,
    "dolice.tenant_id": currentTenantId(),
    "dolice.surface": currentSurface(),
  };
  claudeLatency.record(latencyMs / 1000, attrs);
  claudeTokens.record(res.usage.input_tokens, {
    ...attrs,
    "gen_ai.token.type": "input",
  });
  claudeTokens.record(res.usage.output_tokens, {
    ...attrs,
    "gen_ai.token.type": "output",
  });
  const jpy =
    res.usage.input_tokens * PRICING[model].inputJpy +
    res.usage.output_tokens * PRICING[model].outputJpy +
    (res.usage.cache_creation_input_tokens ?? 0) *
      PRICING[model].cacheCreationJpy +
    (res.usage.cache_read_input_tokens ?? 0) *
      PRICING[model].cacheReadJpy;
  claudeCostJpy.add(jpy, attrs);
}

The histogram buckets follow a 0.1–64 second exponential ladder because Sonnet 4.6 with 200K context and no caching averages 18 seconds, with the slow tail near 50 seconds. The default ms buckets are completely unsuitable for Claude calls.

I make the SLI for cost tracking "within 5% of the Anthropic billing page"

You discover how serious your observability really is by comparing the monthly Anthropic billing page against what your OTel metrics compute. I've held the gap to ±5% for four months running, but the first month was off by 23.4%. Causes and fixes:

-- ClickHouse / TimescaleDB compatible: per-tenant daily cost rollup
SELECT
  toDate(timestamp) AS day,
  attributes['dolice.tenant_id'] AS tenant,
  attributes['gen_ai.response.model'] AS model,
  sum(value) AS jpy_total
FROM otel_metric_sum
WHERE name = 'dolice.claude.cost_jpy'
  AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY day, tenant, model
ORDER BY day DESC, jpy_total DESC;

Three things were generating drift.

1. I conflated response.model with request.model. A request for claude-sonnet-4-6 can come back tagged with a snapshot like claude-sonnet-4-6-20251015, my pricing key wouldn't match, and I'd fall back to a default unit price. The fix was a normaliser that does prefix matching on response.model.
2. I had cache token unit prices set to zero. Anthropic's prompt cache bills at 1.25x normal for cache creation and 0.1x for cache reads. If you forget to feed cache_*_tokens into your counter, workloads that lean on caching come in roughly 12% under-reported.
3. Failed-request accounting. 429s aren't billed, but parts of the 5xx family (especially partial generations) bill for the output tokens that did stream. I changed the rule to: if the error path still carries a usage field, record it.

After those fixes, April measured drift was ¥1,247 / ¥31,890 = 3.91%, and May landed at 2.18%. Putting "monthly drift ≤ 5%" as an SLO on Grafana means I notice immediately when an SDK upgrade changes the snapshot name format.

How to design replay-ability for model swaps

The moment observability pays for itself, in my indie shop, is when Anthropic updates a model or I'm weighing the move from claude-sonnet-4-6 to claude-opus-4-6. Streaming a "shadow" of production traffic at a new model to detect user-visible regressions is the safety net.

# scripts/replay_shadow_traffic.py
"""Replay 24h of production requests against an alternate model and compare quality.
 
Usage:
  python replay_shadow_traffic.py \
      --from claude-sonnet-4-6 \
      --to claude-opus-4-6 \
      --hours 24 \
      --sample 0.02
"""
import argparse
import asyncio
import json
import os
import statistics
from datetime import datetime, timedelta
from anthropic import AsyncAnthropic
import clickhouse_connect
 
client = AsyncAnthropic()
 
async def replay_one(row, target_model: str) -> dict:
    payload = json.loads(row["request_payload"])
    started = datetime.now()
    res = await client.messages.create(
        model=target_model,
        system=payload.get("system"),
        messages=payload["messages"],
        max_tokens=payload["max_tokens"],
    )
    elapsed = (datetime.now() - started).total_seconds()
    return {
        "original_id": row["response_id"],
        "original_model": row["model"],
        "shadow_model": target_model,
        "original_output_tokens": row["output_tokens"],
        "shadow_output_tokens": res.usage.output_tokens,
        "shadow_latency": elapsed,
        "shadow_text": res.content[0].text[:2000],
    }
 
async def main(args):
    ch = clickhouse_connect.get_client(host=os.environ["CH_HOST"])
    since = datetime.now() - timedelta(hours=args.hours)
    rows = ch.query(
        """
        SELECT response_id, model, request_payload, output_tokens
        FROM claude_request_replay
        WHERE timestamp >= %(since)s
          AND model = %(from_model)s
          AND rand() / pow(2, 32) < %(sample)s
        """,
        parameters={
            "since": since,
            "from_model": args.from_model,
            "sample": args.sample,
        },
    ).named_results()
 
    sem = asyncio.Semaphore(4)  # cap at 4 concurrent
    async def bounded(row):
        async with sem:
            try:
                return await replay_one(row, args.to)
            except Exception as e:
                return {"error": str(e), "original_id": row["response_id"]}
    results = await asyncio.gather(*(bounded(r) for r in rows))
 
    ok = [r for r in results if "error" not in r]
    latencies = [r["shadow_latency"] for r in ok]
    ratio_tokens = [
        r["shadow_output_tokens"] / max(1, r["original_output_tokens"])
        for r in ok
    ]
    print(f"Replayed: {len(ok)} / {len(results)} (errors {len(results) - len(ok)})")
    print(f"Shadow latency p50/p95: {statistics.median(latencies):.2f}s / "
          f"{statistics.quantiles(latencies, n=20)[-1]:.2f}s")
    print(f"Output token ratio (shadow / original) median: "
          f"{statistics.median(ratio_tokens):.2f}")
    ch.insert("claude_shadow_results", ok)
 
if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--from", dest="from_model", required=True)
    p.add_argument("--to", required=True)
    p.add_argument("--hours", type=int, default=24)
    p.add_argument("--sample", type=float, default=0.02)
    asyncio.run(main(p.parse_args()))

The trick is that during the production call I write a single row to a claude_request_replay table containing the original request payload. To keep cache spend down, the production path runs with caching ON and the shadow replays with caching OFF. I compare on three axes — "ratio of output token counts (extreme shrink or growth signals something broken)", "p95 latency", and "the first 2,000 characters of output text retained for manual spot checking".

When I actually ran 24h / 2% sampled shadow from Sonnet 4.6 to Opus 4.6, the result was:

• 1,489 replays, 6 errors (0.40%)
• p95 latency: 8.4s → 18.7s (roughly 2.2x)
• Median output-token ratio: 1.18 (Opus is slightly more verbose)
• Estimated cost delta: ¥3,420 / 24h → ¥102,600 / month

Looking at those numbers, I decided that for the everyday chat surface in the manifestation app, Opus didn't earn its keep, and I now route only the deeper paid "consultation" mode at Opus. Before I had this telemetry in place, I made those calls on "feel" — and then turned pale when the billing page arrived. That happened more than once.

Four production stumbles I want to leave a note on

The system isn't done when the wiring is in. Six months of running it, four things bit me.

1. @opentelemetry/sdk-node does not run on Cloudflare Workers

Anything that assumes the Node runtime won't even import inside Workers. I kept only @opentelemetry/api as the shared interface, and swapped the exporter on the Workers side for a small hand-rolled wrapper that posts OTLP/HTTP over fetch. With that, the same instrumentation code now compiles in both the Next.js server action world and the Workers world.

2. Whether to fold 429 retries into the parent span or split them

I initially bundled exponential-backoff retries into a single span, but Datadog APM then flagged those as latency outliers and dragged the SLI down. I switched to a child span per retry, with only an retry.attempts counter on the parent.

3. Streaming responses surface usage only on the last chunk

Over Server-Sent Events, token counts only become final at the last message_delta. To survive mid-stream exceptions, I do a two-stage write: at message_start I attach input_tokens to the span, and at message_stop I append output_tokens. That single change eliminated the "input tokens for half-dead requests vanished from billing" failure mode.

4. Tenant ID propagation needs AsyncLocalStorage to stay consistent

Server actions, cron, webhooks — once you have parallel instrumentation paths, the tenant ID falls off the span attributes. On Node I leaned on AsyncLocalStorage; on Workers I stash it on executionCtx. A bridge like c.executionCtx.props.tenantId is what keeps the two consistent.

My current recommendation

The OSS OTel instrumentation libraries downstream of Anthropic (@arizeai/openinference-instrumentation-anthropic and friends) are well built, but for a small indie operation like mine that mixes multiple vendors, a single thin in-house wrapper across vendors has been the easier long-term call. As long as you align to the GenAI conventions, the attribute names mostly carry over when you later bolt on a dedicated stack like OpenInference or Langfuse.

What I'd push you to put in place during the first month is just three things: span attributes, a cost counter, and a monthly-drift SLI. Shadow traffic and replay can wait until the service has matured to the point where the bill is visible. I didn't set up the shadow pipeline until month five of running production myself, and was just chasing monthly drift anomalies in the meantime.

The next move I'd find interesting is to have Claude itself grade the shadow replay outputs on a 1–5 quality scale to automate a small eval harness. Once your sample is past 100, comparing Sonnet and Opus by hand stops being practical.

I hope this memo helps when you reach the "rebuild the dashboard" phase of running Claude API in production. I'm still tuning mine as I run it, so if you've been through the wringer on observability and have notes of your own, my Linktree is the way to reach me. Thanks for reading to the end.