Observability for Claude Code with OpenTelemetry — A Production-Grade Tracing Guide for Agentic Workflows

Bootstrap OpenTelemetry with the smallest possible footprint

Start with the wrapper that launches Claude Code. Here's a minimal Node.js initializer.

// otel-init.mjs — load this first, before anything else
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";
 
const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "claude-code-agent",
    [SemanticResourceAttributes.SERVICE_VERSION]: process.env.AGENT_VERSION ?? "dev",
    "deployment.environment": process.env.NODE_ENV ?? "development",
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318/v1/traces",
    headers: process.env.OTEL_AUTH_HEADER
      ? { Authorization: process.env.OTEL_AUTH_HEADER }
      : undefined,
  }),
});
 
try {
  sdk.start();
  console.error("[otel] SDK started");
} catch (err) {
  // Telemetry failures must never take production down — fail open.
  console.error("[otel] init failed, continuing without tracing:", err);
}
 
process.on("SIGTERM", async () => {
  try {
    await sdk.shutdown();
  } catch (err) {
    console.error("[otel] shutdown error:", err);
  }
});

The try/catch around sdk.start() is doing real work. The principle is fail open for telemetry: if Honeycomb is down or your collector is unreachable, the agent should keep running. Skip this and you've turned your observability vendor's maintenance window into a production outage of your own.

Run this with OTEL_EXPORTER_OTLP_ENDPOINT set to your collector and you should see [otel] SDK started on stderr. Without the env var, the agent should still run normally; only telemetry will be missing.

Emit a root span from the SessionStart hook

Now wire Claude Code's events into the SDK. The simplest pattern is to start a root span on SessionStart and end it on SessionEnd. If hook semantics are unfamiliar, the Claude Code Hooks complete practical guide covers all eight hook types in depth — read that first if needed.

// hooks/session-start.mjs
import { trace, context } from "@opentelemetry/api";
import { writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
 
import "./otel-init.mjs"; // initialize the SDK before anything else
 
const tracer = trace.getTracer("claude-code-session");
 
const span = tracer.startSpan("claude_code.session", {
  attributes: {
    "session.id": process.env.CLAUDE_SESSION_ID ?? "unknown",
    "session.cwd": process.cwd(),
    "session.user": process.env.USER ?? "unknown",
  },
});
 
// Persist the span context so subsequent hooks can attach as children.
const spanCtx = span.spanContext();
const handoff = {
  traceId: spanCtx.traceId,
  spanId: spanCtx.spanId,
  startedAt: Date.now(),
};
 
const stateFile = join(tmpdir(), `claude-otel-${process.env.CLAUDE_SESSION_ID}.json`);
writeFileSync(stateFile, JSON.stringify(handoff));
console.error(`[otel] root span started: ${spanCtx.traceId}`);
 
// Don't end the span here — SessionEnd will close it.

Claude Code hooks run as separate processes. You can't pass a traceparent header in memory the way you would inside a single HTTP service, so a small state file in /tmp/claude-otel-{sessionId}.json is the practical handoff mechanism for the trace ID and span ID.

The SessionEnd hook then reads this file back and closes the span. We'll cover the cross-process gotchas in the pitfalls section near the end.

Break down each tool call into a child span

The real value lands when you split each tool call into its own span using PreToolUse and PostToolUse.

// hooks/pre-tool-use.mjs — fired before a tool runs
import { trace, context } from "@opentelemetry/api";
import { readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import "./otel-init.mjs";
 
const tracer = trace.getTracer("claude-code-tool");
 
const stateFile = join(tmpdir(), `claude-otel-${process.env.CLAUDE_SESSION_ID}.json`);
const state = JSON.parse(readFileSync(stateFile, "utf8"));
 
// Reconstruct the parent context from the persisted IDs.
const parentCtx = trace.setSpanContext(context.active(), {
  traceId: state.traceId,
  spanId: state.spanId,
  traceFlags: 1,
  isRemote: true,
});
 
const toolName = process.env.CLAUDE_TOOL_NAME ?? "unknown";
const toolInput = process.env.CLAUDE_TOOL_INPUT_PREVIEW ?? "";
 
const span = tracer.startSpan(
  `tool.${toolName}`,
  {
    attributes: {
      "tool.name": toolName,
      "tool.input.length": toolInput.length,
      // Don't ship the full prompt — store length and a short preview only.
      "tool.input.preview": toolInput.slice(0, 200),
    },
  },
  parentCtx
);
 
const toolStateFile = join(
  tmpdir(),
  `claude-otel-tool-${process.env.CLAUDE_TOOL_INVOCATION_ID}.json`
);
writeFileSync(
  toolStateFile,
  JSON.stringify({
    traceId: span.spanContext().traceId,
    spanId: span.spanContext().spanId,
    startedAt: Date.now(),
  })
);

// hooks/post-tool-use.mjs — fired after a tool completes
import { trace, context, SpanStatusCode } from "@opentelemetry/api";
import { readFileSync, unlinkSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import "./otel-init.mjs";
 
const stateFile = join(
  tmpdir(),
  `claude-otel-tool-${process.env.CLAUDE_TOOL_INVOCATION_ID}.json`
);
const state = JSON.parse(readFileSync(stateFile, "utf8"));
 
const tracer = trace.getTracer("claude-code-tool");
const span = tracer.startSpan(`tool.end.${process.env.CLAUDE_TOOL_NAME}`, {
  startTime: state.startedAt,
  links: [
    {
      context: {
        traceId: state.traceId,
        spanId: state.spanId,
        traceFlags: 1,
        isRemote: true,
      },
    },
  ],
});
 
const exitCode = Number(process.env.CLAUDE_TOOL_EXIT_CODE ?? "0");
const duration = Date.now() - state.startedAt;
 
span.setAttributes({
  "tool.exit_code": exitCode,
  "tool.duration_ms": duration,
});
 
if (exitCode !== 0) {
  span.setStatus({ code: SpanStatusCode.ERROR, message: `exit ${exitCode}` });
}
 
span.end();
unlinkSync(stateFile);

The thing worth flagging here is the deliberate refusal to log full tool input as a span attribute. Prompts contain PII, code snippets, and sometimes credentials, and shipping all of that to a third-party observability vendor creates compliance headaches that are easier to avoid than to clean up. I cap the preview at 200 characters and store the length separately. If I genuinely need the full prompt for an investigation, I write it to a separate, access-controlled KV with a hash that I can correlate back to the trace.

Bridge the trace into your MCP servers

When Claude Code calls an MCP server, the MCP protocol (as of April 2026) doesn't standardize traceparent propagation. The practical workaround is to wrap your MCP client and inject W3C Trace Context yourself. The companion piece Claude Code MCP and the 500K context limit covers related MCP constraints worth being aware of.

// mcp-tracing-proxy.mjs — inject traceparent into MCP requests
import { trace, context, propagation } from "@opentelemetry/api";
 
export async function callMcpWithTrace(serverUrl, method, params) {
  const tracer = trace.getTracer("mcp-client");
  const span = tracer.startSpan(`mcp.${method}`, {
    attributes: {
      "mcp.server.url": serverUrl,
      "mcp.method": method,
      "mcp.params.keys": Object.keys(params ?? {}).join(","),
    },
  });
 
  const headers = { "content-type": "application/json" };
  // Inject W3C Trace Context (traceparent) into outbound headers.
  propagation.inject(trace.setSpan(context.active(), span), headers);
 
  try {
    const res = await fetch(serverUrl, {
      method: "POST",
      headers,
      body: JSON.stringify({ jsonrpc: "2.0", method, params, id: 1 }),
    });
    if (!res.ok) {
      throw new Error(`MCP HTTP ${res.status}`);
    }
    const json = await res.json();
    span.setAttribute("mcp.response.has_error", Boolean(json.error));
    return json;
  } catch (err) {
    span.recordException(err);
    span.setStatus({ code: 2 /* ERROR */, message: String(err) });
    throw err;
  } finally {
    span.end();
  }
}

If the MCP server runs OpenTelemetry too, it'll see the incoming traceparent and continue the same trace. Once that bridge is live, you can watch a single trace stretch from Claude Code → MCP server → downstream database in one view. In real production work, this single change cut my mean time to root-cause by something like 5x.

Attach token, model, and cache attributes for cost analysis

Cost attribution becomes possible the moment you start tagging spans with token counts and model identifiers.

// In hooks/post-tool-use.mjs, alongside the duration/exit code attributes:
span.setAttributes({
  "claude.model": process.env.CLAUDE_MODEL ?? "unknown",
  "claude.tokens.input": Number(process.env.CLAUDE_INPUT_TOKENS ?? 0),
  "claude.tokens.output": Number(process.env.CLAUDE_OUTPUT_TOKENS ?? 0),
  "claude.cache.read_tokens": Number(process.env.CLAUDE_CACHE_READ_TOKENS ?? 0),
  "claude.cache.creation_tokens": Number(process.env.CLAUDE_CACHE_CREATION_TOKENS ?? 0),
  // Compute cost downstream — pricing tables move; raw tokens don't.
});

I deliberately don't store the dollar cost on the span itself. Pricing tables shift with model updates and promotional periods, and span attributes are immutable once written. Recording raw token counts and computing cost in the dashboard layer means a price change is one query update away, not a re-instrumentation project. For background on Claude Code cost mechanics, the Claude Code cost visualization and budget guide is a useful primer.

Sampling: surviving thousands of runs per day

The first scaling pain you'll feel isn't latency — it's the storage bill. 5,000 runs per day times 50 spans per run is 250k spans per day. Ship all of that raw and the Honeycomb or Datadog invoice will surprise you in a bad way.

The combination that works in practice has three layers.

• Probabilistic head sampling as a baseline: OTEL_TRACES_SAMPLER=traceidratio with OTEL_TRACES_SAMPLER_ARG=0.1 keeps 10% of traces. This is the floor for noisy projects
• Always retain errors: Combine ParentBased with AlwaysOn so that any trace marked as errored gets through regardless of the head sampler. Tail-Based Sampling at the collector is even better if you can run one
• Force-keep important sessions: Tag specific traces (interactive dev sessions, CI release validations) and keep them unconditionally

Tail-Based sampling can't be done in the Node.js SDK alone, so the realistic deployment ends with an OpenTelemetry Collector hop in the middle.

# otel-collector-config.yaml — Tail-Based sampling rules
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 5000 }
      - name: probabilistic
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }
exporters:
  otlphttp:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlphttp]

Any trace that hits one of the three policies — error, latency over 5 seconds, or the random 10% — is forwarded. Storage drops by 80–90% while the traces you actually care about survive.

Five signals worth a daily look

Once data starts accumulating, narrow the daily dashboard to these five tiles. Anything more and people stop reading it.

• Session completion rate (success / failure / timeout): the headline number for agent health
• P95 latency by tool: which tools tend to stall. Bash, WebFetch, and Edit deserve separate breakdowns
• MCP error rate by server: catches a flaky MCP server before users do
• Token consumption by model: shows where model switching or caching could save money
• Retry distribution: the share of tool calls retried 3+ times is a leading indicator of wasted spend

I keep these five tiles open in Grafana every morning for five minutes. If something looks off, the dashboard links straight into the trace, and investigation time drops dramatically.

Add metrics for trend visibility, not detective work

Traces are excellent for debugging a single problematic run. They are terrible for answering "is the system healthy this week?" Metrics fill that gap. Wire a MeterProvider alongside the tracer so dashboards can render rolled-up time series without scanning every span.

// otel-metrics.mjs — call from otel-init.mjs after sdk.start()
import { metrics } from "@opentelemetry/api";
import { MeterProvider, PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
import { Resource } from "@opentelemetry/resources";
 
const meterProvider = new MeterProvider({
  resource: new Resource({ "service.name": "claude-code-agent" }),
  readers: [
    new PeriodicExportingMetricReader({
      exporter: new OTLPMetricExporter({
        url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.replace("/v1/traces", "/v1/metrics"),
      }),
      exportIntervalMillis: 30_000,
    }),
  ],
});
 
metrics.setGlobalMeterProvider(meterProvider);
 
const meter = metrics.getMeter("claude-code-agent");
 
export const toolDurationHistogram = meter.createHistogram("claude.tool.duration", {
  unit: "ms",
  description: "Duration of a single Claude Code tool invocation",
});
 
export const tokensCounter = meter.createCounter("claude.tokens.total", {
  unit: "1",
  description: "Total tokens consumed by Claude Code, counted by type",
});
 
export const sessionFailureCounter = meter.createCounter("claude.session.failures", {
  description: "Count of agent sessions that ended with a non-zero status",
});

Update the PostToolUse hook to record into these metrics in addition to closing the span.

// inside hooks/post-tool-use.mjs
import { toolDurationHistogram, tokensCounter } from "../otel-metrics.mjs";
 
toolDurationHistogram.record(duration, {
  "tool.name": process.env.CLAUDE_TOOL_NAME ?? "unknown",
  "tool.exit_status": exitCode === 0 ? "ok" : "error",
});
 
tokensCounter.add(Number(process.env.CLAUDE_INPUT_TOKENS ?? 0), {
  "claude.model": process.env.CLAUDE_MODEL ?? "unknown",
  "token.type": "input",
});
 
tokensCounter.add(Number(process.env.CLAUDE_OUTPUT_TOKENS ?? 0), {
  "claude.model": process.env.CLAUDE_MODEL ?? "unknown",
  "token.type": "output",
});

The advantage of metrics over span-scanning is cost. A histogram with 30-second resolution costs almost nothing to store and query for years, while the trace data behind it can be retained for only a few weeks. The split lets you ask "how has p95 latency drifted over the last quarter?" without paying for cold storage on millions of traces.

A Python alternative for projects that don't speak Node

Many Claude Code workflows are wrapped in Python — agent orchestrators, CI scripts, data pipelines. The OpenTelemetry concepts are identical; only the SDK shape changes.

# otel_init.py — import this at the top of any Python wrapper
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
import os
 
resource = Resource.create({
    "service.name": "claude-code-agent",
    "service.version": os.environ.get("AGENT_VERSION", "dev"),
    "deployment.environment": os.environ.get("ENV", "development"),
})
 
provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(
    endpoint=os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4318/v1/traces"),
    headers={"authorization": os.environ["OTEL_AUTH_HEADER"]} if "OTEL_AUTH_HEADER" in os.environ else None,
)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
 
tracer = trace.get_tracer("claude-code-agent")

The BatchSpanProcessor requires explicit shutdown to flush queued spans. In hook scripts that exit quickly, register a try/finally block:

# hooks/post_tool_use.py
import os, json, time
from otel_init import tracer, provider
from opentelemetry.trace import Status, StatusCode
 
state_file = f"/tmp/claude-otel-tool-{os.environ['CLAUDE_TOOL_INVOCATION_ID']}.json"
with open(state_file) as f:
    state = json.load(f)
 
duration_ms = int((time.time() * 1000) - state["startedAt"])
exit_code = int(os.environ.get("CLAUDE_TOOL_EXIT_CODE", "0"))
 
with tracer.start_as_current_span(f"tool.end.{os.environ['CLAUDE_TOOL_NAME']}") as span:
    span.set_attribute("tool.duration_ms", duration_ms)
    span.set_attribute("tool.exit_code", exit_code)
    if exit_code != 0:
        span.set_status(Status(StatusCode.ERROR, f"exit {exit_code}"))
 
# Block until the span has been exported (bounded by 1.5s).
try:
    provider.force_flush(timeout_millis=1500)
finally:
    provider.shutdown()

The Python ergonomics are slightly nicer thanks to start_as_current_span as a context manager. If your team is split between languages, remember that the wire format is identical, so a single collector receives both equally.

Redact prompts properly when you do need them

There are cases where the 200-character preview isn't enough — debugging a strange retry loop usually requires the actual prompt. The wrong move is to drop the full prompt onto the span and ship it to your vendor. The right move is to redact at the source and write the redacted form to a separate, access-controlled store.

// redact.mjs — minimal redaction with a hook for project-specific rules
const PATTERNS = [
  // Common credential shapes
  { rx: /sk-[A-Za-z0-9]{16,}/g, mask: "[REDACTED:api_key]" },
  { rx: /Bearer\s+[A-Za-z0-9._-]+/gi, mask: "[REDACTED:bearer]" },
  // Email addresses
  { rx: /[\w.+-]+@[\w-]+\.[\w.-]+/g, mask: "[REDACTED:email]" },
  // 16-digit numbers (rough credit card heuristic)
  { rx: /\d{16}/g, mask: "[REDACTED:card]" },
];
 
export function redact(input) {
  let out = input;
  for (const { rx, mask } of PATTERNS) {
    out = out.replace(rx, mask);
  }
  return out;
}

Apply this before writing anything to a span attribute or to your prompt-archive store. Pair it with a strict allowlist of which projects are even permitted to archive prompts; default-deny is safer than default-allow for anything PII-adjacent.

What dashboards look like on the major backends

The five-signal dashboard from earlier translates to slightly different query shapes depending on where you ship traces.

In Honeycomb, each signal becomes a saved query: filter service.name = claude-code-agent, group by tool.name, plot P95(duration_ms) for the latency tile. Honeycomb's BubbleUp and trace inspection are particularly good for "why is this one trace 30 seconds long" investigations.

In Grafana Tempo + Prometheus, the metrics we emitted earlier feed Prometheus directly:

# Tool p95 latency over 5m, faceted by tool name
histogram_quantile(0.95, sum(rate(claude_tool_duration_bucket[5m])) by (tool_name, le))
 
# Session failure rate over the last hour
sum(rate(claude_session_failures_total[1h]))
  / sum(rate(claude_session_total[1h]))
 
# Token spend by model in the last 24h
sum by (claude_model) (increase(claude_tokens_total[24h]))

Tempo handles the trace storage and you click from a Prometheus alert into the corresponding trace via TraceQL.

In Datadog, you'll spend more time configuring tags than queries — Datadog's APM does most of the work once service, env, and tool.name are tagged consistently. The cost model is per-host plus per-span, so the Tail-Based Sampling rules from earlier are particularly important here.

The pragmatic choice is: pick whichever backend your team already pays for. The instrumentation we built is identical regardless.

Roll out in stages, not in one big bang

Lastly, a deployment cadence I've found works for teams new to agent observability:

1. Week 1: Local Jaeger, instrumentation only on a single dev agent. Confirm traces render and span attributes look right
2. Week 2: Ship to one production environment with head sampling at 100%, no Tail-Based logic. Watch the volume
3. Week 3: Add the Tail-Based collector, dial head sampling down to 10%, validate that error traces still arrive
4. Week 4: Wire alerts and dashboards. Pull at least one historical incident from the trace store and verify you could have caught it with these alerts in place

Skipping straight to "everything on, everywhere" usually means quietly broken sampling that nobody notices until the storage bill arrives.

Pitfalls worth knowing in advance

Roughly in the order I tripped over them:

• Hooks exit before exporters flush: Hook scripts are short-lived child processes. The default BatchSpanProcessor may exit before traces are sent. Either use SimpleSpanProcessor or call await sdk.shutdown() from a beforeExit handler
• Trace ID file collisions: Persisting trace IDs to /tmp means parallel sessions can race. Always include CLAUDE_SESSION_ID in the filename
• Attribute size limits: Don't try to stuff a full prompt into a single span attribute. The OTel spec safe ceiling is around 12KB per attribute, and exporters may reject larger ones
• Don't proxy stdio MCP servers for trace injection: Trying to splice a tracing proxy in front of a stdio-based MCP server breaks JSON-RPC framing. Inject traceparent only for HTTP/SSE-based MCP servers
• Telemetry latency leaking into your hot path: Synchronous exporters tie hook completion to your collector's response time. Use BatchSpanProcessor and bound the flush time with forceFlush({ timeout: 1500 }) before each hook exits

When traces don't show up: a debug runbook

The most demoralizing failure mode is wiring everything up and seeing nothing in the backend. The order I check, in roughly increasing painfulness:

1. Is the SDK actually starting? Check stderr for the [otel] SDK started line. If it's missing, the wrapper script isn't loading otel-init.mjs early enough — typically a side-effect of dynamic import ordering. Move the import to the top of the file
2. Is the endpoint reachable? curl -X POST $OTEL_EXPORTER_OTLP_ENDPOINT -d '{}' from the same machine. A 4xx is fine; it means the endpoint accepts traffic. Connection refused or DNS errors mean the agent will silently buffer spans and drop them
3. Are spans flushing before exit? Add console.error("[otel] forcing flush") and await sdk.shutdown() at the very end of each hook. If the line prints but no spans land, the issue is on the collector side
4. Is sampling dropping the traces you're looking at? During debug, set OTEL_TRACES_SAMPLER=always_on. If traces show up with always-on but not with the production sampler, your sampler config is wrong
5. Is the collector swallowing them? Check the collector's own logs. A misconfigured tail_sampling policy can drop everything silently. Run the collector with --config-source=file:./otel-collector.yaml and inspect startup output for warnings

If steps 1-5 pass and you still see nothing, swap to a local Jaeger via docker run jaegertracing/all-in-one and point the SDK there. Local Jaeger renders traces immediately, so you can isolate whether the problem is the SDK or your hosted backend.

Observability changes how you choose models

As an indie developer footing my own Claude bill, one side effect surprised me once I started actually looking at trace data: model choice becomes data-driven instead of dogmatic. Before instrumentation, I'd default to the most capable model "to be safe." After instrumentation, the histograms told me that maybe 70% of my tool calls were trivial enough that a faster, cheaper model finished them in half the latency for one-fifth the cost.

This is the kind of decision that's almost impossible to make from logs alone. You need to see the distribution of tool durations split by model, and the error rates split the same way. With OpenTelemetry attributes attached, this becomes a one-line query in any backend:

# p95 latency by model — the comparison that drives model selection
histogram_quantile(0.95,
  sum(rate(claude_tool_duration_bucket[7d])) by (claude_model, le)
)

I now run that query weekly. It's how I caught a regression where one of my agents had silently been pinned to a heavyweight model for tasks that didn't need it. The cost overrun was about 40% of the project's monthly Claude bill, and it had been going on for three weeks. With observability, I caught it the next Monday.

Closing thought

If you take one thing from this guide, let it be that observability for agentic systems is not optional infrastructure. Traditional services can survive on logs because the request/response shape is simple. Agents fan out, retry, and self-correct in ways that turn even simple tasks into multi-second multi-hop journeys. Without traces, you're guessing.

Start small — local Jaeger, two hooks, one model — and trust the process. Once that first trace renders end-to-end, the value becomes self-evident. From there, sampling, dashboards, and metrics layer on naturally as your agent footprint grows.

Pre-deploy checklist

Before rolling this out anywhere with real traffic:

• SDK initialization is wrapped in try/catch; telemetry failure cannot take down the agent
• All four hooks (SessionStart / PreToolUse / PostToolUse / SessionEnd) emit and close spans consistently
• traceparent injection is restricted to HTTP/SSE MCP servers — never stdio
• No span attribute carries full prompts or PII; preview is capped at 200 characters
• Tail-Based sampling captures all errors, all slow traces, and a 10% probabilistic baseline
• Daily span volume forecast fits inside your observability vendor's plan
• Dashboard surfaces the five core signals; alerts fire on session failure rate > 5%, p95 latency > 30s, MCP error rate > 10%

Where to start if this feels like a lot

Don't try to ship everything at once. The honest pacing I'd recommend is one day on SDK init, one day on hook integration, two to three days on sampling tuning, in that order. For a weekend experiment, set up SessionStart plus PreToolUse/PostToolUse only and pipe spans into a local Jaeger container. The moment the trace lights up, the agent stops being a black box.

The deeper sampling and Tail-Based logic can wait until you can see your real data volume. Don't over-engineer day one — the goal is "small and visible," not "perfect and theoretical."

The thing that hits home once you operate agents at scale is that observability isn't a nice-to-have. It's the difference between knowing what your system is doing and hoping. Start now and your operational load three months from now will look entirely different.