A connector failed for two nights and I never noticed — instrumenting my solo setup after observability went to public beta

One morning I opened the generation log and not a single article had shipped.

Tracing it back, one connector call had been failing for two nights straight. The error was swallowed quietly, and no signal ever reached me. The fix itself took five minutes. What stayed with me was that two days of empty runs could not be recovered.

That same week, Claude's connector observability reached public beta. Administrators can now watch adoption, errors, latency, and usage across the product. Reading it, I noticed the obvious: a one-person operation like mine has none of that instrumentation on hand.

Translating enterprise observability into a solo setup

Connector observability is an admin feature for Team and Enterprise. The nightly jobs I run alone do not come with that dashboard.

Even so, the numbers I wanted were the same. Which connector, how often, how fast, and how often failing. With those four visible, the two nights of empty runs would have been caught on the first morning.

As an indie developer running several apps and four sites with jobs firing overnight, I had a habit of checking my AdMob numbers each morning, yet I had never once looked at the health of the connectors carrying those numbers. I was trusting the vessel and inspecting only what it delivered.

So I made a decision. Rather than wait for a management feature, I would slip a single thin meter inside my own jobs. Add only records, and touch the processing itself not at all.

Lay a layer of observation quietly on top of judgment and automation. In the spirit of "just watch," I started by letting the numbers accumulate in silence.

A thin wrapper around connector calls

What I did is simple. I wrap each MCP connector call in one function and drop a single JSONL line capturing start time, duration, outcome, and error class.

// connector-meter.ts — add one meter to connector calls, nothing more
import { appendFile } from "node:fs/promises";
 
type Outcome = {
  ts: string;        // ISO8601 (record time)
  tool: string;      // connector / tool name
  latencyMs: number; // duration
  ok: boolean;       // outcome
  errorClass?: string; // classification on failure
};
 
const LOG = process.env.METER_LOG ?? "./_meter/connector.jsonl";
 
// coarsely classify errors as transient vs structural
function classify(err: unknown): string {
  const m = (err instanceof Error ? err.message : String(err)).toLowerCase();
  if (m.includes("timeout") || m.includes("etimedout")) return "timeout";
  if (m.includes("429") || m.includes("rate")) return "rate_limit";
  if (m.includes("401") || m.includes("403") || m.includes("auth")) return "auth";
  if (m.includes("5")) return "upstream_5xx";
  return "other";
}
 
export async function meter<T>(
  tool: string,
  call: () => Promise<T>,
): Promise<T> {
  const start = Date.now();
  try {
    const result = await call();
    await record({ tool, latencyMs: Date.now() - start, ok: true });
    return result;
  } catch (err) {
    await record({
      tool,
      latencyMs: Date.now() - start,
      ok: false,
      errorClass: classify(err),
    });
    throw err; // record it, but never swallow it
  }
}
 
async function record(o: Omit<Outcome, "ts">): Promise<void> {
  const line = JSON.stringify({ ts: new Date().toISOString(), ...o });
  await appendFile(LOG, line + "\n", "utf8");
}

At the call site, you just wrap the existing connector call in meter().

const issues = await meter("github.list_issues", () =>
  github.listIssues({ repo, state: "open" }),
);

The part I cared about most is keeping throw err. If you swallow the exception for the sake of observation, the meter itself starts hiding failures. Record it, but do not stop the flow. That was the one line I refused to cross.

Refusing to quietly swallow errors is the same stance I wrote about earlier in an input-freshness contract that fails loud. Observation is a way to reduce silence.

A small weekly rollup

I roll up the accumulated JSONL just once a week. A short script that lists, per connector, the number of calls, the failure rate, and the median and p95 latency.

# roll up the last 7 days, grouped by connector
jq -s '
  group_by(.tool)[] |
  {
    tool: .[0].tool,
    calls: length,
    fail_rate: ((map(select(.ok == false)) | length) / length * 100 | floor),
    p50: (map(.latencyMs) | sort | .[(length * 0.50 | floor)]),
    p95: (map(.latencyMs) | sort | .[(length * 0.95 | floor)])
  }
' _meter/connector.jsonl

Looking at the table it produced, things I had only sensed by feel took on a clear shape.

The values above are a one-week snapshot from my own environment. A p95 climb I would have missed by watching averages alone showed up on web.fetch. Usually fast, yet occasionally over two seconds. That "occasionally" is exactly where continuous nightly runs drop things.

The 4% failure rate, once I looked at the breakdown, was mostly timeout. Structural failures like expired auth were zero. In other words, the kind a retry can save — which made the order of what to fix obvious.

One more thing surprised me: the sheer volume. web.fetch ran 1,180 times in a week, far more than I had assumed. Half of my nightly cost and half of my failure surface lived in a connector I rarely thought about. Counting calls, not just failures, quietly reset which connector deserved my attention first.

How the numbers changed the way I set thresholds

Before the meter, I could only describe trouble as "it feels a bit slow." Now I can set thresholds as numbers.

In my setup, per connector, I roughly treat a sustained p95 above 1,500ms as a yellow flag and a weekly failure rate above 5% as a red one. These are not precise optima. But drawing a single line meant that the next morning I could see at a glance where to look.

Splitting error classes helped too. timeout and rate_limit often clear on their own if you wait, while auth needs a person to act right away. The same "failure" is handled entirely differently. Adding one column of classification made the morning judgment much lighter.

This meter gets a step stronger when paired with a way to detect silent skips. Whether the job ran at all belongs to a deadman switch for scheduled jobs; whether what ran was any good belongs to this meter. With the two side by side, I finally felt I could hand the night over.

What I want to add next

What I am working on now is turning the weekly rollup into a light daily check. Roll up only the previous day, and if any connector crossed a threshold, send a single line of notice. No grand monitoring stack required.

If you want to go further into instrumentation, there is the path of leaning on a standard like OpenTelemetry, which I laid out in making AI calls observable with OpenTelemetry. For a solo operation, though, starting from a single JSONL file feels about right.

Enterprise observability will not arrive at my desk. Even so, simply accumulating the four numbers I wanted to see changed how at ease I feel about running automation overnight. A meter is not a flashy feature. But it was a solid move toward noticing, once again, a connector that had quietly gone down.

If something on your side has been failing in the dark, may this nudge you to give it a meter of its own. Thank you for reading along.