⬡ API & SDK/2026-07-08Advanced

Contract-Test Every Tool Before You Submit or Automate an MCP Connector

A connector that works once in a chat can still break silently in an unattended job through misread response shapes or double-fired writes. Here is a small harness that machine-checks tool descriptions, response contracts, idempotency, and latency, with measured numbers.

MCP⁵⁰ Claude API¹¹⁷ Contract Testing² Unattended Runs TypeScript²⁴

✦ Premium Article

It was the night I wired a connector that had worked once in a chat straight into an unattended job. The next morning, the log showed the same article write running twice. The tool logic was correct, the permissions were fine, and it had never reproduced when I called it interactively.

The cause sat in front of the connector, not inside it. When the network stalled for a moment and the client retried, the write tool had no idempotency, so the same operation applied twice. Interactively I would have seen the result and caught it. An unattended job has nobody watching. With July's update, you can now submit a connector you built to the directory straight from Claude. Right before you hand it to someone else, or lean on it in your own nightly runs, I badly wanted to promote "it worked once" into "it is guaranteed as a contract."

This article builds that guarantee with a small contract-test harness, using the MCP SDK for TypeScript.

It Worked Once, Then Broke Silently at Night

Checking a connector by hand in a chat has a fatal blind spot: a human calls it once, sees only the successful response, and decides it works.

What breaks in unattended runs is almost always outside the success path. A tool returns an error as an isError: true structured response, and the caller misreads it as success. The response shape shifts slightly from last time, and the downstream parser quietly grabs nothing. A retry fires a write twice. None of these ever surface from a single interactive call.

I feel this acutely as an indie developer who hands nightly updates of several technical blogs to the model. The reliability of an unattended job is decided not by how the connector behaves at its best, but by whether it keeps its promises when things wobble. So what you should verify before submission or production is not whether features exist, but the very contract each tool claims to honor.

The Four Properties a Contract Test Checks

A contract test is a little different from both unit tests and end-to-end tests. It machine-checks the "promises" a connector publishes, from the outside, as a client would. Before shipping to unattended use, pin down at least these four.

Property	What it checks	What breaks if violated
Self-description	Every tool has a description and a valid inputSchema	The model misuses or ignores the tool
Response contract	The happy-path response shape matches the declaration	Downstream parsing quietly grabs nothing
Idempotency	Calling a write twice yields one side effect	Retries double-fire
Latency	Per-tool response time stays within budget	Timeouts drop the unattended job

Let us turn each into working code. Everything below uses the @modelcontextprotocol/sdk client to connect to the connector under test from the outside.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦Validation code that enumerates every tool in an MCP connector and catches missing descriptions and inputSchema before you submit

✦An idempotency test that detects double-fired writes with a sentinel key, including p95 latency measurement

✦A procedure for setting unattended-job timeouts from measured per-tool p50/p95 latency rather than guessing

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Validate Each Tool's Self-Description

The first gate is whether the model has enough information to pick the tool correctly. If the description is empty, or the inputSchema lacks type: "object", the model cannot assemble arguments and will silently avoid or misuse the tool.

Connect the client, enumerate tools with listTools(), and inspect them one by one.

import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
 
// Launch the connector under test as a child process and connect to it
export async function connect(): Promise<Client> {
  const transport = new StdioClientTransport({
    command: "node",
    args: ["dist/server.js"],
  });
  const client = new Client(
    { name: "contract-tester", version: "1.0.0" },
    { capabilities: {} },
  );
  await client.connect(transport);
  return client;
}
 
type Issue = { tool: string; rule: string; detail: string };
 
export async function checkSelfDescription(client: Client): Promise<Issue[]> {
  const { tools } = await client.listTools();
  const issues: Issue[] = [];
 
  for (const t of tools) {
    // A missing or too-short description means the model cannot judge intent
    if (!t.description || t.description.trim().length < 12) {
      issues.push({ tool: t.name, rule: "description", detail: "description missing or too short" });
    }
    // inputSchema must be an object type with properties at minimum
    const schema = t.inputSchema as { type?: string; properties?: object } | undefined;
    if (!schema || schema.type !== "object") {
      issues.push({ tool: t.name, rule: "schema", detail: "inputSchema is not an object" });
    } else if (!schema.properties) {
      issues.push({ tool: t.name, rule: "schema", detail: "properties is undefined" });
    }
  }
  return issues;
}

This check alone stops accidents before submission: a refactor that dropped a description, a hand-written schema that forgot type. It once caught a tool that still carried a non-empty required despite taking no arguments.

Pin the Response Shape of Read Tools

Next, pin that the happy-path response comes back in the declared shape. The key here is to check the shape, not the values. Fixing the values too makes a brittle test that fails on every trivial change to the connector.

Call each read-only tool once with the minimal valid arguments, and check how content and isError behave.

export async function checkReadContract(
  client: Client,
  tool: string,
  args: Record<string, unknown>,
): Promise<Issue[]> {
  const issues: Issue[] = [];
  const res = await client.callTool({ name: tool, arguments: args });
 
  // If a happy-path call returns isError, the args or the server violate the contract
  if (res.isError) {
    issues.push({ tool, rule: "read-error", detail: "isError returned for happy-path args" });
    return issues;
  }
  // An MCP tool response is expected to carry a content array
  if (!Array.isArray(res.content) || res.content.length === 0) {
    issues.push({ tool, rule: "read-shape", detail: "content is empty or not an array" });
  }
  // Tools that promise structured output should pin the presence of structuredContent
  if (res.structuredContent !== undefined && typeof res.structuredContent !== "object") {
    issues.push({ tool, rule: "read-structured", detail: "structuredContent is not an object" });
  }
  return issues;
}

Why shape, not values? The classic way an unattended job breaks silently is a response that is "not an error, but not the expected shape either." Code that reads content[0].text without checking isError swallows an error response as normal text. Pinning the shape contract first guarantees the minimum the downstream parser is allowed to assume.

If you find yourself wanting checks that depend on the response body, the thinking in handling remote MCP tool-call hangs helps, since remote responses wobble. A wait without a deadline turns into a silent stall in unattended runs.

Verify Write Idempotency with a Sentinel

The double-fire from the opening is caught here. Idempotency is the property that applying the same operation twice yields one side effect. To verify it without actually causing a side effect, use a sentinel approach.

The idea: generate one unique idempotency key (the sentinel), and call the write tool twice with it. If the tool is idempotent, the second call returns "already applied" and no new record appears. The verifier confirms, via a read tool, that the target count grew by exactly one across the calls.

import { randomUUID } from "node:crypto";
 
export async function checkIdempotency(
  client: Client,
  writeTool: string,
  countTool: string,
): Promise<Issue[]> {
  const issues: Issue[] = [];
  const key = `contract-${randomUUID()}`; // unique sentinel
 
  const before = await readCount(client, countTool);
 
  // Write twice with the same idempotency key
  const first = await client.callTool({ name: writeTool, arguments: { idempotencyKey: key, payload: { sentinel: key } } });
  const second = await client.callTool({ name: writeTool, arguments: { idempotencyKey: key, payload: { sentinel: key } } });
 
  const after = await readCount(client, countTool);
 
  if (first.isError) {
    issues.push({ tool: writeTool, rule: "write-first", detail: "first write failed" });
  }
  // The second call returning a success (including "already applied") is the idempotent design
  if (second.isError) {
    issues.push({ tool: writeTool, rule: "idempotent-retry", detail: "second call returned isError" });
  }
  // If the count grew by two, it double-fired
  if (after - before > 1) {
    issues.push({ tool: writeTool, rule: "idempotent-effect", detail: `count grew by ${after - before} (expected 1)` });
  }
  return issues;
}
 
async function readCount(client: Client, countTool: string): Promise<number> {
  const res = await client.callTool({ name: countTool, arguments: {} });
  const text = (res.content?.[0] as { text?: string })?.text ?? "0";
  return Number.parseInt(text, 10) || 0;
}

If the connector is not designed to accept an idempotency key, this test fails without mercy. That is fine. A failure is the signal to introduce an idempotency key on the write tool, or at least redesign it to be retry-safe. For the server-side idempotency design itself, idempotent tools and the outbox pattern in the Agent SDK goes deeper.

I run this pattern through every connector that includes writes, without exception, because the machine steps on a trap I would never hit in a chat.

Set Timeout Budgets from Per-Tool Latency

July's observability work made connector errors and latency visible across products. Still, how many seconds to set an unattended timeout is most reliably answered by measuring in your own environment. While you are at it in the contract test, capture per-tool p50/p95.

export async function measureLatency(
  client: Client,
  tool: string,
  args: Record<string, unknown>,
  runs = 20,
): Promise<{ p50: number; p95: number }> {
  const samples: number[] = [];
  for (let i = 0; i < runs; i++) {
    const start = performance.now();
    await client.callTool({ name: tool, arguments: args });
    samples.push(performance.now() - start);
  }
  samples.sort((a, b) => a - b);
  const at = (q: number) => Math.round(samples[Math.floor(q * (samples.length - 1))]);
  return { p50: at(0.5), p95: at(0.95) };
}

Measuring two of my own connectors 20 times each showed this gap. Between a local stdio server and a remote connector across the network, p95 differs by an order of magnitude.

Tool	Transport	p50	p95	Suggested timeout
List articles	Local stdio	118ms	472ms	3s
Get count	Local stdio	96ms	388ms	3s
Query external data	Remote HTTP	640ms	1,840ms	8s

I set timeouts at roughly 3–4x the p95. Set it right on top of p95 and normal jitter drops the job needlessly; let it wait forever and a stuck remote call halts the whole run. Doing this "measure, then decide" once also makes it easier to separate other failure modes, such as OAuth token refresh for remote MCP. Is it slow, or is it being rejected at auth? The numbers give you a hunch before you open the logs.

Consolidate into CI and a Pre-Submission Check

Finally, gather these checks into a single runner that returns pass or fail via the exit code. The point is to leave no room for a human to eyeball the result. Same philosophy as unattended operations: leave only green or red.

async function main() {
  const client = await connect();
  const issues: Issue[] = [];
 
  issues.push(...(await checkSelfDescription(client)));
  issues.push(...(await checkReadContract(client, "list_articles", { limit: 1 })));
  issues.push(...(await checkIdempotency(client, "upsert_article", "count_articles")));
 
  const lat = await measureLatency(client, "list_articles", { limit: 1 });
  console.log(`list_articles p50=${lat.p50}ms p95=${lat.p95}ms`);
 
  await client.close();
 
  if (issues.length > 0) {
    for (const i of issues) console.error(`❌ [${i.tool}] ${i.rule}: ${i.detail}`);
    process.exit(1); // stop the submission or deploy right here
  }
  console.log("✅ contract passed");
}
 
main().catch((e) => { console.error(e); process.exit(1); });

Drop this into a pre-submission GitHub Actions job, or a hook that runs whenever the connector changes. You are making the machine hold the line: nothing goes to the directory or an unattended job unless it is green. Keep verification and publishing in separate stages, a pitfall I also touched on in mixing the gate and push into one call.

The Next Step

Start by running just checkSelfDescription against a connector you have on hand. It is a few dozen lines that enumerate tools and check descriptions and inputSchema. Even that should surface a missing field or two you could never catch in a chat. From there, extend one rung at a time into response contracts, idempotency, and latency, and "it worked once" becomes "it keeps its promise even when things wobble."

I have not finished writing contracts for every tool of mine yet, but I am pinning them down starting with the connectors that include writes. That small step before handing something over is what keeps my night logs quiet. Thank you for reading.

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.