⬡ API & SDK/2026-04-12Advanced

Testing Claude API Applications — Unit, Integration, and E2E Patterns That Hold Up Against Probabilistic Output

Solve the 'AI output changed and broke my tests' problem for good. Learn to combine mocks, semantic assertions, and snapshot testing into a practical test design pattern for Claude API applications.

claude-api⁸¹ testing⁶ vitest² ci-cd⁸ quality-assurance³ typescript¹¹

✦ Premium Article

The moment you ship a product with Claude API inside it, you hit a wall that traditional software testing never prepared you for. Send the same prompt twice, and you get two different responses. Yesterday's passing test suite turns red overnight. Reaching for expect(response).toBe(exactString) against AI output is like holding a ruler to the wind.

The core problem reduces to one fact: AI output is probabilistic. But probabilistic doesn't mean untestable. Shift your test target from "the exact output string" to "the properties the output must satisfy," and a robust test suite becomes entirely buildable.

The testing architecture below is the one I've refined while running four AI-powered sites in production. From unit tests to E2E, you'll see exactly what to test and how at each layer, with working Vitest + TypeScript code you can drop into your project today.

Why Testing AI Applications Is Fundamentally Different

Traditional software testing rests on determinism. add(2, 3) always returns 5. Claude API applications introduce three sources of uncertainty that invalidate that assumption.

1. Non-deterministic output. Even with temperature: 0, model updates can shift output. Two identical requests separated by a model deployment may produce structurally different responses. You cannot pin AI output the way you pin a database query result.

2. Format instability. Ask for JSON and you might get JSON wrapped in a markdown code fence, or JSON with an extra commentary sentence before it. Structured Output via tool_use stabilizes the schema, but the values inside that schema still fluctuate.

3. Subjective quality. "A good summary" or "a helpful answer" cannot be expressed as assertEquals. The judgment that a human makes when reading output — "yes, this is correct" — needs to be translated into programmatic assertions.

To address all three, I use a modified test pyramid:

Unit tests (70%): Mock the API entirely. Verify application logic at high speed and zero cost.
Semantic tests (20%): Evaluate the meaning of AI output programmatically.
Integration / E2E tests (10%): Hit the real API. Validate end-to-end behavior.

The ratio matters. Too many integration tests and your CI bill spirals. Too few and you miss prompt quality regressions. This balance took months of iteration to get right.

Three Mock Strategies for Claude API — And When to Use Each

Mocking forms the foundation of your unit test layer. There are three distinct approaches, each suited to different test targets.

Strategy 1: SDK-Level Mocking (Recommended Default)

Mock the Anthropic SDK client directly. Simplest to set up, covers the majority of unit test cases.

// src/services/summarizer.ts
import Anthropic from "@anthropic-ai/sdk";
 
export class ArticleSummarizer {
  private client: Anthropic;
 
  constructor(client?: Anthropic) {
    // Dependency injection makes mock injection trivial
    this.client = client ?? new Anthropic();
  }
 
  async summarize(article: string): Promise<{
    summary: string;
    keyPoints: string[];
    readingTime: number;
  }> {
    const response = await this.client.messages.create({
      model: "claude-sonnet-4-6",
      max_tokens: 1024,
      messages: [
        {
          role: "user",
          content: `Summarize the following article. Return JSON:
          {"summary": "under 200 chars", "keyPoints": ["point1", "point2", "point3"], "readingTime": estimatedMinutes}
          
          Article: ${article}`,
        },
      ],
    });
 
    const text =
      response.content[0].type === "text" ? response.content[0].text : "";
 
    try {
      return JSON.parse(text);
    } catch {
      throw new Error(
        `Failed to parse AI response as JSON: ${text.substring(0, 100)}...`
      );
    }
  }
}

// src/services/__tests__/summarizer.test.ts
import { describe, it, expect, vi } from "vitest";
import { ArticleSummarizer } from "../summarizer";
 
function createMockClient(responseText: string) {
  return {
    messages: {
      create: vi.fn().mockResolvedValue({
        content: [{ type: "text", text: responseText }],
        usage: { input_tokens: 100, output_tokens: 50 },
        stop_reason: "end_turn",
      }),
    },
  } as any;
}
 
describe("ArticleSummarizer", () => {
  it("parses a valid JSON response correctly", async () => {
    const mockResponse = JSON.stringify({
      summary: "A summary of the article about AI testing",
      keyPoints: ["Point one", "Point two", "Point three"],
      readingTime: 5,
    });
 
    const client = createMockClient(mockResponse);
    const summarizer = new ArticleSummarizer(client);
    const result = await summarizer.summarize("Test article body...");
 
    // Validate structure and constraints, not exact values
    expect(result).toHaveProperty("summary");
    expect(result.summary.length).toBeLessThanOrEqual(200);
    expect(result.keyPoints).toHaveLength(3);
    expect(result.readingTime).toBeGreaterThan(0);
 
    // Verify API call parameters
    expect(client.messages.create).toHaveBeenCalledWith(
      expect.objectContaining({
        model: "claude-sonnet-4-6",
        max_tokens: 1024,
      })
    );
  });
 
  it("throws a descriptive error on invalid JSON", async () => {
    const client = createMockClient("This is not JSON at all");
    const summarizer = new ArticleSummarizer(client);
 
    await expect(summarizer.summarize("Test article")).rejects.toThrow(
      "Failed to parse AI response as JSON"
    );
  });
 
  it("handles responses with no text blocks", async () => {
    const client = {
      messages: {
        create: vi.fn().mockResolvedValue({
          content: [{ type: "image", source: {} }],
          usage: { input_tokens: 10, output_tokens: 0 },
          stop_reason: "end_turn",
        }),
      },
    } as any;
 
    const summarizer = new ArticleSummarizer(client);
    await expect(summarizer.summarize("test")).rejects.toThrow();
  });
});

The critical insight here: these tests validate application logic, not AI output quality. JSON parsing, error handling, API parameter correctness — all deterministically testable. Save the probabilistic testing for the semantic layer.

Strategy 2: HTTP-Level Interception with MSW

When you need to test request headers (x-api-key, anthropic-version) or simulate streaming responses, Mock Service Worker gives you control at the network level.

// test/mocks/handlers.ts
import { http, HttpResponse } from "msw";
 
export const claudeHandlers = [
  http.post("https://api.anthropic.com/v1/messages", async ({ request }) => {
    const body = (await request.json()) as any;
    const userMessage = body.messages?.[0]?.content || "";
 
    if (userMessage.includes("summarize")) {
      return HttpResponse.json({
        id: "msg_test_001",
        type: "message",
        role: "assistant",
        content: [
          {
            type: "text",
            text: JSON.stringify({
              summary: "Test summary",
              keyPoints: ["Point A", "Point B", "Point C"],
              readingTime: 3,
            }),
          },
        ],
        model: body.model,
        stop_reason: "end_turn",
        usage: { input_tokens: 150, output_tokens: 80 },
      });
    }
 
    return HttpResponse.json({
      id: "msg_test_default",
      type: "message",
      role: "assistant",
      content: [{ type: "text", text: "Default response." }],
      model: body.model,
      stop_reason: "end_turn",
      usage: { input_tokens: 50, output_tokens: 20 },
    });
  }),
];

MSW shines for testing authentication flows and streaming, but its setup cost is higher. Reserve it for tests that specifically need network-layer verification.

Strategy 3: Response Fixtures (Recorded Snapshots)

Record real API responses to files and replay them in tests. First run hits the live API; subsequent runs use the cached fixture.

// test/fixtures/record.ts
import Anthropic from "@anthropic-ai/sdk";
import { writeFileSync, existsSync, mkdirSync } from "fs";
 
const client = new Anthropic();
 
export async function recordFixture(
  name: string,
  params: Anthropic.MessageCreateParams
): Promise<void> {
  const dir = "test/fixtures/responses";
  if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
 
  const response = await client.messages.create(params);
  const fixture = {
    params: { ...params, messages: "[REDACTED]" },
    response,
    recordedAt: new Date().toISOString(),
    modelVersion: response.model,
  };
 
  writeFileSync(`${dir}/${name}.json`, JSON.stringify(fixture, null, 2));
  console.log(
    `Fixture recorded: ${name} (${response.usage.output_tokens} tokens)`
  );
}

One gotcha: when the model version changes, your fixtures go stale. Including recordedAt and modelVersion in the fixture metadata makes it easy to detect and refresh outdated recordings.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦If you've struggled with 'AI output changed and broke my tests,' you'll get test design patterns backed by measured results

✦Measure your mocks against the seven response shapes that actually occur, and close the JSON-parsing gaps that only surface in production

✦Build a system that catches prompt regressions while keeping test API costs under a few dollars per month

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

When Your Mocks Lie — Measuring Mock Fidelity

There's a hole in everything we've built so far.

Every mock in the previous section returns clean, well-formed JSON. But as this article's own opening section pointed out, Claude often wraps JSON in a sentence or two of explanation even when you ask it not to. Which means the unit tests above never exercise the single failure mode this article named as the biggest one.

I fell into exactly this trap. Unit tests green, CI green, and production still logging Failed to parse AI response every day. The mocks were too well behaved, so the suite was only ever validating the world I wished I had.

The size of the gap is measurable rather than debatable. I collected seven response shapes that actually show up in practice and pushed each one through the parsing logic from summarize() above (run on Node.js v22.22 — a pure parsing test, no model calls involved).

Response shape	Original code	Error it produces
1. Bare JSON only	✅ Passes	—
2. Prefaced with prose ("Sure — here's the summary:")	❌ Fails	`Unexpected token 'S'`
3. Wrapped in a json code fence	❌ Fails	`Unexpected token` (backtick)
4. Truncated by max_tokens	❌ Fails	`Unterminated string in JSON at position 70`
5. First block is tool_use	❌ Fails	`Unexpected end of JSON input`
6. Empty content array	❌ Fails	`Cannot read properties of undefined`
7. First block is thinking	❌ Fails	`Unexpected end of JSON input`

One shape out of seven — a 14% pass rate. And that one shape is precisely the only one the mocks reproduced.

Look closely at the error messages, too. The Cannot read properties of undefined from shape 6 and the Unexpected end of JSON input from shapes 5 and 7 tell you nothing about what actually went wrong. When I saw those in production logs, I assumed the prompt was at fault and lost a full day to it. The real cause was treating content[0] as a text block unconditionally.

Making the parser survive every shape

The fix has two layers. First, check stop_reason to decide whether the response is even complete. Second, search the content array for the block you want instead of indexing into position zero.

// src/services/parse-response.ts
import type Anthropic from "@anthropic-ai/sdk";
 
/** Pull the first JSON object out of text, respecting string literals */
function extractJson(text: string): unknown {
  const fence = text.match(/`{3}(?:json)?\s*([\s\S]*?)`{3}/);
  const body = fence ? fence[1] : text;
  const start = body.indexOf("{");
  if (start === -1) throw new Error("No JSON object found in response text");
 
  let depth = 0;
  let inString = false;
  let escaped = false;
 
  for (let i = start; i < body.length; i++) {
    const c = body[i];
    if (inString) {
      if (escaped) escaped = false;
      else if (c === "\\") escaped = true;
      else if (c === '"') inString = false;
      continue;
    }
    if (c === '"') inString = true;
    else if (c === "{") depth++;
    else if (c === "}") {
      depth--;
      if (depth === 0) return JSON.parse(body.slice(start, i + 1));
    }
  }
  throw new Error("JSON ends mid-object (likely truncated by max_tokens)");
}
 
export function parseStructured(response: Anthropic.Message): unknown {
  // 1. Reject incomplete responses before attempting to parse
  if (response.stop_reason === "max_tokens") {
    throw new Error("Response truncated by max_tokens — raise the limit or split the output");
  }
  // 2. If a tool_use block exists, the value is already structured
  const tool = response.content.find((b) => b.type === "tool_use");
  if (tool && tool.type === "tool_use") return tool.input;
 
  // 3. Search for the text block so thinking blocks don't shadow it
  const textBlock = response.content.find((b) => b.type === "text");
  if (!textBlock || textBlock.type !== "text") {
    throw new Error(`No text block in response (stop_reason=${response.stop_reason})`);
  }
  return extractJson(textBlock.text);
}

Running the same seven shapes through this version:

Response shape	Original	parseStructured
1. Bare JSON	✅	✅ Value returned
2. Prose preface	❌	✅ Value returned
3. Code fence	❌	✅ Value returned
4. max_tokens truncation	❌	⚠️ "Response truncated by max_tokens"
5. tool_use first	❌	✅ Value returned
6. Empty content	❌	⚠️ "No text block in response (stop_reason=end_turn)"
7. thinking first	❌	✅ Value returned

One out of seven became five out of seven. The two remaining failures are intentional. Shapes 4 and 6 can only be resolved by a retry or a config change, so failing loudly with a message that names the cause is the correct behavior. What the tests should verify isn't that everything succeeds — it's that failures are diagnosable.

Build mocks from shapes you've observed

With that settled, turn the mock factory itself into a catalog of shapes.

// test/mocks/response-shapes.ts
const FENCE = "`".repeat(3);
 
const VALID = {
  summary: "A summary of an article about the TypeScript type system, covering generics and conditional types.",
  keyPoints: ["Benefits of static typing", "Impact on large codebases"],
  readingTime: 5,
};
 
export const responseShapes = {
  cleanJson: () => msg("end_turn", [{ type: "text", text: JSON.stringify(VALID) }]),
  prosePrefixed: () =>
    msg("end_turn", [
      { type: "text", text: `Sure — here's the summary:\n\n${JSON.stringify(VALID)}` },
    ]),
  fenced: () =>
    msg("end_turn", [
      { type: "text", text: `${FENCE}json\n${JSON.stringify(VALID)}\n${FENCE}` },
    ]),
  truncated: () =>
    msg("max_tokens", [{ type: "text", text: JSON.stringify(VALID).slice(0, 70) }]),
  toolUse: () =>
    msg("tool_use", [{ type: "tool_use", id: "tu_1", name: "emit_summary", input: VALID }]),
  emptyContent: () => msg("end_turn", []),
  thinkingFirst: () =>
    msg("end_turn", [
      { type: "thinking", thinking: "Organizing the article structure...", signature: "sig" },
      { type: "text", text: JSON.stringify(VALID) },
    ]),
};
 
function msg(stopReason: string, content: unknown[]) {
  return {
    id: "msg_test",
    type: "message",
    role: "assistant",
    model: "claude-haiku-4-5",
    stop_reason: stopReason,
    usage: { input_tokens: 100, output_tokens: 50 },
    content,
  } as any;
}

// test/services/parse-response.test.ts
import { describe, it, expect } from "vitest";
import { parseStructured } from "../../src/services/parse-response";
import { responseShapes } from "../mocks/response-shapes";
 
describe("parseStructured — response shape coverage", () => {
  it.each(["cleanJson", "prosePrefixed", "fenced", "toolUse", "thinkingFirst"] as const)(
    "%s yields a usable value",
    (shape) => {
      const result = parseStructured(responseShapes[shape]()) as { summary: string };
      expect(result.summary).toContain("TypeScript");
    }
  );
 
  it("truncation fails with a message naming max_tokens", () => {
    expect(() => parseStructured(responseShapes.truncated())).toThrow(/max_tokens/);
  });
 
  it("empty content fails with the stop_reason attached", () => {
    expect(() => parseStructured(responseShapes.emptyContent())).toThrow(/stop_reason=end_turn/);
  });
});

Since adding this table-driven test, none of the projects I run as an indie developer have logged a single JSON-parsing failure in production. When a new shape shows up, it costs one line in responseShapes to protect that case permanently.

If you want stronger guarantees, stop parsing text at all. Pass your output schema through tools and force it with tool_choice, and the value arrives structured as tool_use.input. Shapes 2, 3, and 7 stop existing, and the parsing branches shrink with them. Reducing what needs defending beats defending it well.

Semantic Assertions — Testing What AI Output Means

Unit tests verify your application logic. Semantic tests verify AI output quality. This is where Claude API testing gets interesting.

Pattern 1: Structural Validation with Zod

Verify that AI output conforms to an expected structure using Zod schemas. This catches format drift before it reaches your users.

// src/validators/ai-response.ts
import { z } from "zod";
 
export const SummaryResponseSchema = z.object({
  summary: z
    .string()
    .min(50, "Summary too short (minimum 50 characters)")
    .max(200, "Summary too long (maximum 200 characters)"),
  keyPoints: z
    .array(z.string().min(10))
    .min(2, "At least 2 key points required")
    .max(5, "Maximum 5 key points"),
  readingTime: z
    .number()
    .int()
    .min(1, "Reading time must be at least 1 minute")
    .max(60, "Reading time seems unrealistic"),
});

Zod validation pulls double duty: it works in your test suite and as runtime validation in production. One schema, two layers of protection.

Pattern 2: Keyword and Negation Checks

Test for the presence of required concepts and the absence of forbidden ones. Not as precise as exact matching, but far more resilient to output variation.

// test/helpers/semantic-assertions.ts
export function assertContainsKeywords(
  text: string,
  required: string[],
  forbidden: string[] = []
): void {
  const missing = required.filter(
    (kw) => !text.toLowerCase().includes(kw.toLowerCase())
  );
  if (missing.length > 0) {
    throw new Error(
      `Missing required keywords: [${missing.join(", ")}]\nFirst 200 chars: ${text.substring(0, 200)}`
    );
  }
 
  const found = forbidden.filter((kw) =>
    text.toLowerCase().includes(kw.toLowerCase())
  );
  if (found.length > 0) {
    throw new Error(
      `Found forbidden keywords: [${found.join(", ")}]\nFirst 200 chars: ${text.substring(0, 200)}`
    );
  }
}

Pattern 3: LLM-as-Judge (AI Evaluating AI)

The most powerful approach, where Claude itself evaluates output quality. Expensive, but irreplaceable for nuanced quality checks.

And this is where the previous section comes straight back. The judge I first wrote pulled its verdict out of content[0] and ran JSON.parse on it — the exact parser we just measured at one shape out of seven. When your evaluator sits on a broken parser, your tests stop measuring quality and start measuring parse failures.

Here's how that plays out. The judge replies "Score: 4. My reasoning is…", the parse fails, and the score is recorded as 0. Prompt quality hasn't moved at all, but the regression suite turns red anyway. Few failures are more misleading to debug.

The fix is to receive the verdict as a tool call. Force the scoring tool with tool_choice and the score and reasoning arrive as structured values — the parsing branch disappears entirely.

// src/evaluation/llm-judge.ts
import Anthropic from "@anthropic-ai/sdk";
 
export interface EvaluationResult {
  score: number; // 1-5
  reasoning: string;
  passed: boolean;
}
 
const JUDGE_TOOL: Anthropic.Tool = {
  name: "submit_evaluation",
  description: "Score an output against the given criteria",
  input_schema: {
    type: "object",
    properties: {
      score: { type: "integer", minimum: 1, maximum: 5, description: "1 is poor, 5 is excellent" },
      reasoning: { type: "string", description: "One or two sentences justifying the score" },
    },
    required: ["score", "reasoning"],
  },
};
 
export async function evaluateWithLLM(
  client: Anthropic,
  input: string,
  output: string,
  criteria: string
): Promise<EvaluationResult> {
  const response = await client.messages.create({
    model: "claude-sonnet-4-6", // judge with a stronger model than production uses
    max_tokens: 512,
    temperature: 0, // cut judgment variance structurally
    tools: [JUDGE_TOOL],
    tool_choice: { type: "tool", name: "submit_evaluation" }, // force a structured verdict
    messages: [
      {
        role: "user",
        content: [
          "Score the following output against the criteria below.",
          `# Criteria\n${criteria}`,
          `# Input\n${input}`,
          `# Output\n${output}`,
        ].join("\n\n"),
      },
    ],
  });
 
  const tool = response.content.find((b) => b.type === "tool_use");
  if (!tool || tool.type !== "tool_use") {
    // Fail loudly instead of swallowing this as a score of 0
    throw new Error(`No verdict returned (stop_reason=${response.stop_reason})`);
  }
 
  const { score, reasoning } = tool.input as { score: number; reasoning: string };
  return { score, reasoning, passed: score >= 3 };
}

Judging with a stronger model than production is deliberate. Evaluating is harder than generating, so economizing here quietly erodes the meaning of your thresholds — the opposite of the call we make for integration tests, where Haiku is plenty.

The throw matters just as much. The old version returned a score of 0 when parsing failed, which collapsed "the output was terrible" and "we couldn't score it" into the same number. Don't record an unmeasurable result as a measurement. Test trustworthiness is decided right here.

Even so, LLM-as-Judge carries one unavoidable problem: scoring consistency isn't guaranteed. The same input and output can draw different scores across runs. temperature: 0 shrinks the spread considerably but doesn't eliminate it. The practical countermeasure is to run the evaluation three times and take the median.

export async function evaluateWithConsensus(
  client: Anthropic,
  input: string,
  output: string,
  criteria: string,
  runs: number = 3
): Promise<EvaluationResult> {
  const results = await Promise.all(
    Array.from({ length: runs }, () =>
      evaluateWithLLM(client, input, output, criteria)
    )
  );
 
  const sorted = [...results].sort((a, b) => a.score - b.score);
  const median = sorted[Math.floor(sorted.length / 2)];
 
  return {
    score: median.score,
    reasoning: `${median.reasoning} (${runs} evaluations: ${results.map((r) => r.score).join(", ")})`,
    passed: median.score >= 3,
  };
}

Attaching all three scores to reasoning is for reading CI logs after the fact. A median of 3 backed by 2, 3, 5 means the judges disagreed and the case sits near your threshold; 3, 3, 3 means it's stable. Looking at the median alone, you'd never see the difference.

Integration Tests Against the Live API — Fighting the Cost Problem

Integration tests hit the real Claude API. The biggest challenge isn't writing them — it's paying for them. An uncontrolled integration test suite can rack up hundreds of dollars monthly in API costs.

Four Rules for Cost-Effective Integration Testing

Rule 1: Use a cheaper model. Run integration tests against claude-haiku-4-5. Response structure and basic quality are verifiable with Haiku. Even if production uses Opus or Sonnet, the risk of model-specific bugs slipping through is minimal.

Rule 2: Constrain max_tokens. Tell the AI "respond in 3 sentences or fewer" in your test prompts. This slashes token consumption dramatically.

Rule 3: Control execution frequency. Don't run integration tests on every push. Limit them to daily scheduled runs or commits explicitly tagged with [integration].

Rule 4: Cache responses. For identical prompts, cache the first response and replay it for subsequent runs within a TTL window.

// test/helpers/cached-client.ts
import Anthropic from "@anthropic-ai/sdk";
import { createHash } from "crypto";
import { existsSync, readFileSync, writeFileSync, mkdirSync } from "fs";
 
const CACHE_DIR = "test/.api-cache";
 
/** Key-order-independent serialization for cache keys */
function stableStringify(value: unknown): string {
  if (Array.isArray(value)) return `[${value.map(stableStringify).join(",")}]`;
  if (value && typeof value === "object") {
    const entries = Object.entries(value as Record<string, unknown>)
      .sort(([a], [b]) => a.localeCompare(b))
      .map(([k, v]) => `${JSON.stringify(k)}:${stableStringify(v)}`);
    return `{${entries.join(",")}}`;
  }
  return JSON.stringify(value);
}
 
export class CachedTestClient {
  private client: Anthropic;
  private ttlMs: number;
 
  constructor(ttlMs: number = 24 * 60 * 60 * 1000) {
    this.client = new Anthropic();
    this.ttlMs = ttlMs;
    if (!existsSync(CACHE_DIR)) mkdirSync(CACHE_DIR, { recursive: true });
  }
 
  async createMessage(
    params: Anthropic.MessageCreateParams
  ): Promise<Anthropic.Message> {
    const cacheKey = createHash("sha256")
      .update(stableStringify(params))
      .digest("hex");
    const cachePath = `${CACHE_DIR}/${cacheKey}.json`;
 
    if (existsSync(cachePath)) {
      const cached = JSON.parse(readFileSync(cachePath, "utf-8"));
      if (Date.now() - cached.timestamp < this.ttlMs) {
        return cached.response;
      }
    }
 
    const response = await this.client.messages.create(params);
    writeFileSync(
      cachePath,
      JSON.stringify({ timestamp: Date.now(), response, params })
    );
    return response;
  }
}

Integration Test Implementation

// test/integration/summarizer.integration.test.ts
import { describe, it, expect } from "vitest";
import { CachedTestClient } from "../helpers/cached-client";
import { ArticleSummarizer } from "../../src/services/summarizer";
import { SummaryResponseSchema } from "../../src/validators/ai-response";
import { assertContainsKeywords } from "../helpers/semantic-assertions";
 
const RUN_INTEGRATION = process.env.RUN_INTEGRATION_TESTS === "true";
 
describe.skipIf(!RUN_INTEGRATION)(
  "ArticleSummarizer — Integration",
  () => {
    const cachedClient = new CachedTestClient();
 
    it("returns a schema-valid summary from the real API", async () => {
      const testArticle = `
        TypeScript's type system adds static typing to JavaScript.
        Features like generics, conditional types, and mapped types
        let you write type-safe code while maintaining flexibility.
        In large projects, compile-time checks catch bugs early.
      `;
 
      const wrappedClient = {
        messages: { create: (p: any) => cachedClient.createMessage(p) },
      } as any;
 
      const summarizer = new ArticleSummarizer(wrappedClient);
      const result = await summarizer.summarize(testArticle);
 
      const validation = SummaryResponseSchema.safeParse(result);
      expect(validation.success).toBe(true);
 
      assertContainsKeywords(
        result.summary,
        ["TypeScript"],
        ["Python", "Java"]
      );
 
      expect(result.keyPoints.length).toBeGreaterThanOrEqual(2);
      expect(result.readingTime).toBeLessThan(10);
    }, 30_000);
  }
);

E2E Tests for AI-Powered User Experiences

E2E tests validate that AI features work correctly through actual user flows. Here's a Playwright implementation.

// e2e/ai-chat.spec.ts
import { test, expect } from "@playwright/test";
 
test.describe("AI Chat Feature", () => {
  test("user sends a question and receives an AI response", async ({
    page,
  }) => {
    await page.goto("/chat");
 
    const input = page.getByPlaceholder("Type your message...");
    await input.fill("What are TypeScript generics?");
    await page.getByRole("button", { name: "Send" }).click();
 
    await expect(page.getByTestId("loading-indicator")).toBeVisible();
 
    const response = page.getByTestId("ai-response").last();
    await expect(response).toBeVisible({ timeout: 30_000 });
 
    const text = await response.textContent();
    expect(text).toBeTruthy();
    expect(text!.length).toBeGreaterThan(50);
 
    await expect(page.getByTestId("loading-indicator")).not.toBeVisible();
    await expect(input).toBeEmpty();
  });
 
  test("displays error message on API failure", async ({ page }) => {
    await page.route("**/api/chat", (route) =>
      route.fulfill({
        status: 429,
        body: JSON.stringify({ error: "Rate limit exceeded" }),
      })
    );
 
    await page.goto("/chat");
    await page.getByPlaceholder("Type your message...").fill("test");
    await page.getByRole("button", { name: "Send" }).click();
 
    await expect(page.getByTestId("error-message")).toContainText(
      /try again|rate limit/i
    );
 
    await expect(
      page.getByRole("button", { name: /retry/i })
    ).toBeVisible();
  });
});

The key E2E consideration: set timeouts generously. Claude API responses typically take 2–15 seconds. The default 5-second Playwright timeout will cause flaky failures. Set AI-related assertions to at least 30 seconds.

CI/CD Integration — A GitHub Actions Blueprint

The key to CI/CD integration is separating execution conditions by test layer.

# .github/workflows/ai-tests.yml
name: AI Application Tests
 
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: "0 9 * * 1-5" # Weekdays 9:00 UTC
 
jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: "npm"
      - run: npm ci
      - run: npx vitest run --reporter=verbose
        env:
          NODE_ENV: test # No API key needed for unit tests
 
  integration-tests:
    if: github.event_name == 'schedule' || contains(github.event.head_commit.message, '[integration]')
    runs-on: ubuntu-latest
    needs: unit-tests
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: "npm"
      - run: npm ci
      - run: npx vitest run --reporter=verbose
        env:
          RUN_INTEGRATION_TESTS: "true"
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY_TEST }}

Critical cost safety measure: use a separate API key for testing. Create a test-only key in the Anthropic dashboard with a monthly spending cap (e.g., $10). This prevents runaway costs if a test bug causes an infinite loop.

Prompt Regression Testing — Changing Prompts Safely

The scariest production scenario is "I tweaked the prompt slightly and output quality dropped off a cliff." Regression tests catch this before users do.

// test/regression/prompt-regression.test.ts
import { describe, it, expect } from "vitest";
import { evaluateWithConsensus } from "../../src/evaluation/llm-judge";
import Anthropic from "@anthropic-ai/sdk";
 
const TEST_CASES = [
  {
    name: "Technical article summarization",
    input:
      "React Server Components render on the server side, reducing client bundle size...",
    criteria:
      "Technically accurate, accessible to beginners. Must mention server-side rendering and performance benefits.",
    minScore: 3,
  },
  {
    name: "Error message generation",
    input:
      "TypeError: Cannot read properties of undefined (reading 'map')",
    criteria:
      "Correctly identifies the error cause and provides specific, actionable fix steps.",
    minScore: 3,
  },
];
 
const RUN_REGRESSION = process.env.RUN_REGRESSION_TESTS === "true";
 
describe.skipIf(!RUN_REGRESSION)("Prompt Regression Suite", () => {
  const client = new Anthropic();
 
  TEST_CASES.forEach(({ name, input, criteria, minScore }) => {
    it(`${name}: scores >= ${minScore}`, async () => {
      const response = await client.messages.create({
        model: "claude-haiku-4-5",
        max_tokens: 512,
        messages: [{ role: "user", content: input }],
      });
 
      const output =
        response.content[0].type === "text"
          ? response.content[0].text
          : "";
 
      const evaluation = await evaluateWithConsensus(
        client,
        input,
        output,
        criteria
      );
 
      console.log(
        `  ${name}: score=${evaluation.score}, reason=${evaluation.reasoning}`
      );
 
      expect(evaluation.score).toBeGreaterThanOrEqual(minScore);
    }, 60_000);
  });
});

Common Mistakes and Pitfalls

Pitfall 1: Exact String Matching

The most common failure pattern. expect(response).toBe("specific string") doesn't work with AI output. Use structural validation (Zod), keyword checks, or LLM-as-Judge instead.

// Don't do this
expect(summary).toBe("TypeScript is a statically typed language.");
 
// Do this instead
expect(summary.length).toBeGreaterThan(20);
expect(summary.length).toBeLessThan(200);
assertContainsKeywords(summary, ["TypeScript", "type"]);
const validation = SummarySchema.safeParse({ summary });
expect(validation.success).toBe(true);

Pitfall 2: Running All Tests Against the Real API

Hundreds of API calls per CI run adds up fast. Stick to the test pyramid: 70% unit (mocked), 20% semantic, 10% integration (real API). Your wallet will thank you.

Pitfall 3: Forgetting to Test Streaming

If production uses stream: true, but your tests only verify standard responses, you're missing an entire class of bugs — mid-stream disconnections, chunk reassembly failures, partial JSON in the final chunk.

// Mock a streaming response
const mockStream = {
  async *[Symbol.asyncIterator]() {
    yield {
      type: "content_block_start",
      content_block: { type: "text", text: "" },
    };
    yield {
      type: "content_block_delta",
      delta: { type: "text_delta", text: "partial " },
    };
    yield {
      type: "content_block_delta",
      delta: { type: "text_delta", text: "response" },
    };
    yield { type: "message_stop" };
  },
};

Pitfall 4: Model Version Drift Between Test and Production

Testing with claude-haiku-4-5 while production runs claude-sonnet-4-6 can mask model-specific behaviors. Structured Output format precision varies across models. Run periodic integration tests with the production model too.

Pitfall 5: Using Production Data in Tests

Real user data leaking into test fixtures is a compliance nightmare. Always use synthetic test data. Every fixture committed to your repository should be safe to publish publicly.

Your Three Next Steps

Testing Claude API applications is a "grow it incrementally" problem, not a "get it perfect on day one" problem. Start tomorrow with these three actions, in order:

Build a response-shape catalog and run your SDK mocks against it. Start with five: prose-prefixed, code-fenced, max_tokens-truncated, tool_use, and empty content. Measured against those shapes, a parser that only expects clean JSON passed one out of seven. Zero cost, and you can start today.
Add Zod schema validation. Works in tests and as runtime validation in production. One schema, two layers of protection.
Set up a weekly integration test in CI. Use the CachedTestClient to control costs while catching prompt quality drift early.

Once your test suite gives you confidence that changes won't break things, you'll find yourself iterating on prompts much more aggressively — and that's exactly where the real product improvements come from.

If you'd rather eliminate the parse failures before they reach production, "Claude API JSON Output Fails: 5 Root Causes and Fixes" covers the causes in detail, and "Trusting Claude's Structured Output in Production — Validation Gates and Repair Loops" walks through moving to a design where there's no text to parse. For retry and error-classification design, see "Claude API Error Handling and Retry Strategies."

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.