Eliminating Flaky Tests Systematically with Claude Code

CI turns red, Slack pings you, and you open the log to find a familiar test failing. You hit Retry, it goes green, and you close the tab. Sound familiar?

Flaky tests quietly erode trust in your CI pipeline. The moment a team starts assuming "it'll pass on retry," real regressions get filed under the same shrug. I once shipped a release where a genuinely broken assertion was waved through as "probably flaky" — and the bug surfaced in production through a customer report. Since then, I've stopped treating flaky tests as a "fix later" problem and started treating them as the highest-priority diagnostic work in my queue.

This article walks through a practical workflow for using Claude Code to dismantle flaky tests methodically. There is no magical one-shot command, but if you classify the cause correctly, Claude Code becomes a remarkably capable partner.

Flaky tests are almost always one of three failure modes

The first move is to stop treating "intermittent test failure" as a single thing. The actual cause almost always falls into one of three buckets.

Order dependency: Test A passing leaves global state that breaks test B. Database rows, singletons, environment variables, module-level caches — anything that persists across tests is a candidate.

Timing dependency: A setTimeout(500) waiting for an animation, or a fixed sleep(1) waiting for a job to finish. CI environments fluctuate, and any wait that's "long enough on my laptop" will eventually fail somewhere with higher load.

External state dependency: Wall-clock time, randomness, network calls to third-party APIs, file system state. Anything that varies between runs is a fault line.

Each category demands a completely different fix. Diagnosing the category before asking Claude Code to repair anything is what separates a real fix from a sleep-and-pray.

The first request to Claude Code is "classify, don't fix"

When I encounter a flaky test, I never start with "fix this." I start with this:

Read this test file and classify the most likely cause from the three categories below:
 
1. Order dependency (global state pollution)
2. Timing dependency (async or wait-based)
3. External state dependency (time, randomness, APIs, files)
 
Cite specific lines and explain why. If multiple categories apply, rank them by likelihood.
 
Target: tests/integration/payment.test.ts

The reason matters. If you ask for a fix up front, Claude Code patches the most visible suspect — which might not be the root cause. The biggest risk in flaky-test work is landing a "symptom mask" that makes the next regression even harder to diagnose.

Step 1: Get a 100% reproduction script before fixing anything

Once you have a hypothesis, the next priority is reproducibility. A flaky test you can reproduce on demand is no longer flaky — it's a regular bug.

The reproduction strategy depends on the category.

For order dependency: shuffle the test order and run many iterations.

# Vitest
npx vitest run --sequence.shuffle --sequence.seed=12345
 
# Jest
jest --testSequencer=./scripts/random-sequencer.js

I often ask Claude Code: "Write a bash script that runs Vitest with random order 50 times and collects the seeds and test names that fail." It returns something like:

#!/usr/bin/env bash
# scripts/find-flaky.sh — surface order-dependent flakiness
set -uo pipefail
 
ITERATIONS="${1:-50}"
LOG_DIR="./.flaky-logs"
mkdir -p "$LOG_DIR"
 
> "$LOG_DIR/failures.tsv"
echo -e "iter\tseed\tfailed_test" > "$LOG_DIR/failures.tsv"
 
for i in $(seq 1 "$ITERATIONS"); do
  SEED=$RANDOM
  echo "▶ iteration $i (seed=$SEED)"
 
  if ! OUTPUT=$(npx vitest run --sequence.shuffle --sequence.seed="$SEED" --reporter=json 2>&1); then
    FAILED=$(echo "$OUTPUT" | jq -r '.testResults[].assertionResults[] | select(.status=="failed") | .fullName' 2>/dev/null | tr '\n' ',')
    echo -e "${i}\t${SEED}\t${FAILED}" >> "$LOG_DIR/failures.tsv"
  fi
done
 
echo ""
echo "=== Failure log ==="
cat "$LOG_DIR/failures.tsv"

A typical output:

=== Failure log ===
iter  seed   failed_test
3     8421   integration > payment > should reject expired card,
12    19284  integration > payment > should reject expired card,
27    4477   integration > payment > should reject expired card,

The same test failing at deterministic seeds is almost always order-dependent. If different tests fail at random, you likely have a shared resource being mutated across cases.

For timing dependency: skip the shuffle. Just run the same test repeatedly (--repeat=100 or a bash loop) and measure the failure rate.

For external state dependency: vary the external — change the system clock, drop the network, swap the random seed — and watch which axis triggers the failure.

Step 2: Distinguish a real fix from a coverup

Once reproduction is solid, ask for the fix. By default, Claude Code will often suggest one of:

A retry wrapper (expect(...).toEventuallyEqual(...))
A larger setTimeout or sleep
An it.skip to silence the test

These hide the symptom. They don't fix the cause. I send Claude Code an explicit constraint list:

Fix the root cause of this flake. The following approaches are forbidden:
 
- Increasing wait times (sleep, setTimeout, larger waitFor timeouts)
- Adding retries
- Skipping the test
- Swallowing failures with try/catch
 
Instead, use one of:
 
A. Reset shared state in beforeEach/afterEach so tests don't leak into each other
B. Wait on events or observable state, not on time
C. Make external dependencies deterministic (frozen clock, mocks, seeded randomness)

That single constraint block changes the quality of the proposed fix dramatically. The "wait on state, not time" rule alone solves a huge fraction of Playwright and Testing Library flakes.

Common before/after patterns

Before — fragile timing:

// Wait for animation by sleeping
await page.click("#open-modal");
await new Promise((r) => setTimeout(r, 500)); // 500ms isn't guaranteed
expect(await page.isVisible("#modal-content")).toBe(true);

After — wait on the actual state:

// Wait until the modal is genuinely visible
await page.click("#open-modal");
await page.waitForSelector("#modal-content[data-state='open']", {
  state: "visible",
  timeout: 5000, // upper-bound guard, not the actual wait
});
expect(await page.isVisible("#modal-content")).toBe(true);

The timeout here is an upper bound, not a fixed delay. The wait returns as soon as the condition is satisfied, which is what makes it robust on slow CI days.

Before — testing randomness with random data:

test("IDs are unique", () => {
  const id1 = generateId();
  const id2 = generateId();
  expect(id1).not.toBe(id2); // can collide by chance
});

After — seed the randomness, test the property:

import { mockRandom } from "./helpers/mock-random";
 
test("100 generated IDs do not collide (seeded)", () => {
  const restore = mockRandom([0.1, 0.2, 0.3]); // deterministic outputs
  try {
    const ids = Array.from({ length: 100 }, generateId);
    expect(new Set(ids).size).toBe(100);
  } finally {
    restore();
  }
});

If you want to assert "no collisions," extract the randomness as a separate concern and inject deterministic values into the test. The property under test stops depending on luck.

Step 3: Build a system that catches the next one

Patching individual flakes one at a time is whack-a-mole. To stop the same patterns from breeding, I keep three habits in any team I work with.

1. Run randomized order in CI by default. A test suite that only passes in declared order is a stack of dominoes waiting for a refactor to topple it. Yes, you'll spend the first week fixing tests that "always passed." That's the debt you didn't know you had.

2. Wrap all wait helpers. Have Claude Code sweep the test directory for ad-hoc waits:

Scan all files under tests/ and list every occurrence of:
 
- new Promise(resolve => setTimeout(resolve, ...))
- await sleep(...)
- await new Promise((r) => setTimeout(r, ...))
 
Output a table of file path, line number, and wait duration in ms.

You'll get a clear migration target — replace each with a project-wide waitForCondition helper. Claude Code is very good at this kind of broad mechanical sweep.

3. Track flake history. A test that flaked, was "fixed," and starts flaking again is signaling something different. At minimum, surface "tests retried most often in the last 30 days" so you can see regressions early.

What to do next

Next time CI fails and you catch yourself thinking "that test again," resist the Retry button. Open Claude Code, ask it to classify the failure into one of the three categories, and stop there. You haven't fixed anything yet — but naming the failure as "this is timing-dependent" is the actual breakthrough. From there, the constraint block above (no longer waits, no retries, no skips) keeps the fix honest, and the systemic habits keep the next flake from sneaking in unnoticed.