Claude Code Says 'All Tests Pass' but CI Goes Red — Designing Around Exit Code Pitfalls

You ask Claude Code to verify a change, get back a confident "all tests pass," and push the branch. A few minutes later, GitHub Actions paints the dashboard red. In twelve years as an indie developer, this is the single category of incident that grew the fastest after I started leaning on Claude Code.

The strange part is that Claude Code is not lying. If you scroll back through the transcript, the model genuinely read something like pass: 142 in the output. What broke is one layer below — the way the shell and the test runner reported the real outcome, and whether that signal ever reached Claude Code's summarizer.

This article walks through the four patterns I keep running into with Vitest, Jest, pytest, and go test, plus one pipeline trap that quietly disables the fix for the second one. Everything here comes from repositories I maintain solo. At the end we assemble a three-layer defense that, in practice, drives the recurrence rate close to zero.

The symptom — three ways "tests passed" becomes a red CI

You usually notice this in one of three shapes.

The first is the cleanest: Claude Code writes ✅ all tests passed, you push without diffing, and the CI fails on the exact same test files. Nothing has changed in the code; what changed was the environment in which the tests actually ran.

The second is more subtle. The transcript contains Tests: 1 failed, 141 passed, in black and white, but Claude Code's final summary still says "everything is green." Here, the text on screen and the exit code the shell returned have drifted apart, and the summarizer leaned on the wrong one.

The third is the one that hides longest. Claude Code reports that a test run "timed out," recovers, runs a narrower subset, sees passed, and concludes that the full suite is healthy. Watch mode is almost always involved.

The surface symptoms vary. The underlying problem is the same one in two flavors: either the Bash exit code does not reflect the real state of the tests, or it does, but Claude Code was set up not to see it.

Here is the mapping from symptom to cause, so you can jump to whichever row sounds familiar.

Cause 1: Shell chaining that quietly swallows failures

The most frequent root cause is a Bash command Claude Code itself composed, where the chaining quietly absorbs failure.

# A shape Claude Code reaches for too eagerly
npm install && npm run build && npm test || echo "test failed but continuing"

A human would catch the bug instantly — of course we want to halt when tests fail. But Claude Code has seen plenty of || true and || echo "continuing" patterns in CI examples, and it will reproduce that shape if you do not push back. The final exit code becomes 0 because echo succeeded, and the model interprets the command as having succeeded.

The fix is to make the exit code explicit, every time:

npm test
TEST_EXIT=$?
echo "TEST_EXIT=$TEST_EXIT"
[ "$TEST_EXIT" -eq 0 ] || exit "$TEST_EXIT"

Once Claude Code sees this pattern used in a project, it tends to imitate it elsewhere in the same session. Adding a line to CLAUDE.md — "after running tests, always confirm TEST_EXIT=$? before reporting" — locks the habit in across sessions.

Cause 2: The runner itself returns exit 0 for a non-success

Even if you tame the shell, the test framework can still hand back a clean exit code in situations that aren't really a "pass."

Jest and Vitest both expose --passWithNoTests, which exits green when zero test files are discovered. Useful in CI bootstrap, dangerous in a working branch. Pair it with a leftover tests/__obsolete__ directory and a misconfigured glob, and you will get a green run with zero tests actually executing.

pytest has the same surface area in another shape — pytest -o "norecursedirs=tests" or an over-aggressive -k filter can result in "no tests collected," reported as success.

go test ./... is normally strict, but -run TestSomeRegex matching nothing still exits 0. I have watched Claude Code, in the middle of a refactor, narrow -run to a specific function name, get zero matches, and report the suite as passing.

The cheapest defense is to bake a sanity check into the run that asserts a lower bound on tests actually executed:

npx vitest run --reporter=json --outputFile=vitest-report.json
COUNT=$(jq '.numTotalTests // 0' vitest-report.json)
[ "$COUNT" -ge 10 ] || { echo "Too few tests executed ($COUNT)"; exit 1; }

The exact threshold doesn't matter; the point is to make "we ran no tests" indistinguishable from "we ran failing tests" as far as the exit code is concerned. Both must now turn the bar red.

Note the --outputFile instead of a | tee. That choice is the entire next section.

The trap that disables your fix: pipes rewrite the exit code

Someone adds the count check above, and the same incident keeps happening. Nine times out of ten, there is a single pipe in the command.

# Looks like careful verification
npx vitest run --reporter=json | jq '.numTotalTests'
echo $?   # → 0, as long as jq succeeded. Even when vitest exited 1.

By default, a pipeline reports the exit status of its last command. Both jq and tee finish cleanly once they have consumed the bytes handed to them. The 1 that vitest returned is dropped at the seam.

Five seconds at a prompt is enough to see it:

$ (exit 1) | cat; echo $?
0
$ set -o pipefail; (exit 1) | cat; echo $?
1

What makes this one nasty is that it wears the costume of a hardened command. A || true looks suspicious the moment you read it. A | tee report.json looks like diligence. I spent several days satisfied that I had added a count check, never suspecting that the line was protecting nothing at all. Tracing the run back to PIPESTATUS was not a comfortable afternoon.

There are two ways out. The first is to stop piping — most runners can write the report themselves.

# vitest / jest
npx vitest run --reporter=json --outputFile=vitest-report.json
npx jest --ci --json --outputFile=jest-report.json

The second is to reach for set -o pipefail or PIPESTATUS when a pipe is genuinely required.

set -o pipefail
npx vitest run --reporter=json | tee vitest-report.json
TEST_EXIT=$?
 
# Without pipefail, read each stage's status directly
npx vitest run --reporter=json | tee vitest-report.json
echo "vitest=${PIPESTATUS[0]} tee=${PIPESTATUS[1]}"

set -o pipefail is a shell option, and you cannot assume it survives from one tool invocation to the next. Put it at the head of the same command as the test run, or bake it into scripts.test as bash -o pipefail -c '...'. And remember that PIPESTATUS is a bash array — it is unavailable in CI steps that run under sh or dash.

Cause 3: Watch mode lets Claude Code read only the latest line

This bit me twice on a React Native side project. Leaving npx jest --watch running while letting Claude Code edit code means Jest re-runs on every save. The output ends up looking like:

144 passed, 1 failed
...
145 passed (filter changed)

Claude Code, drawn by the chronologically latest line, takes 145 passed as the source of truth. In reality, that one failing test was simply no longer in scope for the next watch cycle.

The rule is simple: do not give Claude Code commands that can run forever.

# Watch mode is for humans, not for Claude Code
npx jest --watch              # ❌
# Always use a one-shot, deterministic command
npx jest --ci --reporters=default --reporters=summary  # ✅

The --ci flag, in particular, suppresses interactive behavior and ensures the exit code reflects the real run. Adding it removes nearly all of the "latest line wins" failures I used to hit.

Cause 4: Environment differences flip the script

Sometimes the same command behaves differently locally and in CI because the scripts.test entry is itself environment-aware:

{
  "scripts": {
    "test": "if [ \"$CI\" = \"true\" ]; then vitest run --coverage; else vitest run; fi"
  }
}

In CI, coverage thresholds in vitest.config.ts apply, and dipping below 80% lines fails the run. Locally, you only see the test output and pass without ever triggering the coverage gate. Claude Code reports a clean run; CI disagrees.

This is a project-config problem more than a Claude Code problem, but you can dramatically narrow the gap by training Claude Code to run with CI=true:

# Recommended: match the CI execution environment locally
CI=true npm test

One line in CLAUDE.md ("always run tests with CI=true") makes Claude Code's local runs reproduce the same gates CI applies.

A three-layer defense

None of the four causes above can be eliminated by Claude Code alone. Combined, however, they tend to disappear under a three-layer defense.

Layer 1 — explicit rules in CLAUDE.md. Three lines are usually enough: always run tests with CI=true, never use watch mode in agent sessions, and always capture TEST_EXIT=$? immediately after the test command. Claude Code reads CLAUDE.md heavily, and these rules then carry across sessions without you re-stating them.

Layer 2 — a PostToolUse hook that forces the outcome into the summary.

This is where I first got the mechanics wrong. A hook does not receive the tool result through environment variables; it receives a JSON payload on stdin. tool_response carries the command's stdout and stderr, but you cannot count on reading a clean exit code from it. So let the test command record the outcome, and let the hook do nothing but read it back.

# Called from scripts.test in package.json
set -o pipefail
npx vitest run --reporter=json --outputFile=vitest-report.json
TEST_EXIT=$?
COUNT=$(jq '.numTotalTests // 0' vitest-report.json)
mkdir -p .claude && echo "$TEST_EXIT $COUNT" > .claude/last-test-exit
[ "$TEST_EXIT" -eq 0 ] || exit "$TEST_EXIT"

#!/usr/bin/env bash
# .claude/scripts/assert-test-exit.sh
SENTINEL=".claude/last-test-exit"
[ -f "$SENTINEL" ] || exit 0
read -r TEST_EXIT COUNT < "$SENTINEL"
rm -f "$SENTINEL"
 
if [ "$TEST_EXIT" -ne 0 ]; then
  echo "The test run failed (exit=$TEST_EXIT). Do not summarize it as a success." >&2
  exit 2
fi
if [ "${COUNT:-0}" -lt 10 ]; then
  echo "Only ${COUNT} tests executed, below the expected floor." >&2
  exit 2
fi

// .claude/settings.json
{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": ".claude/scripts/assert-test-exit.sh" }
        ]
      }
    ]
  }
}

The load-bearing detail is exit 2. When a PostToolUse hook exits with 2, whatever it wrote to stderr is fed back to Claude Code — right before the summary is composed. The model would have to actively contradict a sentence that says "the test run failed" to call it green. A hook that merely echoes and exits 0 produces none of that. I lost half a day to that distinction.

Layer 3 — CI that matches what Claude Code runs. Reuse the same command shape in GitHub Actions, with the same CI=true and the same exit-code check. A failure mode that only manifests in CI is much easier to debug if CI is running the same script as the agent.

# .github/workflows/test.yml (excerpt)
- name: Run tests with the same command as Claude Code
  env:
    CI: "true"
  run: |
    npm test
    TEST_EXIT=$?
    [ "$TEST_EXIT" -eq 0 ] || exit "$TEST_EXIT"

A real incident from my own repository

On one of the apps I run solo, I once let Claude Code push a change after a clean "all tests passed" report. CI immediately blocked the next App Store upload. The cause turned out to be a vitest --testPathPattern 'unit/' line I had left in CLAUDE.md during a refactor. A large block of black-box tests was being silently filtered out, so passed: 23 was technically accurate — just not the whole story.

I now keep this single line in CLAUDE.md:

When running tests, always run npx vitest run --reporter=json | jq .numTotalTests and confirm the number is at or above the expected floor.

One line, but the same class of incident has not recurred. Claude Code follows written rules with surprising fidelity, so each time you fall into a hole, leave a marker at the edge for your future self.

Wrap-up — distrust the summary, not the agent

When Claude Code reports "all tests passed," the right thing to doubt is not the model — it is the chain of exit codes between the test runner and the shell. Strip out || true, treat --passWithNoTests as a sharp tool, and keep watch mode strictly for humans. Three or four small changes are enough to stop the cycle of green-locally-red-in-CI almost entirely.

Next time you hand a test run to Claude Code, open package.json first and read scripts.test once. If you find || true or --passWithNoTests, that is a landmine waiting for your future self. Defusing it costs nothing today and makes the collaboration with Claude Code dramatically safer tomorrow.

I hope this saves someone else the round-trip I lost to it. Thanks for reading.