Claude Code Says 'All Tests Pass' but CI Goes Red — Designing Around Exit Code Pitfalls

You ask Claude Code to verify a change, get back a confident "all tests pass," and push the branch. A few minutes later, GitHub Actions paints the dashboard red. In twelve years of solo app development, this is the single category of incident that grew the fastest after I started leaning on Claude Code.

The strange part is that Claude Code is not lying. If you scroll back through the transcript, the model genuinely read something like pass: 142 in the output. What broke is one layer below — the way the shell and the test runner reported the real outcome, and whether that signal ever reached Claude Code's summarizer.

This article walks through the four most common patterns I — Masaki Hirokawa, an indie developer running AdMob-monetized apps with 50M+ cumulative downloads since 2014 — have hit using Vitest, Jest, pytest, and go test, and then assembles a three-layer defense that, in practice, drives the recurrence rate to essentially zero.

The symptom — three ways "tests passed" becomes a red CI

You usually notice this in one of three shapes.

The first is the cleanest: Claude Code writes ✅ all tests passed, you push without diffing, and the CI fails on the exact same test files. Nothing has changed in the code; what changed was the environment in which the tests actually ran.

The second is more subtle. The transcript contains Tests: 1 failed, 141 passed, in black and white, but Claude Code's final summary still says "everything is green." Here, the text on screen and the exit code the shell returned have drifted apart, and the summarizer leaned on the wrong one.

The third is the one that hides longest. Claude Code reports that a test run "timed out," recovers, runs a narrower subset, sees passed, and concludes that the full suite is healthy. Watch mode is almost always involved.

The surface symptoms vary. The underlying problem is the same one in two flavors: either the Bash exit code does not reflect the real state of the tests, or it does, but Claude Code was set up not to see it.

Cause 1: Shell chaining that quietly swallows failures

The most frequent root cause is a Bash command Claude Code itself composed, where the chaining quietly absorbs failure.

# A shape Claude Code reaches for too eagerly
npm install && npm run build && npm test || echo "test failed but continuing"

A human would catch the bug instantly — of course we want to halt when tests fail. But Claude Code has seen plenty of || true and || echo "continuing" patterns in CI examples, and it will reproduce that shape if you do not push back. The final exit code becomes 0 because echo succeeded, and the model interprets the command as having succeeded.

The fix is to make the exit code explicit, every time:

npm test
TEST_EXIT=$?
echo "TEST_EXIT=$TEST_EXIT"
[ "$TEST_EXIT" -eq 0 ] || exit "$TEST_EXIT"

Once Claude Code sees this pattern used in a project, it tends to imitate it elsewhere in the same session. Adding a line to CLAUDE.md — "after running tests, always confirm TEST_EXIT=$? before reporting" — locks the habit in across sessions.

Cause 2: The runner itself returns exit 0 for a non-success

Even if you tame the shell, the test framework can still hand back a clean exit code in situations that aren't really a "pass."

Jest and Vitest both expose --passWithNoTests, which exits green when zero test files are discovered. Useful in CI bootstrap, dangerous in a working branch. Pair it with a leftover tests/__obsolete__ directory and a misconfigured glob, and you will get a green run with zero tests actually executing.

pytest has the same surface area in another shape — pytest -o "norecursedirs=tests" or an over-aggressive -k filter can result in "no tests collected," reported as success.

go test ./... is normally strict, but -run TestSomeRegex matching nothing still exits 0. I have watched Claude Code, in the middle of a refactor, narrow -run to a specific function name, get zero matches, and report the suite as passing.

The cheapest defense is to bake a sanity check into the run that asserts a lower bound on tests actually executed:

npx vitest run --reporter=json | tee vitest-report.json
COUNT=$(jq '.numTotalTests' vitest-report.json)
[ "$COUNT" -ge 10 ] || { echo "Too few tests executed ($COUNT)"; exit 1; }

The exact threshold doesn't matter; the point is to make "we ran no tests" indistinguishable from "we ran failing tests" as far as the exit code is concerned. Both must now turn the bar red.

Cause 3: Watch mode lets Claude Code read only the latest line

This bit me twice on a React Native side project. Leaving npx jest --watch running while letting Claude Code edit code means Jest re-runs on every save. The output ends up looking like:

144 passed, 1 failed
...
145 passed (filter changed)

Claude Code, drawn by the chronologically latest line, takes 145 passed as the source of truth. In reality, that one failing test was simply no longer in scope for the next watch cycle.

The rule is simple: do not give Claude Code commands that can run forever.

# Watch mode is for humans, not for Claude Code
npx jest --watch              # ❌
# Always use a one-shot, deterministic command
npx jest --ci --reporters=default --reporters=summary  # ✅

The --ci flag, in particular, suppresses interactive behavior and ensures the exit code reflects the real run. Adding it removes nearly all of the "latest line wins" failures I used to hit.

Cause 4: Environment differences flip the script

Sometimes the same command behaves differently locally and in CI because the scripts.test entry is itself environment-aware:

{
  "scripts": {
    "test": "if [ \"$CI\" = \"true\" ]; then vitest run --coverage; else vitest run; fi"
  }
}

In CI, coverage thresholds in vitest.config.ts apply, and dipping below 80% lines fails the run. Locally, you only see the test output and pass without ever triggering the coverage gate. Claude Code reports a clean run; CI disagrees.

This is a project-config problem more than a Claude Code problem, but you can dramatically narrow the gap by training Claude Code to run with CI=true:

# Recommended: match the CI execution environment locally
CI=true npm test

One line in CLAUDE.md ("always run tests with CI=true") makes Claude Code's local runs reproduce the same gates CI applies.

A three-layer defense

None of the four causes above can be eliminated by Claude Code alone. Combined, however, they tend to disappear under a three-layer defense.

Layer 1 — explicit rules in CLAUDE.md. Three lines are usually enough: always run tests with CI=true, never use watch mode in agent sessions, and always capture TEST_EXIT=$? immediately after the test command. Claude Code reads CLAUDE.md heavily, and these rules then carry across sessions without you re-stating them.

Layer 2 — a PostToolUse hook that forces the exit code into the summary.

// .claude/hooks.jsonc
{
  "PostToolUse": [
    {
      "matcher": "Bash",
      "command": "test \"$CLAUDE_TOOL_EXIT_CODE\" -eq 0 || echo \"⚠️ Bash exit=$CLAUDE_TOOL_EXIT_CODE — this run failed. Include in the summary.\""
    }
  ]
}

With this hook in place, a non-zero exit code is injected as a clearly worded warning just before Claude Code summarizes. The summarizer would have to actively ignore the warning to call the run a success.

Layer 3 — CI that matches what Claude Code runs. Reuse the same command shape in GitHub Actions, with the same CI=true and the same exit-code check. A failure mode that only manifests in CI is much easier to debug if CI is running the same script as the agent.

# .github/workflows/test.yml (excerpt)
- name: Run tests with the same command as Claude Code
  env:
    CI: "true"
  run: |
    npm test
    TEST_EXIT=$?
    [ "$TEST_EXIT" -eq 0 ] || exit "$TEST_EXIT"

A real incident from my own repository

On one of the apps I run solo, I once let Claude Code push a change after a clean "all tests passed" report. CI immediately blocked the next store upload. The cause turned out to be a vitest --testPathPattern 'unit/' line I had left in CLAUDE.md during a refactor. A large block of black-box tests was being silently filtered out, so passed: 23 was technically accurate — just not the whole story.

I now keep this single line in CLAUDE.md:

When running tests, always run npx vitest run --reporter=json | jq .numTotalTests and confirm the number is at or above the expected floor.

One line, but the same class of incident has not recurred. Claude Code follows written rules with surprising fidelity, so each time you fall into a hole, leave a marker at the edge for your future self.

Wrap-up — distrust the summary, not the agent

When Claude Code reports "all tests passed," the right thing to doubt is not the model — it is the chain of exit codes between the test runner and the shell. Strip out || true, treat --passWithNoTests as a sharp tool, and keep watch mode strictly for humans. Three or four small changes are enough to stop the cycle of green-locally-red-in-CI almost entirely.

Next time you hand a test run to Claude Code, open package.json first and read scripts.test once. If you find || true or --passWithNoTests, that is a landmine waiting for your future self. Defusing it costs nothing today and makes the collaboration with Claude Code dramatically safer tomorrow.

I hope this saves someone else the round-trip I lost to it. Thanks for reading.