Triaging iOS Test Failures Fast: Parsing .xcresult with Claude Code

Shape only the failures into minimal JSON

Claude Code can read long JSON, but handing it the whole tree spends tokens on tests that passed. I convert the result into an array of only the failures, narrowed to test name, failure message, and file/line, before handing it over. That saves tokens and, just as importantly, focuses Claude Code's attention on the part that actually matters.

This jq walks the test-results tests tree recursively and keeps only the Failed leaves.

#!/usr/bin/env bash
# extract-failures.sh — emit only failed tests from a .xcresult as minimal JSON
set -euo pipefail
 
XCRESULT="${1:?usage: extract-failures.sh <path-to-.xcresult>}"
 
xcrun xcresulttool get test-results tests --path "$XCRESULT" \
  | jq '
    # Flatten the test node tree recursively
    def leaves:
      if has("children") and (.children | length > 0)
      then .children[] | leaves
      else . end;
    [ .testNodes[]? | leaves ]
    | map(select(.result == "Failed"))
    | map({
        name: .name,
        identifier: (.nodeIdentifier // .name),
        # Aggregate the failure detail nodes (message / source line)
        failures: [ (.children[]? | select(.nodeType == "Failure Message")
                     | .name) ]
      })
  '

The actual key names in test-results tests shift a little across Xcode point releases. On my machine (Xcode 16.x) the failure detail sat in a child node whose nodeType was Failure Message, but some setups put result at the top. So I run the raw output through jq 'keys' or jq '.testNodes[0]' first, look at the structure, then adjust the extraction. Hard-coding this means an empty array silently appears the day you update Xcode, and you won't notice.

The script output ends up as a short JSON like this:

[
  {
    "name": "testPurchaseRestoresEntitlement()",
    "identifier": "StoreKitTests/testPurchaseRestoresEntitlement()",
    "failures": [
      "XCTAssertEqual failed: (\"none\") is not equal to (\"pro\") - entitlement is not pro after restore (StoreKitTests.swift:84)"
    ]
  }
]

At this size, what reaches Claude Code is a few hundred tokens. With 200 passing tests and 3 failures, only the 3 are sent.

A slash command that returns cause and fix

I hand the shaped failure JSON to Claude Code and ask for a cause hypothesis and a candidate fix. Typing the same instructions every time got old, so I keep it as a slash command under .claude/commands/ in the project.

---
description: Analyze failed tests in a .xcresult and propose causes and fixes
allowed-tools: Bash(./scripts/extract-failures.sh:*), Read, Grep
---
 
Analyze the most recent test failures.
 
First, fetch the failures:
 
!`./scripts/extract-failures.sh "$1"`
 
For each failure in the JSON above, work through these in order.
 
1. Summarize in one line what was expected vs what was observed
2. Using the file/line, open the test and the implementation it exercises with Read
3. Judge whether each failure is "likely a flake" or "likely a real bug" and state why
4. Only for likely real bugs, propose a minimal fix as a diff
 
Label any uncertain guess as a "hypothesis" and do not assert it as fact.

$1 is the path to the .xcresult. Lines starting with ! tell Claude Code to run the command and embed its output in the prompt. Then /triage-tests ./build/Result.xcresult gives you the summary through the fix in one shot.

The part I care about most is forcing the flake-vs-real-bug split first. StoreKit tests occasionally fail on sandbox response latency, and chasing that as a real bug risks breaking code that was correct. Adding this judgment visibly cut the rework I did before applying a fix.

Auto-triage when CI fails

Locally you can run the slash command by hand, but when CI (Xcode Cloud or GitHub Actions) fails, nobody's there. I always pass -resultBundlePath to the CI test step so the .xcresult is reliably kept, and run a headless Claude Code triage only on failure.

# CI test run, with a fixed result-bundle path
xcodebuild test \
  -scheme MyWallpaperApp \
  -destination 'platform=iOS Simulator,name=iPhone 16,OS=latest' \
  -resultBundlePath "$PWD/build/Result.xcresult" \
  | xcbeautify
 
# Only triage if the test step just failed
if [ "${PIPESTATUS[0]}" -ne 0 ]; then
  ./scripts/extract-failures.sh "$PWD/build/Result.xcresult" > build/failures.json
  claude -p "For each failure in build/failures.json, judge flake vs real bug and summarize" \
    --allowedTools "Read,Grep" \
    > build/triage.md
fi

claude -p is Claude Code's non-interactive (headless) mode. Drop the output into triage.md and you can attach it as a CI artifact or post it to Slack. In the morning, one notification tells you "two failures overnight: one flake, one real bug in restore."

To go further, a Claude Code Stop hook can analyze the .xcresult automatically whenever a local test agent stops. Hooks live in ~/.claude/settings.json or the project's .claude/settings.json.

{
  "hooks": {
    "Stop": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "test -f build/Result.xcresult && ./scripts/extract-failures.sh build/Result.xcresult > build/failures.json || true"
          }
        ]
      }
    ]
  }
}

This hook only extracts failures to a file on each stop. Stuffing heavy analysis into a hook slows the stop down, so I keep extraction in the hook and run the analysis when I actually want to read triage.md.

Where this trips you up, and how I avoid it

A few things that cost me time in real use — each one hard to see once you're stuck.

First, pointing -resultBundlePath at an existing path errors out. A .xcresult can't be overwritten, so a leftover at the same path makes xcodebuild fail. In CI I always rm -rf build/Result.xcresult before the test run. Locally, I've spun my wheels expecting a fresh result while the old one was still sitting there.

Second, running multiple -destinations in parallel bundles several test runs inside one .xcresult. test-results summary sums everything, but to know which simulator failed you have to read the device info on the tests nodes. In CI I pin a single destination first for reproducibility, then parallelize once it's stable.

Third, writing the jq against a hard-coded shape breaks silently on an Xcode point update. As above, looking at the raw JSON before writing the script is, in the end, the fastest path — that's my conclusion after many update cycles as a solo developer. I leave a comment at the top of extract-failures.sh reminding me that zero failures might mean "everything passed" or "the extraction broke."

Fourth, skipping token reduction lets attachment metadata leak into the JSON and inflate cost. A .xcresult can carry UI-test screenshots, and their references are nearly useless for failure analysis. Narrowing to failure-node text, as this shaping does, saves both tokens and attention.

One next step

Run xcrun xcresulttool get test-results tests --path <your.xcresult> once on a currently failing test and look at the structure yourself. Then tweak the key names in extract-failures.sh to match your Xcode, and failure analysis becomes a single command from that day on. Seeing the structure once is the insurance that lets you notice fast when a future update breaks it.

If you're running several apps solo too, I hope this saves you an afternoon. Thanks for reading.