Until I planted a failing sample, my unattended checks had never once failed

What I measured, one call shape at a time

Guessing at a fix would only move the problem somewhere else. So I wrote a deliberately failing check and varied only the shape of the call.

# gate.py — a dummy check that always reports a violation and exits 1
import sys
 
print("VIOLATION: 3 files failed", file=sys.stderr)
print("detail line 1")
print("detail line 2")
sys.exit(1)

Called from Bash 5.1, here is what came back.

The tee row is the one that ran against my expectations. I had added it purely to keep a log, with no intention of changing the output, so I assumed it left the status alone too. Being the rightmost element of the pipeline was reason enough for it to overwrite the failure on its left.

One local swallows a whole function's failure

Of the three, local was the hardest to spot. It appears while you are tidying code into functions, so it slips in as a side effect of refactoring.

# The shape that loses the failure
run_gate_bad() {
  local out=$(python3 gate.py 2>&1)   # $? is local's status (always 0)
  return $?
}
run_gate_bad; echo "exit=$?"          # → exit=0
 
# The shape that keeps it (split declaration from assignment)
run_gate_good() {
  local out
  out=$(python3 gate.py 2>&1)         # $? is python3's status
  return $?
}
run_gate_good; echo "exit=$?"         # → exit=1

The fix is two lines instead of one. shellcheck flags this pattern as SC2155, so running it once on the day you write a check surfaces the problem right there. I make a point of running shellcheck whenever I add a new check, and only then.

Where set -e cannot help you

For a while I believed that putting set -euo pipefail at the top made everything safe. In practice there are several contexts where set -e steps aside.

( set -e; python3 gate.py >/dev/null 2>&1; echo "never reached" )
# → the subshell ends with exit=1 (correct behaviour)
 
( set -e; python3 gate.py >/dev/null 2>&1 || echo "gate failed"; echo "reached anyway" )
# → both messages print, and the subshell exits 0

The moment you write the right-hand side of ||, the shell treats that line as one whose failure has already been handled. The same applies inside an if condition and on the left of &&. I came to read set -e as a guard for lines that are not holding a failure of their own.

If you do hold one, you have to drop it yourself.

if ! python3 gate.py >/dev/null 2>&1; then
  echo "gate failed" >&2
  exit 1                      # if you catch it, you end it
fi

Keeping a poisoned sample so failure is proven every run

Even after finding the cause, I stayed uneasy for a while. Closing three leaks does nothing about the next line I add doing the same thing.

So I changed the question. Instead of a human reading the check to confirm it is correct, the machine confirms every run that the check is still able to fail.

Two files were all it took: a "poisoned" sample containing one known violation, and a clean sample that certainly passes. They sit in a fixed location, and the check is pointed at both of them before it is pointed at anything real. If the poisoned one does not fail, the day's results are treated as void.

# fixtures/bad.txt   … contains exactly one known violation
# fixtures/good.txt  … certainly passes

I keep the samples deliberately minimal. Larger fixtures need maintenance whenever the check's rules change, and they make it harder to isolate why the poison stopped working.

The 40-line harness that checks the checker

Here is the version I keep in place, generalised. It takes a check command and the two samples, and looks at nothing but whether the expected exit codes come back.

#!/usr/bin/env bash
# check_the_checker.sh — proves that the check itself can still fail
# usage: ./check_the_checker.sh <check-command> <poisoned-sample> <clean-sample>
set -uo pipefail          # -e is left out on purpose: we judge the status ourselves
 
GATE=${1:?check command required}
BAD=${2:?poisoned sample required}
GOOD=${3:?clean sample required}
 
fail=0
 
# Discard output, keep only the exit status.
# The point is that no pipe sits here — adding head or tee breaks the measurement itself.
run() {
  "$GATE" "$1" >/dev/null 2>&1
  echo $?
}
 
bad_code=$(run "$BAD")
good_code=$(run "$GOOD")
 
if [ "$bad_code" -eq 0 ]; then
  echo "NG: the check did not fail on a known violation (exit=$bad_code)" >&2
  fail=1
else
  echo "OK: poisoned sample gave exit=$bad_code"
fi
 
if [ "$good_code" -ne 0 ]; then
  echo "NG: the check failed on a clean sample (exit=$good_code)" >&2
  fail=1
else
  echo "OK: clean sample gave exit=$good_code"
fi
 
exit "$fail"

Pointed at a healthy check, it passes like this.

OK: poisoned sample gave exit=1
OK: clean sample gave exit=0
harness exit=0

And pointed at a check broken the same way mine was — with head added for readable logs:

#!/usr/bin/env bash
# reproduction of the broken check (head added to tidy the log)
grep 'FORBIDDEN' "$1" | head -1
exit $?          # returns head's exit code

NG: the check did not fail on a known violation (exit=0)
OK: clean sample gave exit=0
harness exit=1

Three weeks of not noticing, surfaced by two files on the first try. Seeing that output was the moment the weight came off my shoulders.

Where the harness sits matters too. Not immediately before the check, but first thing in the day. Placed right before the check, the harness call can break in exactly the same way and go down with it. Mine runs alone at the start, and nothing else proceeds on a day it comes back red.

The lines I draw now

Three of them.

No pipe on the line that calls a check. Readability is the check's own job — it can format its output itself. When I really do want to pipe, set -o pipefail or ${PIPESTATUS[0]} goes with it, always.

If I catch a failure with ||, I end it myself with exit 1 on the spot. Catching and doing nothing is the habit I try hardest to avoid.

And the poisoned sample stays. The day I add a new check, I feed it the poison first and watch it turn red before pointing it at anything real.

An unattended system is safe to let go of only for as long as it can tell you it broke. Lose that, and what remains is something doing an impression of working. The same blind spot shows up in how often a schedule actually fires, so The scheduled run that silently halved, and the reconciliation that caught it and It exists but you can't write to it — probing capabilities before an unattended run may help you see the shape of it.

Pick one check you run unattended and feed it a single file containing a known violation. If it doesn't go red, the green it has accumulated starts counting from today. That's where I started too.

I'm still finding leaks of my own, and I'd rather compare notes than pretend the list above is complete.