What you keep the agent from walking matters more than what you hand it

The default search was skipping exactly what I most wanted to see

Here is where those 136 files were sitting.

By default ripgrep honors .gitignore and skips dot-prefixed files. That is the whole story behind its speed, and for everyday code search it is a kind and sensible default.

Auditing a folder you handed to an agent inverts it. The first things skipped were the agent's own memory and the skill definitions the agent reads. The places I most wanted eyes on were the places most reliably out of frame.

A fast search isn't fast. It just isn't looking.

Until I ran this, I had assumed a larger result count meant broader coverage. In fact, of the two searches that finished, the one with fewer results was simply skipping more.

Six half-deleted temp files (the .fuse_hidden kind, left behind when an editor still holds a file that has been removed) also surfaced for the first time this way. Neither category had ever appeared in a default search.

Measure the scope once, before you hand the folder over

This is not a check worth repeating daily. I wanted something to run once, when a folder is connected or reorganized, that draws the map. Here is the version I actually use.

#!/usr/bin/env bash
# scope-probe.sh — measure a folder's scan scope once, before an agent walks it.
# usage: ./scope-probe.sh <folder> [pattern]
set -uo pipefail
 
ROOT="${1:?usage: scope-probe.sh <folder> [pattern]}"
PAT="${2:-TODO}"
SKIP=(node_modules .git .next .venv dist build vendor __pycache__)
 
cd "$ROOT" || exit 1
 
# --- 1. How many files is anything going to have to walk? (one find pass only) ---
find . -type f -print0 2>/dev/null | awk -v RS='\0' -v skip="${SKIP[*]}" '
  BEGIN { n = split(skip, s, " ") }
  { total++
    hit = 0
    for (i = 1; i <= n; i++) if (index($0, "/" s[i] "/")) { c[s[i]]++; hit = 1; break }
    if (!hit) work++ }
  END {
    printf "  total       %8d\n", total
    for (i = 1; i <= n; i++) if (c[s[i]]) printf "  %-10s  %8d  (%4.1f%%)\n", s[i], c[s[i]], c[s[i]]*100/total
    printf "  real work   %8d  (%4.1f%%)\n", work, work*100/total }'
 
# --- 2. Does the count change between the default search and an explicit scope? ---
GLOBS=(); for d in "${SKIP[@]}"; do GLOBS+=(-g "!**/$d/**"); done
 
t0=$(date +%s%3N); rg -l --no-ignore --hidden "${GLOBS[@]}" -- "$PAT" . 2>/dev/null | sort > /tmp/scope_full.txt; t1=$(date +%s%3N)
t2=$(date +%s%3N); rg -l -- "$PAT" . 2>/dev/null | sort > /tmp/scope_default.txt; t3=$(date +%s%3N)
 
echo "  explicit scope  $(wc -l < /tmp/scope_full.txt) files  $(( t1 - t0 )) ms"
echo "  default         $(wc -l < /tmp/scope_default.txt) files  $(( t3 - t2 )) ms"
 
# --- 3. Show what the default quietly dropped, grouped by where it lives ---
MISS=$(comm -23 /tmp/scope_full.txt /tmp/scope_default.txt | tee /tmp/scope_missed.txt | wc -l)
echo "  missed by default: ${MISS}"
[ "$MISS" -gt 0 ] && awk -F/ '{print "    " $2 "/" $3}' /tmp/scope_missed.txt \
  | sort | uniq -c | sort -rn | head -5
 
# Zero missed is the healthy state. Anything else belongs in your scan design.
[ "$MISS" -eq 0 ]

Running it against my folder:

$ ./scope-probe.sh "/path/to/workspace" premium
  total         103845
  node_modules    81359  (78.3%)
  .git             7092  ( 6.8%)
  __pycache__        12  ( 0.0%)
  real work       15382  (14.8%)
  explicit scope  4277 files  11132 ms
  default         4142 files  6370 ms
  missed by default: 136
     78     rorklab.net/public
     46     gemilab.net/public
      3     .agents/skills
      2     rorklab.net/src
      2     gemilab.net/src

The single find pass with classification in awk is deliberate: walking is the expensive part. Calling find once per bucket burns tens of seconds checking how expensive your checking is.

The exit code on the last line lets the probe live inside an unattended check. Zero missed returns 0; anything else returns 1. Wired into CI or a PostToolUse hook, it tells you on the day a folder's shape changes.

Longer exclude lists do not make you safer

Writing the probe surfaced a miss running the other direction. One file was dropped by the explicit scope and present only in the default result.

./_documents/_quality_audit/_scripts/__pycache__/article_gate.py.bak_2026-09-02

That is my own doing: I had added __pycache__ to the exclude list. A dated backup of a real script was sitting inside it. "Python puts generated files there, so I don't need to look" pushed a copy a human had placed there out of frame along with everything else.

Every exclusion buys speed and sells a blind spot. Each one you add deserves a single look for whether a person has left something in that directory. Skipping that look is precisely where the hole opens.

For a while I believed a longer exclude list made a scan smarter. That did not go well. Now, when I add an exclusion, I diff the two result sets with comm on the spot.

Declare the scope on the command, not in an ignore file

Putting the rules in .gitignore or .rgignore looks tidier, but there is a catch: --no-ignore disables .gitignore, .ignore, and .rgignore all at once. I checked this directly.

# with "sub/" listed in .rgignore
$ rg -l needle .
./top.txt                 # sub/ is skipped
 
$ rg -l --no-ignore needle .
./sub/a.txt               # .rgignore is disabled too, so it shows up
./top.txt

So "write the scope into an ignore file" and "use --no-ignore to see everything" cannot be combined. You pick one.

I lean toward the second. In an unattended task I'd rather have the scope written on the command than have results bend to whatever each repository happens to keep in its .gitignore. It makes the output easier to read six hours later.

# One fixed scan scope, called the same way by hand and by unattended tasks.
scan() {  # scan <pattern> [path]
  rg -l --no-ignore --hidden \
     -g '!**/node_modules/**' -g '!**/.git/**' -g '!**/.next/**' \
     -- "$1" "${2:-.}"
}

--hidden is the part not to drop. Without it, dot-prefixed directories — where an agent's configuration and memory live — stay out of frame even with --no-ignore set.

Three traps that remain

First, in a cloud-synced folder, files that have not been materialized yet slip past a bash-side search without a sound. Declaring your scope correctly does not help here; it needs separate handling, which I wrote up in Why Cowork's bash says the file is missing while Finder shows it just fine.

Second, the search with the most hits is not the correct one. The third row above — --no-ignore with no scope declared — was worst on both count and time, because removing the ignore rules without naming a scope sends it straight into node_modules until the clock runs out.

Third, the walking budget is part of the design. An unattended run has a finite amount of time, and two heavy searches can exhaust it on their own. In my check tasks I allow exactly one explicitly scoped search. For the same reason I stopped letting two tasks touch one folder at once, which I worked through in The lock I left in a shared folder shut out every run after the first.

Narrowing what you hand over in the first place is the other lever, and keys are the fastest way into that question — I counted mine in I counted the keys in my connected folder: filenames found 5 of the 18.

One next step

Run scope-probe.sh once against the folder you have connected right now. The only line worth reading is the last one: how many files the default search missed.

Zero means the defaults are fine for that folder. Anything above zero is the part of it your unattended tasks have never seen. Mine came back with 136, and the agent's own memory was inside them.

Measure once, on the day you connect it. That alone removes one morning of failing quietly.