Grep's First 250 Lines Touched Only 42 of the 183 Files That Matched

Result 2: the amount moves a lot with how you ask

Same hook, different output shapes:

I had expected --max-columns to be the big lever, assuming long lines were what inflated the output. It cut only 2%.

The reason is line count and the line prefix, not line length. The body of a matched line has a median of 58 characters, a 90th percentile of 167, and a maximum of 517. Prose lines in this repository are short. Meanwhile, the content/articles/ja/claude-code/…mdx:123: prefix on each line adds up to about 112,000 characters, so roughly 37% of the output was paths and line numbers.

Switching to -o with 40 characters either side trims about 20%, but each match gets its own line, so the line count rose from 1,312 to 1,433.

Result 3: the first 250 lines reach less than a quarter of the matching files

Now the main point. In my environment, the Grep tool's description says that content mode cuts off at 250 lines unless head_limit is set (the default output_mode is files_with_matches, so the flood happens when content mode is chosen). Please check your own environment's tool descriptions for its limit.

So of the 1,312 lines, how many of the 183 files do the first 250 touch?

• Files touched by the first 250 lines: 42 (23%)
• Bytes in those 250 lines: 49,172 (16% of the full output)
• The top 3 files alone account for 217 lines (17%): 88, 76, and 53 lines
• 61 files matched exactly once; 67 files matched five or more times

In other words, most of the 250 lines are filled by a few articles that mention the word again and again, and most of the 61 files that mention it once never come into view.

What you see also changed from run to run. ripgrep runs in parallel, so the order is not fixed. Six runs of the same command touched between 32 and 51 files in the first 250 lines, and the category mix moved around.

import subprocess
from collections import Counter
 
def head_files(cmd, limit=250):
    out = subprocess.run(cmd, shell=True, capture_output=True).stdout.decode()
    top = out.splitlines()[:limit]
    files = {l.split(":", 1)[0] for l in top}
    # count the category part of content/articles/ja/<category>/<slug>.mdx
    return len(files), Counter(f.split("/")[3] for f in files)
 
for i in range(6):
    n, cats = head_files("rg -n -i hook content/articles/ja")
    print(i + 1, n, dict(sorted(cats.items())))
 
n, cats = head_files("rg -n --sort path -i hook content/articles/ja")
print("sorted", n, dict(sorted(cats.items())))

An excerpt of the actual output, four of the six runs plus the path-sorted run:

The last row was the counterintuitive one. Not a single claude-code article came into view. The category with the most articles about hooks is claude-code, yet cutting the first 250 lines in path order shows only api-sdk and claude-ai, which sort earlier alphabetically.

If an answer came back saying "this is mostly covered in the api-sdk articles," the search would not have been wrong. The way the visible range was cut would have decided the answer. And nothing in the wording of the answer shows that a cut happened.

Estimating before you read: scout_hits.py

So I wrote a script that counts, before reading everything, how much will be cut. The problem it solves: it reports the match count, file count, and first-N reach in tens of milliseconds.

#!/usr/bin/env python3
"""scout_hits.py — estimate hit count, file count and misses before reading all of rg's output.
Usage: python3 scout_hits.py PATTERN [PATH] [--limit 250]
"""
import subprocess
import sys
from collections import Counter
 
 
def rg(args):
    # rg exits 1 when nothing matches; only treat exit 2 as a real failure
    p = subprocess.run(["rg", *args], capture_output=True)
    if p.returncode == 2:
        sys.exit(p.stderr.decode(errors="replace"))
    return p.stdout.decode(errors="replace").splitlines()
 
 
def main():
    argv = sys.argv[1:]
    limit = 250
    if "--limit" in argv:
        i = argv.index("--limit")
        limit = int(argv[i + 1])
        del argv[i:i + 2]
    pattern = argv[0]
    path = argv[1] if len(argv) > 1 else "."
 
    lines = rg(["-n", "--sort", "path", "-i", pattern, path])
    if not lines:
        print("no matches")
        return
    per_file = Counter(l.split(":", 1)[0] for l in lines)
    total_bytes = sum(len(l.encode()) + 1 for l in lines)
    head = lines[:limit]
    seen = {l.split(":", 1)[0] for l in head}
    first_m1 = rg(["-n", "-m1", "-i", pattern, path])
    m1_bytes = sum(len(l.encode()) + 1 for l in first_m1)
 
    print(f"{len(lines)} lines / {len(per_file)} files / {total_bytes:,} bytes in full")
    print(f"first {limit} lines touch: {len(seen)} / {len(per_file)} files "
          f"({len(seen) / len(per_file):.0%})")
    top3 = per_file.most_common(3)
    print("top 3 files hold: {} lines ({:.0%})".format(
        sum(n for _, n in top3), sum(n for _, n in top3) / len(lines)))
    print(f"-m1 reaches every file in {len(first_m1)} lines / {m1_bytes:,} bytes")
    verdict = "start with -l or -m1" if len(lines) > limit else "fine to read in full"
    print("verdict:", verdict)
 
 
if __name__ == "__main__":
    main()

The result for hook:

1312 lines / 183 files / 305,309 bytes in full
first 250 lines touch: 42 / 183 files (23%)
top 3 files hold: 217 lines (17%)
-m1 reaches every file in 183 lines / 49,494 bytes
verdict: start with -l or -m1

For a narrower word, PreToolUse, the picture changes:

141 lines / 32 files / 38,516 bytes in full
first 250 lines touch: 32 / 32 files (100%)
top 3 files hold: 48 lines (34%)
-m1 reaches every file in 32 lines / 9,049 bytes
verdict: fine to read in full

The hook run took about 64 ms. That is light enough for a preliminary count, and it makes the call "if it fits in 250 lines, just read it all" cheap to make.

Why count with --sort path? Without a fixed order the numbers shift every time you estimate. The real search keeps shifting, but I prefer to keep the estimate as a reproducible number. That is simply my preference.

The order I chose: -l, then -m1, then narrow down

Based on the measurements, here is the order I settled on:

1. File names only (-l): 15 KB and 183 lines give the whole picture, including which categories the hits are scattered across
2. One hit per file (-m1): 49 KB touches all 183 files and shows briefly in what context each file uses the word
3. Scope down, then read everything: narrow to the files or folders that caught your eye in step 2, then read all hits with -n

For step 3, it is easier to have Claude do the narrowing than to do it by hand. In CLAUDE.md I write something like this:

## How to search
- For a term likely to match a lot, do not print contents first. Get the file list (files_with_matches) to understand the range
- When viewing contents, start with one hit per file (the -m1 equivalent) and read everything only for the files that need it
- If a search was cut off at the first N lines, say in the answer that the result is partial

The third line exists to leave a trace of the cutoff in the answer. As we saw, the wording of an answer does not show that a cutoff happened.

Pitfalls

Things that tripped me up while measuring:

• --max-columns barely reduces volume: in a repository of short prose lines it saved only 2%. It matters more where logs or minified files dominate
• -o adds lines: when one line has several matches, each match gets its own line (1,312 became 1,433)
• The prefix is about 37% of the volume: the deeper the paths, the more bytes go to showing where rather than what. That is why -l is so light
• git grep does not see untracked files: here everything was tracked, so the outputs matched, but a new article in progress would be outside its range. The same scope worry appears in taking stock of what Cowork does not read
• ripgrep respects .gitignore: when a search returns nothing, the problem runs the other way. I cover that in what to check when Glob and Grep return nothing

What I did not measure

Some limits I should be upfront about:

• I did not measure tokens. Japanese and English differ in tokens per character, so the byte ratio is not the token ratio
• I did not observe which command Claude actually picks when it searches. The tables are from running ripgrep directly
• The target is one repository of Japanese prose. A source-code-heavy repository should differ in both line length and hit distribution

If your numbers come out different, I would be glad to hear about it.

A next step

If you want to review context use more broadly, optimizing the context budget is a good companion.

The smallest steps you can take today:

1. Pick one term you often search for in your repository, run it through scout_hits.py, and see whether the verdict reads "start with -l or -m1"
2. For a term that gets that verdict, try -l then -m1 once, and compare what you see with the first 250 lines

Choosing one search angle with care reduces misses more than competing on search speed. That is what I took away from this measurement.