The same skill lived in two places, and only one copy still had its files

Three rewrites took four hits down to two

Wanting to run the same check across all sixty-six, I reached for a one-line grep first.

grep -oE '(references|scripts|assets|evals|templates)/[A-Za-z0-9_./-]+\.(md|py|sh|json|txt|csv)' SKILL.md

Across every skill this reported four missing references. Opening them one by one, two turned out to be wrong.

It had been matching _scripts/ukiyoe_batch.py — a line in an operations skill for the wallpaper apps I build as an indie developer, pointing at a directory outside the repository entirely, not at a bundled file. The pattern was ignoring the leading underscore and matching scripts/.

I added a lookbehind to fix that, excluded fenced code blocks (examples are not necessarily bundled artifacts), and added mjs to the extension list.

The rerun reported four again. Same count, different contents.

✗ antigravitylab-site → scripts/generate-content.mjs
✗ rorklab-site        → scripts/generate-content.mjs
✗ seo-audit           → references/ai-writing-detection.md
✗ skill-creator       → evals/evals.json

The first two came from a build command written into a table inside SKILL.md.

| Build command | `node scripts/generate-content.mjs && ...` |

That invokes a script in the site repository. It is not shipped with the skill. It had simply been out of scope in the previous pass because of the extension list.

Meanwhile, excluding fenced blocks had made the earlier _scripts/ukiyoe_batch.py hit disappear. The count landed on four both times, which means if I had been reading counts instead of contents, I would have concluded it was fixed.

That is the part that stayed with me. The output of a detector is not a number; it is a list you have to open. A matching count is not evidence of correctness.

The third pass added an exclusion for arguments to executable commands. That gave two — the real ones.

Checking that bundled files exist

Here is where the third pass settled. It runs as written.

#!/usr/bin/env python3
"""Verify that the files a SKILL.md names are actually bundled with it (v3).
 
v1 (naive regex) and v2 mistook external paths and command arguments for
bundled references. v3 excludes three cases:
  (1) preceded by _ or /  ... part of another directory (e.g. _scripts/x.py)
  (2) inside a code fence ... an example, not necessarily a bundled file
  (3) argument to a runner ... calls like node scripts/build.mjs
"""
import os, re, sys
 
BUNDLE_DIRS = ("references", "scripts", "assets", "evals", "templates", "examples", "data")
EXTS = ("md", "py", "sh", "json", "txt", "csv", "yaml", "yml", "js", "mjs")
RUNNERS = ("node", "python", "python3", "bash", "sh", "npx", "deno", "ruby", "pnpm", "yarn")
 
REF_RE = re.compile(
    r"(?<![A-Za-z0-9_/-])(?:\./)?(?:" + "|".join(BUNDLE_DIRS)
    + r")/[A-Za-z0-9_./-]+\.(?:" + "|".join(EXTS) + r")"
)
RUNNER_RE = re.compile(r"(?:" + "|".join(RUNNERS) + r")\s+$")
FENCE_RE = re.compile(r"^\s*(?:```|~~~)")
 
 
def extract_refs(text):
    refs, inside = set(), False
    for line in text.splitlines():
        if FENCE_RE.match(line):
            inside = not inside
            continue
        if inside:
            continue
        for m in REF_RE.finditer(line):
            if RUNNER_RE.search(line[: m.start()]):   # (3) argument to a runner
                continue
            refs.add(m.group(0).lstrip("./"))
    return sorted(refs)
 
 
def main(roots):
    skills = refs_total = 0
    missing = []
    for root in roots:
        if not os.path.isdir(root):
            print(f"scan target does not exist: {root}", file=sys.stderr)
            sys.exit(2)          # never let a skipped scan pass as success
        for name in sorted(os.listdir(root)):
            d = os.path.join(root, name)
            md = os.path.join(d, "SKILL.md")
            if not os.path.isfile(md):
                continue
            skills += 1
            with open(md, encoding="utf-8", errors="replace") as f:
                refs = extract_refs(f.read())
            refs_total += len(refs)
            for r in refs:
                if not os.path.isfile(os.path.join(d, r)):
                    missing.append((name, r))
    print(f"skills scanned: {skills} / bundled references: {refs_total}")
    for name, r in missing:
        print(f"  x {name} -> {r}")
    print(f"missing: {len(missing)}")
    return 1 if missing else 0
 
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Against my sixty-six skills:

skills scanned: 66 / bundled references: 51
  x seo-audit -> references/ai-writing-detection.md
  x skill-creator -> evals/evals.json
missing: 2

129 milliseconds. Cheap enough to leave in CI permanently.

The sys.exit(2) is there so a nonexistent scan directory cannot quietly report zero missing files. One mistyped path and the check runs against nothing, then passes. I have made that mistake before, so a skipped scan now gets its own exit code.

$ python3 skill_refs.py .agents/skills /nonexistent/skills
scan target does not exist: /nonexistent/skills
exit=2

Comparing completeness across duplicate names

The second tool pulls out only the pairs that share a name, and puts body equality next to file counts.

#!/usr/bin/env python3
"""Compare completeness when the same skill name exists in several scopes."""
import hashlib, os, sys
from collections import defaultdict
 
 
def digest(path):
    with open(path, "rb") as f:
        return hashlib.sha256(f.read()).hexdigest()[:12]
 
 
def inventory(root):
    out = {}
    for name in sorted(os.listdir(root)):
        d = os.path.join(root, name)
        md = os.path.join(d, "SKILL.md")
        if not os.path.isfile(md):
            continue
        files = [
            os.path.relpath(os.path.join(dp, fn), d)
            for dp, _, fns in os.walk(d)
            for fn in fns
            if not fn.startswith(".")        # keep .DS_Store out of the count
        ]
        out[name] = {"sha": digest(md), "files": sorted(files)}
    return out
 
 
def main(roots):
    seen = defaultdict(list)
    for root in roots:
        if not os.path.isdir(root):
            print(f"scan target does not exist: {root}", file=sys.stderr)
            sys.exit(2)
        for name, info in inventory(root).items():
            seen[name].append(info)
 
    dupes = {n: v for n, v in seen.items() if len(v) > 1}
    print(f"duplicate names: {len(dupes)}")
    risky = 0
    for name, entries in sorted(dupes.items()):
        verdict = "body match" if len({e["sha"] for e in entries}) == 1 else "body differs"
        counts = [len(e["files"]) for e in entries]
        flag = ""
        if max(counts) - min(counts):
            # identical bodies still lose their references if one side is thinner
            risky += 1
            only = set().union(*[set(e["files"]) for e in entries]) - set.intersection(
                *[set(e["files"]) for e in entries]
            )
            flag = f"  <- one side only: {', '.join(sorted(only))}"
        print(f"  {name}: {verdict} / file counts {counts}{flag}")
    print(f"pairs with matching bodies but missing files: {risky}")
    return 1 if risky else 0
 
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

The output:

duplicate names: 5
  find-skills: body match / file counts [1, 1]
  frontend-design: body match / file counts [2, 1]  <- one side only: LICENSE.txt
  seo-audit: body match / file counts [3, 1]  <- one side only: evals/evals.json, references/ai-writing-detection.md
  sleek-design-mobile-apps: body match / file counts [1, 1]
  web-design-guidelines: body match / file counts [1, 1]
pairs with matching bodies but missing files: 2

223 milliseconds.

The frontend-design gap is a LICENSE.txt, so nothing breaks. But deciding that nothing breaks requires seeing the gap first. The tool's job ends at making it visible; a person decides which gaps matter.

Hidden files are excluded from the count because on macOS a stray .DS_Store lands on one side and makes the check complain forever. A check that cries wolf stops being read.

Delete one copy, or pick a source of truth

Three options were on the table.

I took the third. The project copy is the source of truth, and both scripts above run before push. I did not take the first because resolution order can differ between environments. Rather than deleting the copy that loses, I would rather be safe whichever copy wins.

For a team, I think the second option is the honest one. When people's local setups differ, "one name, one entity" is easier to guarantee by design than by inspection.

This shape of oversight probably exists elsewhere

Generalized, the lesson is short. Matching bodies do not mean complete artifacts.

SKILL.md, package.json, config files — all of them invite the "same thing in two places" arrangement, and all of them tempt you to compare only the body.

Since the reference is written in the body, its existence is machine-checkable. Leaving a machine-checkable thing to the eye was the lapse here.

On designing SKILL.md itself, I wrote about keeping the body short and pushing detail into references/ in The second half of my SKILL.md was never being read — a 200-line budget and progressive disclosure. That design holds up well. What I had not pictured until now is that the place you push detail into can go missing. If you need to swap a SKILL.md without restarting, My edited SKILL.md was not taking effect — /reload-skills and .claude/skills autoloading covers the mechanics.

One command to run tomorrow

If you keep more than one skills directory, start by counting the shared names.

comm -12 <(ls .agents/skills | sort) <(ls ~/.claude/skills | sort)

If nothing comes back, none of this concerns you yet. If even one line does, running find . -type f on both copies is worth the minute. I got five lines, and two of them were real.

Thank you for reading. Honestly, the only reason I opened the list instead of stopping when the count came back as four twice was half luck. Writing the steps down is how I stop relying on that.