I Linted 36 Unattended-Run Prompts: All 7 'Use This Tool' Hits Turned Out to Be Bans

Reading the 7 lines: all of them were bans

I opened each suspect line. These are the kinds of lines that came up.

Not one of the 7 was an instruction to use a tool. The cause was that my negation list depended on how the sentence ended. It caught 使わない, but a one-step change in conjugation or phrasing (使わず, 使わずに, 回避) turned a ban into a suspect.

The third row is a type that no dictionary fix can rescue at line level: the heading declares the ban, and the line itself holds only a name.

There was a second problem. Most of the 36 files were not prompts a task actually reads. They were request memos, hand-over notes and a README, and the "do not use Read" lines inside them were being counted too, so my denominator was too large.

The fix: match on stems, narrow the target

Two changes. First, match negation on word stems: putting 使わ in the list catches 使わない, 使わず and 使わずに together, and I did the same with 呼ばな, 呼び出さな and 使用しな.

Second, I removed 絶対 ("absolutely") from the list. I only noticed this one on the way back: 絶対 is emphasis, not negation. If someone writes "use AskUserQuestion, absolutely", a list containing 絶対 passes it as a ban. A word I had added to reduce false alarms was hiding a real violation.

To narrow the target, only files whose frontmatter has a name: count as runnable prompts. Memos and READMEs have none, so they drop out naturally.

#!/usr/bin/env python3
"""Lint for unattended-run prompts.
Counts lines that tell the task to USE a tool that raises a permission dialog.
  - any suspect line                 -> exit 1 (fix before the run)
  - only the top ban header missing  -> warning (exit 0)"""
import json, pathlib, re, sys
 
DIALOG_WORDS = ["AskUserQuestion", "request_cowork_directory", "request_access",
                "allow_cowork_file_delete", "computer-use"]
FILE_TOOLS = re.compile(r"(?<![A-Za-z_])(Read|Write|Edit)(?![a-z_])")
# Match on the STEM so 使わない / 使わず / 使わずに are all caught by 使わ.
# 絶対 ("absolutely") is emphasis, not negation: leave it out, or
# "AskUserQuestion を絶対に使う" (absolutely use it) passes as a ban.
NEGATION = re.compile(r"禁止|呼ばな|呼び出さな|使わ|使用しな|してはなら|回避|"
                      r"[Nn]ever|[Dd]o not|[Dd]on't")
BAN_HEADER = "権限ダイアログ禁止"
 
def is_task_prompt(text: str) -> bool:
    m = re.match(r"---\n(.*?)\n---", text, re.S)
    return bool(m and re.search(r"^name:", m.group(1), re.M))
 
def scan_text(name: str, text: str) -> dict:
    res = {"file": name, "ban_header": BAN_HEADER in text, "suspects": [], "bans": 0}
    for no, line in enumerate(text.splitlines(), 1):
        if not (any(w in line for w in DIALOG_WORDS) or FILE_TOOLS.search(line)):
            continue
        if NEGATION.search(line):
            res["bans"] += 1
        else:
            res["suspects"].append((no, line.strip()[:90]))
    return res
 
def main(root: str) -> int:
    rows = []
    for p in sorted(pathlib.Path(root).glob("*.md")):
        t = p.read_text(encoding="utf-8", errors="replace")
        if is_task_prompt(t):
            rows.append(scan_text(p.name, t))
    fail = [r for r in rows if r["suspects"]]
    warn = [r for r in rows if not r["ban_header"]]
    print(json.dumps({"task_prompts": len(rows), "fail": len(fail), "warn_no_header": len(warn)},
                     ensure_ascii=False))
    for r in fail:
        for no, l in r["suspects"][:3]:
            print(f'FAIL {r["file"]}:L{no}: {l}')
    for r in warn:
        print(f'warn {r["file"]}: no ban header at top ({r["bans"]} ban line(s) in the body)')
    return 1 if fail else 0
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Three notes on why it is written this way. The lookarounds around Read exclude neighbouring letters so that README and Reader are not matched. A missing ban header is only a warning, and the exit code is 1 only when a suspect line exists. And because the result comes back as an exit code, chaining it with && before a run means a day with an unfixed prompt simply does not start the task.

Test the dictionary itself

The moment right after adding or removing a word is the most fragile one. So I wrote the expectations as an eight-row table and run it each time. Four ban lines should not be suspects; four use-instructions should.

import importlib.util
spec = importlib.util.spec_from_file_location("pl", "prompt_lint.py")
pl = importlib.util.module_from_spec(spec); spec.loader.exec_module(pl)
 
CASES = [  # (body, should it be a suspect?)
    ("Read/Write/Edit ツールは使わず bash のみ", False),
    ("AskUserQuestion は呼ばない", False),
    ("許可ダイアログ回避のため Write を避ける", False),
    ("Never call request_access", False),
    ("最初に AskUserQuestion で確認してください", True),
    ("ファイルは Read で開いて確認する", True),
    ("AskUserQuestion を絶対に使う", True),   # 絶対 is emphasis, not a ban
    ("Read して内容を整理する", True),
]
bad = 0
for text, want in CASES:
    got = bool(pl.scan_text("t", "\n" + text)["suspects"])
    ok = got == want
    bad += (not ok)
    print(("OK " if ok else "NG ") + f"suspect={got!s:5} <- {text}")
raise SystemExit(bad)

Running it printed OK on all eight rows, exit code 0. Keeping row 7, the "absolutely use it" line, with its expected value True means the test fails if someone puts 絶対 back into the list.

Running it on the real prompts

I ran the corrected version over the same folder.

python3 test_prompt_lint.py && python3 prompt_lint.py "$PROMPT_DIR"; echo "exit=$?"

The output: 19 runnable prompts, 0 with suspect lines, 5 warnings for a missing ban header, exit code 0. The 36 files narrowed to 19, and the 7 suspects became 0.

The 5 warnings had no header at the top but 2 to 3 ban lines each in the body, so there had never been a risk of stalling. I did not treat them as failures; I will add the headers the next time I edit them. The missing header was a matter of findability, not risk.

The "bare names under a heading" type still comes up as a suspect in the current version. The file from row 3 is absent from this result only because it is a document without name:. If I wrote the same bullet in a runnable prompt, the current version would stop on a false alarm. I have not yet decided between making the lint remember headings and judging lines by them, or moving my habit to writing "do not use" on every line.

Where it still goes wrong

The comparison in the second row is still manual. I count whether the 19 linted prompts match the registered tasks by eye. That is the next thing I want to automate.

Putting it into your own setup

This is how I put it in, in five steps. Followed in order, it takes under half an hour.

1. Decide where the prompts live, and make sure only runnable prompts (with name:) are *.md files there
2. Replace DIALOG_WORDS and FILE_TOOLS with the tools that raise a dialog in your environment
3. Run test_prompt_lint.py first and confirm all eight rows print OK
4. Run the main script and open each FAIL line. If it is a ban, add the stem to the dictionary; if it is a use-instruction, rewrite it as a bash operation
5. Chain it with && just before the task starts, so a day with exit code 1 does not run the task. I recommend this placement

What to take away

Linting an unattended prompt means counting the occurrences without negation, not the occurrences of banned words. Build the negation list on stems, not endings, and keep emphasis words out of it. When you touch the list, run a table of expectations.

As a first step, swap DIALOG_WORDS for the tools that raise a dialog in your own environment and run it once over your prompt folder. If you get 0, it is not luck; it is a 0 you checked. For the wider setup of overnight runs, see Running Cowork Scheduled Tasks in Practice — From a Morning Digest to Unattended Weekly Reports. The problem of a check that passes silently is covered from another angle in Until I planted a failing sample, my unattended checks had never once failed.