Half My Scheduled-Task Logs Were Unreadable, So I Added One Mark Line Instead of Widening the Regex

Widening the regex recovered only 35 entries

I expected more stages to win entries back. The header and body stages together found 11 and 24, so 35 entries, about 3% of 1,076.

Worse, the widening had a side effect. I read through the 24 entries the body stage had caught, and some matched a word that was not a result at all:

unresponsive-server check → FAILED in 2,032 ms

That line is output from a verification I ran while reviewing something. It is not the outcome of the task. But the word FAILED is in the text, so a loose pattern counts that entry as a failed task.

This ran against my expectation. The more I loosened the pattern to shrink the unreadable pile, the more false alarms I produced, because outcome words also appear in prose with other meanings.

So I changed direction. Instead of making the reader smarter, I made the writer promise one thing: a single line in a fixed shape at the very end of each run.

The writer: one function, one line

The spec for that line has four parts.

• One line, tab-separated: RESULT, date, task, status, detail
• Status is one of SUCCESS, SKIPPED, PENDING, FAILED. Nothing else is accepted
• It goes into its own file, separate from the human-readable daily log
• The date is pinned to JST

# Source this file. Call it exactly once in the last step: log_result <task> <STATUS> [detail]
log_result() {
  local task="$1" status="$2" detail="${3:-}" day
  case "$status" in SUCCESS|SKIPPED|PENDING|FAILED) ;; *) echo "log_result: unknown status '$status'" >&2; return 2;; esac
  day="$(TZ=Asia/Tokyo date +%F)"
  detail="${detail//$'\t'/ }"; detail="${detail//$'\n'/ }"      # flatten to one line, no tabs
  printf 'RESULT\t%s\t%s\t%s\t%s\n' "$day" "$task" "$status" "$detail" >> "${LOG_DIR:?LOG_DIR is not set}/_results.tsv"
}

Returning 2 on an unknown status protects the line itself. If a typo like PENDINGG slips in even once, every reader has to carry a branch for unknown states forever. Rejecting it at write time means the reader only needs to know four words.

Flattening tabs and newlines in the detail field is for the same reason: a tab in the detail shifts the columns, and the reader ends up discarding the row.

Usage is a single call in the last step of the task.

source "$HOME/scripts/log_result.sh"
log_result daily-content SUCCESS "push=abc123"

The reader: one command each morning

Writing the line is not enough if the reader does not know which tasks to expect. A task that silently stopped leaves nothing to find. The absence of a line becomes an anomaly only once the reader knows a line should have been there. So the check script carries the list of expected tasks and the weekdays they run.

#!/usr/bin/env python3
"""Morning check: for each expected task, find that day's RESULT line and judge it.
Usage: morning_check_results.py <LOG_DIR> [YYYY-MM-DD]   exit code 0 = all OK / 1 = needs attention"""
import sys, csv, datetime, pathlib, collections
 
EXPECTED = {            # task name: weekdays it runs (Mon=0 … Sun=6). Every day = all seven
    "daily-content": set(range(7)),
    "premium-thu":   {3},
}
OK_STATES = {"SUCCESS", "SKIPPED"}      # SKIPPED means stopped on purpose with a reason, so it is not an anomaly
 
def load(path):
    latest, bad = {}, 0
    if not path.exists():
        return latest, bad
    with path.open(encoding="utf-8") as fh:
        for row in csv.reader(fh, delimiter="\t"):
            if len(row) < 4 or row[0] != "RESULT":
                bad += 1; continue               # count broken lines and move on
            _, day, task, status, *rest = row
            latest[(day, task)] = (status, rest[0] if rest else "")   # last line of the day wins
    return latest, bad
 
def main():
    log_dir = pathlib.Path(sys.argv[1])
    day = sys.argv[2] if len(sys.argv) > 2 else datetime.date.today().isoformat()
    wd = datetime.date.fromisoformat(day).weekday()
    latest, bad = load(log_dir / "_results.tsv")
    problems = []
    for task, weekdays in EXPECTED.items():
        if wd not in weekdays:
            continue
        status, detail = latest.get((day, task), ("MISSING", "no mark was written"))
        mark = "ok " if status in OK_STATES else "NG "
        print(f"{mark}{task:<14}{status:<8}{detail}")
        if status not in OK_STATES:
            problems.append(task)
    if bad:
        print(f"(unreadable lines: {bad} )")
    sys.exit(1 if problems else 0)
 
if __name__ == "__main__":
    main()

I verified it against a test file of marks, not the real logs. It mixes failure and success on the same day, a broken line, and an expected task with no mark.

--- Thursday (2026-10-22)
ok daily-content SUCCESS retry ok
NG premium-thu   MISSING no mark was written
(unreadable lines: 1)
exit=1

daily-content shows ok even though FAILED and SUCCESS both appear that day, because the last line wins. I did not want a day that recovered on retry counted as an anomaly. premium-thu has no mark at all, so it is MISSING and the exit code is 1.

Wire that exit code into any notification and you will notice the morning something stopped without a word.

The order I rolled it out in

1. Run the measuring script on the real logs and get the silent share. If it is above roughly 10%, I would say adding marks is clearly worth it
2. Put log_result.sh in place and add one log_result call to the last step of a single task
3. Let it run for about a week and confirm the mark file grows the way you expect, then extend it to the other tasks
4. Write the expected-task list into the check script and wire exit code 1 to a notification

I recommend trying one task before rolling it out to all of them. Doubts about status words and the detail field almost always show up on the first task.

Three pitfalls

First, last-line-wins hides mid-run failures. If the final retry succeeds, the day shows ok. To see how often retries happen, keep the raw mark file, not just the check output. I split the roles: the check is today's verdict, the mark file is the history.

Second, the expected-task list goes stale. If I stop a task and forget to remove it, MISSING appears every morning. If I add a task and forget to list it, that task is never checked. I added one line to my add/remove-a-task routine: update the list.

Third, do not backfill old logs. I was tempted to fill the 515 silent entries with guesses, and decided against it. Guessed states lined up as facts make the check meaningless. The past stays as it was, and marks start today.

My decision

I chose not to make the reader smarter, but to narrow the writer's promise to one line. The logs can stay as free-form as they like for human eyes. Only what a machine reads goes into a separate, fixed shape at the end.

Don't try to unify log formats; set aside one machine-readable line. That is the whole takeaway.

If you have a task that has been running for a while, run the measuring script and look at your silent share first. That number tells you whether adding marks is worth it. For related reading, checking the output itself before exit is covered in Logged as success, but it produced nothing, and the day a log overwrote the previous day's file is in My Unattended Task Was Overwriting the Previous Day's Log.