One morning I opened the log at the usual hour and found only the first line from the night before. It belongs to a small task that crawls release notes for the libraries my apps depend on, plus a couple of store policy pages, while I sleep.
The process was alive. It wasn't burning CPU. There was no error. It simply never came back.
The cause turned out to be one of the pages: a server that opens its response and never closes it. And at the time, Claude Code was built to wait for that kind of server indefinitely.
Where to look first when nothing comes back
When a task stalls, the prompt and the permission settings are the tempting places to start. That's where I started, and it got me nowhere. I trimmed the prompt, I widened the permissions, and every morning looked the same.
I had the order wrong. With a stall, find out who is waiting before you touch what you wrote. Whether the waiting is on your side, the tool's side, or the remote server's side changes the repair entirely.
So I look in this order:
- The last line of the log — which tool call is it sitting just after?
- The process state — has it died, or is it alive and waiting?
- The URL on that line, hit by hand with
curl
The third step usually answers it. A page that opens fine in a browser can still hang when you fetch it.
# Without --max-time, this call hangs on your side too
curl -sS -o /dev/null -w 'http=%{http_code} total=%{time_total}s\n' \
--max-time 20 "https://example.com/release-notes"
echo "exit=$?"When --max-time fires, curl exits with 28. If you see 28 here, the problem is not what you wrote.
Reproducing a server that never closes
It helps to have the misbehaving server on your own machine, so you can check whether a fix actually bites. No dependencies needed.
// slow-server.mjs — sends headers and a partial body, then never closes
import { createServer } from "node:http";
createServer((req, res) => {
res.writeHead(200, { "Content-Type": "text/html; charset=utf-8" });
res.write("<html><body><p>preparing the report…</p>");
// The body is never finished and res.end() is never called
}).listen(8080, () => {
console.log("listening on http://127.0.0.1:8080");
});Start it with node slow-server.mjs and point the curl above at http://127.0.0.1:8080. You get exit 28 after exactly 20 seconds. Hand the same URL to WebFetch on a build older than v2.1.268 and it waits.
That is the shape of a failure that only bites once a night. When I'm at the keyboard, I notice and stop it myself. Unattended, nobody stops it.
0 does not mean "don't wait" — it removes the deadline
Version 2.1.268, released on September 10, draws a line under this. The changelog says WebFetch no longer hangs indefinitely on a server that keeps the response open without finishing, that a fetch now fails after 300 seconds, and that CLAUDE_CODE_WEBFETCH_DEADLINE_MS overrides the deadline.
I misread it the first time. I took 0 to mean "give up immediately" and put it in the crawl task's environment. The next morning looked exactly like the mornings before the fix. 0 turns the deadline off.
| Value | How WebFetch waits | When it fits |
|---|---|---|
| unset | Fails after 300 seconds (the default) | Interactive work at the keyboard |
60000 | Fails after 60 seconds | Crawls with many targets, where giving up early is fine |
0 | No deadline; waits on a response that never closes | Only to restore the old behaviour deliberately |
For a crawl, even 300 seconds can be too generous. With twelve targets, three of which never close, that alone spends fifteen minutes on waiting. I've settled on 60 seconds.
{
"env": {
"CLAUDE_CODE_WEBFETCH_DEADLINE_MS": "60000"
}
}If it isn't worth a settings file, pass it per run instead.
CLAUDE_CODE_WEBFETCH_DEADLINE_MS=60000 claude -p "$(cat ./prompts/watch-release-notes.txt)"An inner deadline and an outer ceiling
With a deadline in place, the worst case for a single fetch is finally something I can state. The worst case for the whole task still isn't: sixty seconds stacks up once per target, and WebFetch is not the only thing that can stall.
So the runner carries a second ceiling of its own.
#!/usr/bin/env bash
set -uo pipefail
LOG="${HOME}/logs/watch-release-notes-$(date +%F).log"
mkdir -p "$(dirname "$LOG")"
# Inner: the per-fetch deadline
export CLAUDE_CODE_WEBFETCH_DEADLINE_MS=60000
start=$(date +%s)
# Outer: the whole task. If TERM doesn't end it, KILL follows 30 seconds later
timeout --signal=TERM --kill-after=30s 20m \
claude -p "$(cat ./prompts/watch-release-notes.txt)" >> "$LOG" 2>&1
code=$?
elapsed=$(( $(date +%s) - start ))
case "$code" in
0) printf 'ok elapsed=%ss\n' "$elapsed" >> "$LOG" ;;
124) printf 'CEILING outer limit reached elapsed=%ss\n' "$elapsed" >> "$LOG" ;;
*) printf 'FAILED exit=%s elapsed=%ss\n' "$code" "$elapsed" >> "$LOG" ;;
esac
exit "$code"The outer number is just the inner deadline times the number of targets, plus some slack. There is nothing principled about twenty minutes — it came from the length of my own list.
The reason both are needed is that they answer different questions. The inner one decides to give up on this fetch; the outer one decides that tonight is over. Put one ceiling inside the tool and one outside the run. Since I started placing them that way, I haven't opened a log to find the tail blank.
Make the giving-up visible by morning
A deadline alone only gets you halfway. If nothing records that it fired, you won't notice the pages that never made it in. A quiet gap can be worse than a stall.
So the reachability check now happens outside the task, before the crawl starts.
#!/usr/bin/env bash
set -uo pipefail
SRC="./urls.txt" # one URL per line
TODAY="${HOME}/logs/urls-$(date +%F).txt"
REPORT="${HOME}/logs/probe-$(date +%F).tsv"
: > "$TODAY"; : > "$REPORT"
while read -r url; do
[ -z "$url" ] && continue
t0=$(date +%s)
http=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 20 "$url") || http="000"
printf '%s\t%s\t%ss\n' "$url" "$http" "$(( $(date +%s) - t0 ))" >> "$REPORT"
[ "$http" = "200" ] && printf '%s\n' "$url" >> "$TODAY"
done < "$SRC"
# Only the URLs that answered go into the real prompt
echo "probe: $(wc -l < "$TODAY") of $(grep -cve '^$' "$SRC") targets answered"Rows with http=000 are the ones that didn't answer. One file in the morning tells you what last night was missing. For a crawl with a fixed list of targets, this pre-pass did more for me than any tuning inside the task.
If you're also thinking about how often those pages get fetched at all, choosing a WebFetch cache TTL that matches how you actually work is the companion piece. A deadline is about abandoning one fetch; a TTL is about making fewer of them.
Before tonight's run, set CLAUDE_CODE_WEBFETCH_DEADLINE_MS to something short, just once, and check the tail of the log in the morning. Seeing that it isn't blank changes how the whole night reads.
If you keep something running while you sleep, I hope one piece of this is useful to you. Thank you for reading.