Why Claude Code's WebFetch Returns 403 or Empty Pages — and How to Fix It

"I asked Claude Code to read a technical doc and got a 403." "The response is 200 OK, but the body is basically empty — just navigation, no article text." If you use Claude Code daily for automation, these are the two failure modes that quietly break pipelines you thought were rock solid. As someone running four Lab sites with auto-publishing pipelines, I run into one or the other a few times every month.

The good news: once you slow down and look at it carefully, every WebFetch failure I have ever debugged collapses into four root causes. Here is how to tell them apart, and what I actually do in each case.

A quick mental model of WebFetch

Under the hood, Claude Code's WebFetch tool asks Anthropic's servers to issue an HTTP GET against the URL, then hands the resulting HTML to the model as text. The crucial detail: it is not a browser.

That means:

• No JavaScript execution. Sites built with React, Next.js, or Vue that render content client-side return only the initial HTML shell — usually a spinner and a few navigation links.
• A fixed User-Agent. Some sites — especially those running Cloudflare's Bot Fight Mode — bounce that User-Agent on sight.
• Short-term caching. Hitting the same URL repeatedly in a short window can serve a stale cached response.
• Domain allowlists matter. If you've launched Claude Code with --allowedDomains, anything outside that list is blocked before the request goes out.

So when you see a 403, an empty body, a timeout, or "outdated" data, it always maps back to one of those four axes.

Cause 1: Cloudflare or similar anti-bot returning 403

Anthropic's own help docs, raw GitHub files, Stack Overflow, and many other sites sit behind Cloudflare. WebFetch requests against them often come back as:

Web fetch failed: HTTP 403 — the server refused to fulfill the request

This is Cloudflare's Managed Challenge or Bot Fight Mode looking at the User-Agent and TLS fingerprint and concluding "this clearly isn't a browser." In my experience, agents that pull technical documentation tend to hit it most.

What I actually do

First, I look for an alternative source for the same information. https://github.com/owner/repo/blob/main/README.md often 403s, but https://raw.githubusercontent.com/owner/repo/main/README.md returns the raw text. Many vendors now publish a *.json reference or an llms.txt next to the docs site.

If I genuinely need that exact URL, I fall back to Claude in Chrome (the Chrome MCP). The prompt looks something like:

Try WebFetch on this URL. If it returns 403 or an empty page,
fall back to mcp__Claude_in_Chrome__navigate, then read the body
with mcp__Claude_in_Chrome__get_page_text.
URL: https://example.com/docs

Because Claude in Chrome runs a real browser, it usually passes the Cloudflare challenge. The same pattern is what lets me operate JavaScript-heavy SaaS dashboards — AdMob is the obvious example from my app business — through Claude without needing custom automation.

It is tempting to drop down to curl or wget from the Bash tool, but in Cowork and many restricted environments outbound network access is limited, and using bash to deliberately bypass WebFetch isn't aligned with how Cowork is meant to be used. Reaching for Claude in Chrome is the cleaner answer.

Cause 2: Client-side rendered pages with no body

The status is 200 OK, the response is large, but <main> contains nothing more than "Loading..." or "Enable JavaScript." This is sneakier than a 403 because nothing visibly fails.

Common tells:

• Navigation links are readable, but the article body is missing
• A search results list shows "0 items" even though the site clearly has content
• A pricing table renders with empty cells

It happens on SPA-style React, Next.js, and Vue apps where the initial HTML is just an application shell and the real content gets fetched after hydration.

Fix: confirm whether SSR is on, then switch to Claude in Chrome

The fastest check is "View Source" in a real browser. If the body text you wanted isn't in the raw HTML, WebFetch cannot retrieve it.

Modern Next.js with the App Router often does SSR, but older Vercel setups, or Vite + React + React Router projects, frequently leave content fully client-rendered.

In that case I switch to:

This page is likely an SPA and WebFetch won't see the body.
Use mcp__Claude_in_Chrome__navigate first, then call
mcp__Claude_in_Chrome__get_page_text to read the rendered body.

My four Lab sites (Claude Lab, Gemini Lab, Antigravity Lab, Rork Lab) are themselves Next.js 16 + App Router, and after every deploy I spot-check view-source: to confirm article bodies are in the HTML. Being WebFetch-friendly is, in effect, being friendly to AI crawlers in general.

Cause 3: Blocked by --allowedDomains before the request goes out

When you run Claude Code in CI or on a schedule, it is common to lock down --allowedDomains to a curated set. Anything outside that list fails with:

WebFetch is not allowed for domain: stackoverflow.com
Allowed domains: docs.anthropic.com, github.com

This is working as intended — it skips the human confirmation dialog while keeping fetches inside a known safe boundary. The trap is forgetting to update the list as new domains become relevant; the production job just quietly stops working.

What I do

I have been shipping indie apps since 2014, and the longer something runs, the more the allowlist becomes a thing you grow. Rather than hard-coding it in scripts, I keep it in a config file:

# Example: ~/.claude-code/allowed-domains.txt
docs.anthropic.com
www.anthropic.com
raw.githubusercontent.com
api.github.com
developer.apple.com
developer.android.com
admob.google.com
firebase.google.com

Then in CI:

claude code --allowedDomains "$(paste -sd, ~/.claude-code/allowed-domains.txt)" ...

When a new domain needs to be added, I do it via a pull request so the history of "why was this allowed" is preserved. Small thing, but it pays off six months later.

Cause 4: A stale cached response

WebFetch keeps a short-lived cache; the same URL hit again quickly may return a cached body. If you're checking a real-time status page or a price, "I refreshed but nothing changed" is usually this.

Fix

The simplest workaround is to append a throwaway query parameter:

https://example.com/status?_=1716120000

Most servers ignore an unknown _= parameter, so the content isn't affected. For WebFetch, though, it counts as a different URL — and the cache is bypassed.

That said, caching exists to save you money and latency. Use this trick deliberately, not by default. For things that genuinely change daily — my morning SEO reports off GA and GSC, for instance — I append a date-based parameter rather than a timestamp, so within the same day I still get the cached result.

Keep a fixed triage order

The most practical thing in day-to-day work is knowing the order in which to rule things out. This is the checklist I run:

1. Look at the status code. 403 or 429 → cause 1. 200 → check 2 and 4.
2. Open view-source: and search for the body text. Missing? → cause 2; switch to Claude in Chrome.
3. Inspect --allowedDomains if you are in CI. This is the silent killer.
4. Append a throwaway query parameter and refetch. If the result changes → cause 4.

In my experience, almost nothing escapes this list. The flip side is that hammering WebFetch without diagnosing first quietly racks up API spend, so the habit of stopping to triage early actually saves money over time.

A practical next step

If you need just one takeaway, it is this: keep Claude in Chrome (the Chrome MCP) on hand as a permanent fallback channel. You can toggle it on from Cowork's top-right; once enabled, it handles both 403 and empty-page cases out of the box. Running four Lab sites alongside an indie app business that has crossed 50 million downloads, I lean on this combination every day, and having a robust second channel has paid for itself many times over in both reliability and operating cost.

Hope this helps you debug your own pipelines.