Letting Claude Read Live Pages: Implementing web_fetch Without the Footguns

Read the response as three layers

The returned content array roughly follows this order. When you write the parsing code, thinking in these three layers keeps it straightforward.

1. text — the model's statement that it will read the page
2. server_tool_use — the fetch request (target URL in input.url)
3. web_fetch_tool_result — the result; the body lives in content.content (a document block)

An actual result looks like this:

{
  "type": "web_fetch_tool_result",
  "tool_use_id": "srvtoolu_01234567890abcdef",
  "content": {
    "type": "web_fetch_result",
    "url": "https://example.com/article",
    "content": {
      "type": "document",
      "source": {
        "type": "text",
        "media_type": "text/plain",
        "data": "Full text of the page..."
      },
      "title": "Article Title"
    },
    "retrieved_at": "2026-06-15T01:30:00Z"
  }
}

For PDFs, source.type becomes base64 and media_type becomes application/pdf, so you get base64 data instead of plain text. For summarization you can ignore the distinction, but if you want to store or parse the fetched body yourself, remember to branch between text and base64. retrieved_at matters more than it looks: web_fetch sits behind a cache, so a stale timestamp is your signal that "the latest" was actually a slightly older version.

The biggest trap: the URL must already be in the conversation

This is the one detail that is easy to miss on a quick read of the docs. web_fetch cannot retrieve a URL that has never appeared in the conversation context.

Concretely, the URLs it can fetch are limited to:

• URLs written in a user message
• URLs in client-side tool results
• URLs that came out of a previous web_search or web_fetch

In other words, a URL that Claude assembles by guessing from the prompt ("this is probably the link") gets rejected. This is by design, to prevent data exfiltration — it stops an attacker from getting a URL with secret data baked into the query string fetched.

My first version got stuck right here. I had Claude build a category-page URL from an article slug and tried to fetch it. The result was url_not_allowed. The fix is simple: always write the URL you want read explicitly into the user message yourself. If you need to start from a search, pair it with web_search and let it fetch a URL that appears in those results.

resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=4096,
    messages=[
        {
            "role": "user",
            "content": "Find the latest Claude Code release notes and lay out only the "
                       "single newest change, with citations.",
        }
    ],
    tools=[
        {"type": "web_search_20250305", "name": "web_search", "max_uses": 3},
        {
            "type": "web_fetch_20250910",
            "name": "web_fetch",
            "max_uses": 5,
            "citations": {"enabled": True},
        },
    ],
)

This "search, pick a candidate, fetch the full text, answer with citations" flow fits automation where you do not have the URL in hand. If you can name the page, fetch alone; if you do not yet know where it lives, combine with search.

Cap tokens and attack surface at the same time

Using web_fetch carries no extra charge by itself. The cost is purely the tokens of the fetched body once it lands in context. The flip side is that reading a large page unguarded eats tokens fast. As a rough guide, a 100 kB documentation page reaches about 25,000 tokens.

This is where max_content_tokens earns its keep. If the fetched body exceeds the limit, it gets truncated, so for summarization I set it around 8,000. For a 25,000-token page that compresses the intake to roughly 32%, and when you run dozens of monthly scheduled jobs, that one line visibly changes the bill.

Two more worth pairing with it are allowed_domains and max_uses. The former pins retrieval to domains you trust; the latter caps how many fetches a single request can make.

tools=[
    {
        "type": "web_fetch_20250910",
        "name": "web_fetch",
        "max_uses": 5,
        "allowed_domains": ["platform.claude.com", "docs.claude.com"],
        "max_content_tokens": 8000,
        "citations": {"enabled": True},
    }
]

One caveat: you cannot set allowed_domains and blocked_domains in the same request — it is one or the other. For automation with a known set of trusted sources, I recommend the allow-list approach. Starting from an explicit allow list feels far less accident-prone than trying to plug holes after the fact with blocked_domains.

Errors arrive as HTTP 200

Another trap is how errors come back. A failed fetch is not an HTTP error status; it arrives inside a 200 response body as a web_fetch_tool_error. If you only wrote an exception handler, you will never notice the failure.

def extract_fetch_errors(resp):
    errors = []
    for block in resp.content:
        if getattr(block, "type", None) != "web_fetch_tool_result":
            continue
        inner = block.content
        if getattr(inner, "type", None) == "web_fetch_tool_error":
            errors.append(inner.error_code)
    return errors
 
codes = extract_fetch_errors(resp)
if codes:
    # In automation, log it here and optionally retry against another domain
    print("web_fetch failed:", codes)

The possible error_code values are worth knowing: url_too_long (URL over 250 characters), url_not_allowed (blocked by domain rules or URL validation), url_not_accessible (the target returned an HTTP error), too_many_requests (rate limited), unsupported_content_type (anything other than text and PDF), max_uses_exceeded, and the internal unavailable.

In practice the ones I hit most were unsupported_content_type and "the body came back nearly empty." web_fetch does not support pages rendered dynamically with JavaScript. Point it at a client-rendered SPA and you get the skeleton with thin content. If you already know the target is a dynamic page, it is more realistic to choose a different retrieval path than to fight web_fetch.

For large documents, reach for dynamic filtering

When you need only a specific section out of a long PDF or a huge document, the newer tool version web_fetch_20260209 helps. Here Claude writes code to filter the fetched body before it loads into context, keeping only what you need and discarding the rest, which holds token use down while preserving accuracy.

resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=4096,
    messages=[
        {
            "role": "user",
            "content": "From this long PDF, extract only the passages about pricing: "
                       "https://example.com/whitepaper.pdf",
        }
    ],
    tools=[
        {"type": "web_fetch_20260209", "name": "web_fetch"},
        {"type": "code_execution_20250825", "name": "code_execution"},
    ],
)

Dynamic filtering requires the code execution tool to be enabled, and note that supported models are limited (newer generations such as Fable 5 and Opus 4.8). Conversely, if the page you fetch is small to begin with, the benefit is thin and the setup more complex. My own split is: a page of a few kB goes through the classic web_fetch_20250910, while a research-paper PDF in the hundreds-of-thousands-of-tokens range goes through web_fetch_20260209. Choosing by how much content you pull in, rather than by how new the tool version is, makes both the results and the cost easier to predict.

One next step

Pin a single trusted documentation page with allowed_domains, cap it with max_content_tokens set to 8,000, and have that minimal setup summarize it. Log both retrieved_at and any error codes there, and the production behavior becomes far easier to read once you ship it. Details of the spec do shift, so confirm the tool version and beta header in the official Anthropic documentation each time.

Reading primary sources directly is an unglamorous step, but it moved the needle on factual accuracy more than almost anything else. I hope it helps if you're wrestling with the same accuracy problem in your own generation.