Let Claude Actually See the Images Your Tools Return — Use Image Blocks in tool_result and Cut Tokens by Roughly 10x

Pin down the before/after cost with real numbers

To make the difference concrete, here is an estimate for passing the same 1024×1024 PNG two ways.

Image token count is roughly "height × width ÷ 750". For 1024×1024 that is 1024 * 1024 / 750 ≈ 1,398 tokens. Knowing this formula lets you decide, before sending, whether a tool's image needs downscaling. Returning a 2048×2048 image costs about 5,600 tokens, but if 1024 is enough for the judgment, downscaling before you send cuts that to a quarter.

Conversely, the string anti-pattern makes this estimate useless. Text is billed per character, and base64 inflates the original by about 1.37x. That is how a single image becomes 20,000 tokens.

Wire it into the agent loop

A real loop looks like this: Claude returns a tool_use, your code runs the tool and gets an image, you return that image as an image block in the tool_result, and Claude looks at the image to make its next decision.

import base64, anthropic
 
client = anthropic.Anthropic()
 
def render_thumbnail(candidate_id: str) -> bytes:
    # In practice, generate or fetch the thumbnail. Returns PNG bytes here.
    ...
 
def run_review(candidate_id: str):
    messages = [{
        "role": "user",
        "content": f"Generate the thumbnail for candidate {candidate_id} and rate its legibility from 1 to 5.",
    }]
 
    while True:
        resp = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            tools=[{
                "name": "render_thumbnail",
                "description": "Generate and return the thumbnail PNG for a candidate ID",
                "input_schema": {
                    "type": "object",
                    "properties": {"candidate_id": {"type": "string"}},
                    "required": ["candidate_id"],
                },
            }],
            messages=messages,
        )
 
        if resp.stop_reason != "tool_use":
            return resp  # the final rating came back
 
        # Handle the tool_use blocks
        tool_results = []
        for block in resp.content:
            if block.type != "tool_use":
                continue
            png = render_thumbnail(block.input["candidate_id"])
            b64 = base64.standard_b64encode(png).decode("ascii")
            tool_results.append({
                "type": "tool_result",
                "tool_use_id": block.id,
                "content": [{
                    "type": "image",
                    "source": {"type": "base64", "media_type": "image/png", "data": b64},
                }],
            })
 
        messages.append({"role": "assistant", "content": resp.content})
        messages.append({"role": "user", "content": tool_results})

If you forget messages.append({"role": "assistant", "content": resp.content}), you get the "tool_result block missing corresponding tool_use block" error. The required order is: append the assistant's tool_use back into the history, then return the tool_result in the very next user turn.

The shape is identical in the TypeScript SDK

The structure does not change with the language. As long as you make content an array and include an image block, it works.

import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
 
const toolResult = {
  type: "tool_result" as const,
  tool_use_id: toolUseId,
  content: [
    {
      type: "image" as const,
      source: { type: "base64" as const, media_type: "image/png" as const, data: b64 },
    },
    { type: "text" as const, text: "This is the latest App Store screenshot." },
  ],
};

After I swap out App Store screenshots, I show the new image to Claude through a tool and have it do a first-pass check for clipped text or content spilling outside the device frame. Showing the actual image gets far more specific feedback than describing it in words. Back when I passed base64 as a string here, I was wasting tens of thousands of tokens on every check.

URL and the Files API as alternatives

Embedding base64 in every message makes requests heavy. If you reference the same image repeatedly, you can change the source type.

In a pipeline that shows the same figure over and over, uploading once to the Files API and referencing by file_id avoids resending base64 each time. For a loop like mine that inspects thumbnails in a morning batch, I upload the fixed reference image up front and only send the changing candidate image as base64.

Pitfall checklist

A few things that trip people up mid-implementation.

• Get the media type right: use exactly one of image/png, image/jpeg, image/gif, or image/webp, matching the real bytes. Declaring a PNG as image/jpeg makes decoding fail.
• Downscale before sending: an oversized image costs more tokens and gets downscaled internally anyway. Dropping it to the resolution you actually need keeps both cost and reproducibility stable.
• Not auto-shown to users: an image returned as a tool result is visible to Claude, but it is not automatically rendered inline in your app's final answer. To show it in a chat UI, you have to draw it explicitly in your own frontend.
• Order and pairing: keep the tool_use (assistant) → tool_result (user) correspondence intact. If you call multiple tools in parallel, return a result for every tool_use_id.
• Per-image size limit: the API caps the size of a single image. Rather than throwing the full-resolution original at it, design your tool to return a downscaled inspection copy.

The one line to change first

If you already have a tool returning images, the smallest improvement you can ship today is clear. Find where you put base64 into content as a string, make content an array, and replace it with a {"type": "image", "source": {...}} block. That alone cuts the token cost of the same image to roughly a tenth, and for the first time lets Claude actually see the content and judge it.

I hope this helps with your own implementation. I am still finding the sweet spot as I run this in practice, but when you write a tool that returns images, just checking one thing first — "am I making Claude read this as text, or letting it see it as an image?" — already goes a long way.