◉CLAUDE LABJP
●WATERMARK — Since 11/16, text from Opus 4.6, Sonnet 4.5 and other models carries a watermark; non-default temperature and similar settings may drop it●CC 2.1.325 — Claude Code is still on 2.1.325 (11/13), with no newer release yet●SONNET 4.5 — Retires on the Claude API on 11/30, 13 days left. Sweep model IDs in code and saved settings in one pass●AUTOMATION — A user reports recurring AI jobs set up without run counts or a spending cap, burning through a weekly allowance●CONNECTOR — If connectors vanish only in new sessions, compare the /mcp list against an older session first●NEW — Cowork Feels Worse? Run the Same Request in Three Fresh Sessions●WATERMARK — Since 11/16, text from Opus 4.6, Sonnet 4.5 and other models carries a watermark; non-default temperature and similar settings may drop it●CC 2.1.325 — Claude Code is still on 2.1.325 (11/13), with no newer release yet●SONNET 4.5 — Retires on the Claude API on 11/30, 13 days left. Sweep model IDs in code and saved settings in one pass●AUTOMATION — A user reports recurring AI jobs set up without run counts or a spending cap, burning through a weekly allowance●CONNECTOR — If connectors vanish only in new sessions, compare the /mcp list against an older session first●NEW — Cowork Feels Worse? Run the Same Request in Three Fresh Sessions
Articles/API & SDK
⬡ API & SDK/2026-06-22Advanced

Your Claude Files API Storage Is Quietly Filling Up — Dedup With a Content-Hash Ledger and Reap the Orphans

Use the Files API in an automated pipeline and the same file gets uploaded again and again while orphaned files pile up unnoticed. Here is a content-hash dedup ledger plus an orphan GC design, with working code.

Claude API147Files APICost Optimization9Operations25

✦ Premium Article

About two weeks after wiring the Files API into an automated pipeline, I ran GET /v1/files on a whim and went a little pale: dozens of copies of the same reference data, distinguished only by their dates. The upload code was working correctly. The problem was that neither a "don't upload what I already uploaded" mechanism nor a "clean up what I no longer use" mechanism existed anywhere in my code.

The Files API exists so you can "upload once and reference many times" — but the once is the caller's responsibility, not the API's. Used occasionally by hand, nothing accumulates. In an automated workflow that re-uploads the same reference data every day, orphaned files and storage charges pile up silently. This article describes how to stop that pileup with two layers — a content-hash ledger and an orphan GC — based on what I actually built into the Dolice Labs auto-publishing pipeline as an indie developer.

Basic upload steps are covered in the Claude Files API basics guide, so here I focus only on not letting files accumulate.

Why the Files API Fills Up Silently

Files API uploads are not idempotent. POST /v1/files the same PDF twice and you get two distinct entities with different file_id values. To the API they are separate objects, so there is no error and no warning.

This bites hard in automation. Run a job that "uploads reference data every morning and generates articles from that file_id" daily, and even when the content is byte-for-byte identical to yesterday, a new file_id is born each time. Thirty per month; a hundred and twenty across four sites. Each is small, but unreferenced entities keep sitting in storage.

Files API storage is billed on the bytes you retain. So "files you no longer use but never deleted" remain a daily billing line item while doing nothing useful. By the time I noticed, reference data that should have needed four entries had ballooned to nearly eighty.

It helps to split the trap in two. One is duplicate uploads (uploading the same content twice); the other is orphaned files (still present though nothing references them anymore). You stop the former before uploading and reap the latter on a schedule.

The Content-Hash Ledger

The most straightforward way to stop duplicates is to "check, before uploading, whether you have uploaded this content before." The key to that check is a hash of the content.

Compute SHA-256 over the file's bytes and keep a hash → file_id map in a ledger on your side. If the hash of the content you want to upload is in the ledger, reuse the existing file_id. If not, upload for the first time and write the result back into the ledger. That alone guarantees "never upload the same content twice."

Where the ledger lives depends on your workload. I started with a local JSON file and moved to a KV store once multiple processes began running concurrently. The requirement is simple: look up a file_id by hash, by whatever means.

I chose SHA-256 because the chance of a collision is effectively zero in practice. The odds of two different files happening to share a hash and getting mixed up are astronomically small — not worth considering at the scale of reference data. I avoided older hashes like MD5 precisely so that, if a day ever came when I suspected a collision, I would not waste time ruling it out. The longer-lived the mechanism — and a ledger is long-lived — the more the foundation pays to be on the safe side.

import hashlib
import json
from pathlib import Path
from anthropic import Anthropic
 
client = Anthropic()  # ANTHROPIC_API_KEY from the environment
LEDGER_PATH = Path("file_ledger.json")
BETA = "files-api-2025-04-14"  # the Files API needs a beta header
 
 
def load_ledger() -> dict:
    if LEDGER_PATH.exists():
        return json.loads(LEDGER_PATH.read_text())
    return {}
 
 
def save_ledger(ledger: dict) -> None:
    # write to a temp file then swap, to avoid corruption mid-write
    tmp = LEDGER_PATH.with_suffix(".tmp")
    tmp.write_text(json.dumps(ledger, ensure_ascii=False, indent=2))
    tmp.replace(LEDGER_PATH)
 
 
def content_hash(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()
 
 
def get_or_upload(path: str) -> str:
    """Return an existing file_id for identical content; upload if unseen."""
    data = Path(path).read_bytes()
    digest = content_hash(data)
 
    ledger = load_ledger()
    if digest in ledger:
        return ledger[digest]["file_id"]  # reuse — no upload
 
    uploaded = client.beta.files.upload(
        file=(Path(path).name, data, "application/octet-stream"),
        betas=[BETA],
    )
    ledger[digest] = {"file_id": uploaded.id, "name": Path(path).name}
    save_ledger(ledger)
    return uploaded.id

The crucial point is that the hash is computed over the bytes, not the filename. Judge by filename and you miss the case where the name stays the same but the content changes. Judge by content and, conversely, files with different names but identical content collapse into one.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN
✦If you have been ignoring the orphaned files and storage charges that grow with every upload, you can now stop the duplication at its root with a content-hash ledger
✦You will get working orphan-GC logic built on list and delete, with a double-check that prevents accidental deletion of files still in use
✦For pipelines that upload files every day, you can keep storage flat while raising your reference reuse rate
Secure payment via Stripe · Cancel anytime
✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

or
Unlock all articles with Membership →
Share

Thank You for Reading

Claude Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

  • ✦Copy-paste ready implementation code
  • ✦New advanced guides published daily
  • ✦$5/mo or $10 for lifetime access
View Membership →

Related Articles

⬡ API & SDK2026-08-01
Swapping Tools Mid-Conversation Without Losing Your Prompt Cache
Tools can now be added and removed between turns, but the tool block sits at the very front of the cache prefix. I measured 12 mutation patterns with fingerprints and built a stable-core plus volatile-tail registry with a guard that refuses unsafe changes.
⬡ API & SDK2026-04-11
Running Claude's Message Batches Overnight — What the 50% Discount Actually Covers
What I learned running Claude API Message Batches for real: what the 50% discount covers, the custom_id character restriction, and designing around the 24-hour window and 29-day retention.
⬡ API & SDK2026-11-17
If You Rely on Claude's Text Watermark, Check temperature in the Request Body, Not Your Code
Since November 16, text from Opus 4.6, Opus 4.5, Sonnet 4.5, and Haiku 4.5 carries Anthropic's text watermark, but non-default temperature, top_p, or top_k may drop it. Here is how I log the request body once and decide.