Your Claude Files API Storage Is Quietly Filling Up — Dedup With a Content-Hash Ledger and Reap the Orphans

About two weeks after wiring the Files API into an automated pipeline, I ran GET /v1/files on a whim and went a little pale: dozens of copies of the same reference data, distinguished only by their dates. The upload code was working correctly. The problem was that neither a "don't upload what I already uploaded" mechanism nor a "clean up what I no longer use" mechanism existed anywhere in my code.

The Files API exists so you can "upload once and reference many times" — but the once is the caller's responsibility, not the API's. Used occasionally by hand, nothing accumulates. In an automated workflow that re-uploads the same reference data every day, orphaned files and storage charges pile up silently. This article describes how to stop that pileup with two layers — a content-hash ledger and an orphan GC — based on what I actually built into the Dolice Labs auto-publishing pipeline as an indie developer.

Basic upload steps are covered in the Claude Files API basics guide, so here I focus only on not letting files accumulate.

Why the Files API Fills Up Silently

Files API uploads are not idempotent. POST /v1/files the same PDF twice and you get two distinct entities with different file_id values. To the API they are separate objects, so there is no error and no warning.

This bites hard in automation. Run a job that "uploads reference data every morning and generates articles from that file_id" daily, and even when the content is byte-for-byte identical to yesterday, a new file_id is born each time. Thirty per month; a hundred and twenty across four sites. Each is small, but unreferenced entities keep sitting in storage.

Files API storage is billed on the bytes you retain. So "files you no longer use but never deleted" remain a daily billing line item while doing nothing useful. By the time I noticed, reference data that should have needed four entries had ballooned to nearly eighty.

It helps to split the trap in two. One is duplicate uploads (uploading the same content twice); the other is orphaned files (still present though nothing references them anymore). You stop the former before uploading and reap the latter on a schedule.

The Content-Hash Ledger

The most straightforward way to stop duplicates is to "check, before uploading, whether you have uploaded this content before." The key to that check is a hash of the content.

Compute SHA-256 over the file's bytes and keep a hash → file_id map in a ledger on your side. If the hash of the content you want to upload is in the ledger, reuse the existing file_id. If not, upload for the first time and write the result back into the ledger. That alone guarantees "never upload the same content twice."

Where the ledger lives depends on your workload. I started with a local JSON file and moved to a KV store once multiple processes began running concurrently. The requirement is simple: look up a file_id by hash, by whatever means.

I chose SHA-256 because the chance of a collision is effectively zero in practice. The odds of two different files happening to share a hash and getting mixed up are astronomically small — not worth considering at the scale of reference data. I avoided older hashes like MD5 precisely so that, if a day ever came when I suspected a collision, I would not waste time ruling it out. The longer-lived the mechanism — and a ledger is long-lived — the more the foundation pays to be on the safe side.

import hashlib
import json
from pathlib import Path
from anthropic import Anthropic
 
client = Anthropic()  # ANTHROPIC_API_KEY from the environment
LEDGER_PATH = Path("file_ledger.json")
BETA = "files-api-2025-04-14"  # the Files API needs a beta header
 
 
def load_ledger() -> dict:
    if LEDGER_PATH.exists():
        return json.loads(LEDGER_PATH.read_text())
    return {}
 
 
def save_ledger(ledger: dict) -> None:
    # write to a temp file then swap, to avoid corruption mid-write
    tmp = LEDGER_PATH.with_suffix(".tmp")
    tmp.write_text(json.dumps(ledger, ensure_ascii=False, indent=2))
    tmp.replace(LEDGER_PATH)
 
 
def content_hash(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()
 
 
def get_or_upload(path: str) -> str:
    """Return an existing file_id for identical content; upload if unseen."""
    data = Path(path).read_bytes()
    digest = content_hash(data)
 
    ledger = load_ledger()
    if digest in ledger:
        return ledger[digest]["file_id"]  # reuse — no upload
 
    uploaded = client.beta.files.upload(
        file=(Path(path).name, data, "application/octet-stream"),
        betas=[BETA],
    )
    ledger[digest] = {"file_id": uploaded.id, "name": Path(path).name}
    save_ledger(ledger)
    return uploaded.id

The crucial point is that the hash is computed over the bytes, not the filename. Judge by filename and you miss the case where the name stays the same but the content changes. Judge by content and, conversely, files with different names but identical content collapse into one.