Is This New Wallpaper Too Close to One We Already Ship? Near-Duplicate Detection with a Perceptual-Hash Pre-Filter and Claude Vision

The two-stage pipeline

The flow looks like this.

1. Compute a perceptual hash for every image in the catalog once, and store it in a ledger.
2. Enumerate only the close candidate pairs by Hamming distance between hashes.
3. For candidate pairs only, send both images together to Claude Vision and get a structured JSON verdict on whether they are essentially the same.
4. Auto-process the clear verdicts; route the gray zone to a human queue with a reason attached.

Step 2 is the crux. If you fail to narrow enough there, the Vision calls in step 3 become an all-pairs comparison and the cost falls apart. Let's go through it in order.

Stage one: narrow candidate pairs with a perceptual hash

There are several perceptual hashes, but the one that was straightforward to implement and suited wallpapers well was dHash (difference hash). You shrink the image to a 9x8 grayscale and compare horizontally adjacent pixels to build 64 bits.

from PIL import Image
 
def dhash(path: str, hash_size: int = 8) -> int:
    # One extra column of width so we can bit-encode horizontal neighbor diffs
    img = (
        Image.open(path)
        .convert("L")
        .resize((hash_size + 1, hash_size), Image.LANCZOS)
    )
    px = list(img.getdata())
    w = hash_size + 1
    bits = 0
    idx = 0
    for row in range(hash_size):
        for col in range(hash_size):
            left = px[row * w + col]
            right = px[row * w + col + 1]
            bits |= (1 if left > right else 0) << idx
            idx += 1
    return bits  # 64-bit integer
 
 
def hamming(a: int, b: int) -> int:
    # Number of differing bits. 0 means identical hash; smaller is more similar.
    return bin(a ^ b).count("1")

A smaller Hamming distance means a closer appearance. In my setup, a threshold of 10 bits or fewer caught the recolored and lightly cropped near-duplicates with almost no misses. That is deliberately loose. Recall you lose in the pre-filter cannot be recovered downstream by Vision, so the filter leans toward "when in doubt, let it through."

Enumerating candidate pairs naively is an all-pairs comparison. Once a catalog grows to a few thousand images, that is a non-trivial amount of computation. I got burned early by running a double loop over every pair. In production I replace it with prefix bucketing on the high bits, or a BK-tree that indexes only the near neighbors by Hamming distance. The idea stays the same either way: pass only the pairs within the threshold to the next stage.

from itertools import combinations
 
def candidate_pairs(hashes: dict[str, int], threshold: int = 10):
    # hashes: {image_id: dhash}
    # Fine for small sets; swap in a BK-tree/bucketing past a few thousand images.
    for (id_a, h_a), (id_b, h_b) in combinations(hashes.items(), 2):
        d = hamming(h_a, h_b)
        if d <= threshold:
            yield (id_a, id_b, d)

Stage two: adjudicate close pairs by sending both images to Claude Vision

Once the candidates are narrowed, I hand each pair to Claude. What matters here is that a single message can carry multiple images. I show both at once and ask whether they are essentially the same wallpaper.

I always want a structured output, so I force a reporting tool with tool_choice and receive it against a schema. That is far more robust than carving JSON out of free text with a regular expression afterward (I organized structured-output validation and repair in Building robust structured output with tool use and JSON Schema).

import base64
import anthropic
 
client = anthropic.Anthropic(api_key="YOUR_API_KEY")
 
def _b64(path: str) -> str:
    with open(path, "rb") as f:
        return base64.standard_b64encode(f.read()).decode()
 
REPORT_TOOL = {
    "name": "report_duplicate",
    "description": "Return a verdict on whether two wallpapers are essentially the same",
    "input_schema": {
        "type": "object",
        "properties": {
            "duplicate": {"type": "boolean"},
            "similarity": {"type": "number", "description": "0.0 to 1.0"},
            "reason": {"type": "string", "description": "The basis for the verdict, in one sentence"},
        },
        "required": ["duplicate", "similarity", "reason"],
    },
}
 
def adjudicate(path_a: str, path_b: str) -> dict:
    msg = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=300,
        system=(
            "You judge duplicates in a wallpaper catalog. "
            "Decide whether two images are essentially the same "
            "(including recoloring, cropping, and minor edits). "
            "Treat genuinely different designs, in composition or motif, as duplicate=false."
        ),
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Are these two essentially the same wallpaper?"},
                {"type": "image", "source": {
                    "type": "base64", "media_type": "image/jpeg", "data": _b64(path_a)}},
                {"type": "image", "source": {
                    "type": "base64", "media_type": "image/jpeg", "data": _b64(path_b)}},
            ],
        }],
        tools=[REPORT_TOOL],
        tool_choice={"type": "tool", "name": "report_duplicate"},
    )
    for block in msg.content:
        if block.type == "tool_use":
            return block.input
    raise RuntimeError("No tool output was returned")

I chose Haiku 4.5 as the judge. Whether two wallpapers are the same is not hard enough to warrant a top-tier model. If the pre-filter has narrowed the candidates well, a fast, inexpensive model is enough here. Sending only the hard cases to a stronger model is the same idea I used last time in Per-class threshold calibration and abstain routing.

How I set the thresholds

Without checking against numbers, a threshold is just a hunch. I hand-labeled about 200 pairs from the catalog as duplicate or not, and calibrated against them.

With the perceptual hash alone, nearly forty percent of the surfaced pairs were "close in distance but a different design." Sky and ocean wallpapers tend to share a light-dark skeleton, and the hash cannot separate them. Inserting Vision here lifted precision from 0.61 to 0.95, and because the pre-filter stays loose, misses barely increased.

Re-checking the regression on the labeled pairs after every calibration became a habit. Tweak the threshold, and watch whether recall has dropped — in numbers. I treat that small extra step as insurance that lets me publish with a clear head.

Cost design that avoids the all-pairs comparison

Because Vision is only called after the pre-filter, cost is essentially set by the number of candidate pairs. That is where the design earns its keep.

Here are the measurements from re-auditing the whole catalog once. About 2,180 images. An all-pairs comparison is roughly 2.375 million combinations, and running Vision over all of them is a non-starter. Passing them through the perceptual-hash pre-filter dropped the candidates to 812 pairs — about 0.03 percent of all pairs. Vision ran only those 812 times.

Had I streamed everything to Vision without a pre-filter, the calls would have ballooned by a factor of thousands, and neither the cost nor the wait would have been realistic. The perceptual hash computes locally in an instant, and once computed a hash is stored in the ledger and reused. In other words, "trim the population with cheap computation, hold the expensive judgment to a minimum" is what caps the cost.

If you are judging at scale, batching the candidate pairs into Message Batches eases both unit price and throughput.

Abstaining on the gray zone, and avoiding false flags on legitimate series

What frightens me about automation is a machine deciding, on its own, the cases that are not black and white. I split similarity into three bands.

Not forcing a verdict on the gray zone was the surest way to prevent wrongful deletions. The human queue carries the reason Vision returned, so a review takes seconds. A single line like "different palette but identical composition" is enough of a foothold to decide.

One more thing I could not do without in production was excluding legitimate variations. Wallpapers include series where I deliberately offer color variants. Those are not duplicates; they are legitimate products. Falsely flagging them risks deleting something I made in earnest. I gave each image a series_id and excluded pairs within the same series from adjudication up front. Before the machine's eye, honor the human context that says "this is a different thing, on purpose." That is a line I try not to cross.

Settling into a weekly incremental run

After the one full sweep, there is no need to re-run everything each time. Comparing only the newly added images against the existing hash ledger is enough.

Matching a batch of 60 new images against 2,180 existing ones is about 130,000 pairs. Through the perceptual-hash pre-filter, the candidates fall to a few dozen, and the Vision verdicts finish in a moment. Since a hash is computed once at add time and appended to the ledger, even the filter's computation is negligible in the incremental run.

Once it settled into this shape, I could finally answer the vague "I feel like I've shipped this before" with numbers. If it is close, it lines up in the human queue with a reason; if it is not, it passes quietly. Moving that judgment from a hunch to a procedure was the real gain.

Wrapping up

An exact hash is airtight for finding identical files, but it cannot measure visual closeness. To catch near-duplicates, a two-stage design — trim the population cheaply with a perceptual hash and leave only the final call to Claude Vision — balanced both cost and accuracy. Calibrate thresholds against labeled pairs, send the gray zone to a human instead of deciding, and exclude legitimate series by context. Hold those three, and even a solo operation can turn its pre-publish duplicate check into a procedure.

Start by hand-labeling about 200 pairs from your own catalog and measuring the precision of the perceptual hash alone. Then watch how far the numbers move when you add one Vision stage on top. Verifying it on your own catalog is, I think, the surest place to begin.

I hope this helps with your own implementation. Thank you for reading.