Precision Lives in the Second Stage: Reranking a Personal Knowledge Search with Claude

Implementation, Part 1: Gather Candidates Roughly with Embeddings

The first stage depends on your embedding model and store, but from the second stage's view it is merely "something that returns a list of candidates." So we fix only that interface first. Anthropic does not ship an embedding model, so this part is left to your own choice. For the concrete way to store vectors and search neighbors with pgvector, I will defer to "Running persistent memory in production with Claude and pgvector" and focus here on what happens after the candidates arrive.

from dataclasses import dataclass
 
@dataclass
class Candidate:
    id: str
    text: str
    score: float  # first-stage similarity (higher = closer)
 
def recall_candidates(query: str, top_n: int = 20) -> list[Candidate]:
    """Gather a rough top_n via embedding search.
    Interface is agnostic to the embedding model or store (pgvector, etc.)."""
    vec = embed(query)                          # your embedding function
    rows = vector_store.search(vec, k=top_n)    # cosine top_n
    return [Candidate(id=r.id, text=r.text, score=r.similarity) for r in rows]

top_n is 20 to leave room for the answer-bearing passage to always be "inside the candidate set." Narrow this to 3 and no matter how smart the second stage is, it cannot pick what was never a candidate. Misses can only be prevented in the first stage, so recall stays wide.

Implementation, Part 2: Reorder Candidates with Claude

The second stage has two crux points. One is to receive the scoring in a reliably structured form. The other is to pass all candidates in a single request. Score them one at a time and the rubric drifts between calls, the order becomes unstable, and cost climbs with every round trip; at 20 candidates that is 20x the round trips, which shows up directly in production latency and spend. Show all candidates at once and Claude can score them on a consistent scale while comparing them against each other.

For structured receipt we use tool use and force the tool with tool_choice, which guarantees the output is JSON in the given schema. For the full discipline including schema validation and repair, "Guarding Claude API structured output with a schema-validate-and-repair loop" goes deeper.

import os
from anthropic import Anthropic
 
client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
 
RERANK_TOOL = {
    "name": "rank_candidates",
    "description": "Score how precisely each candidate answers the question, 0.0 to 1.0",
    "input_schema": {
        "type": "object",
        "properties": {
            "rankings": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "id": {"type": "string"},
                        "relevance": {"type": "number"},  # 0.0-1.0
                        "reason": {"type": "string"},      # why this score
                    },
                    "required": ["id", "relevance", "reason"],
                },
            },
        },
        "required": ["rankings"],
    },
}
 
def rerank(query: str, cands: list[Candidate], model: str) -> list[dict]:
    # Batch candidates into one request; per-item calls drift the rubric and add round trips.
    passages = "\n\n".join(f"[{c.id}]\n{c.text}" for c in cands)
    msg = client.messages.create(
        model=model,
        max_tokens=1024,
        tools=[RERANK_TOOL],
        tool_choice={"type": "tool", "name": "rank_candidates"},
        messages=[{
            "role": "user",
            "content": (
                f"Question: {query}\n\n"
                "Score the following candidates by how precisely they answer the question. "
                "Judge whether it directly answers the question, not whether the paraphrase is similar.\n\n"
                f"{passages}"
            ),
        }],
    )
    block = next(b for b in msg.content if b.type == "tool_use")
    ranked = block.input["rankings"]
    ranked.sort(key=lambda r: r["relevance"], reverse=True)
    return ranked

The key is spelling out in the prompt: judge whether it directly answers, not whether the paraphrase is similar. Without this, Claude too is pulled toward the same topical closeness as the embedding, and adding a second stage loses its point. Fixing the scoring measure in words is what gives the second stage its value.

Placed beside a naive first-stage-only implementation, the difference collapses into a few lines.

# Before: return the embedding top-3 as-is (topically close, but sometimes not the answer)
def search_naive(query: str) -> list[Candidate]:
    return recall_candidates(query, top_n=3)
 
# After: gather broadly, then let Claude reorder by answerability
def search_two_stage(query: str, model: str) -> list[dict]:
    cands = recall_candidates(query, top_n=20)
    return rerank(query, cands, model=model)

When You Are Not Confident, Do Not Answer

The nice part of adding a second stage is not only better ordering. Because the score comes back as a number, you can notice when "there simply is no good candidate." If the top relevance does not reach a threshold, return "no answer" instead of forcing the top of the list. This abstention prevents confidently wrong answers on searches you are not sure about.

ABSTAIN = 0.55  # if the top score is below this, return "no answer"
 
def search(query: str, top_k: int = 3,
           model: str = "claude-haiku-4-5-20251001") -> dict:
    cands = recall_candidates(query, top_n=20)
    if not cands:
        return {"sources": [], "reason": "no candidates found"}
    ranked = rerank(query, cands, model=model)
    if ranked[0]["relevance"] < ABSTAIN:
        # Close in meaning but not the answer. Do not force the top of the list.
        return {"sources": [], "reason": "no candidate met the confidence bar"}
    by_id = {c.id: c for c in cands}
    picked = [r for r in ranked[:top_k] if r["relevance"] >= ABSTAIN]
    return {
        "sources": [
            {"id": r["id"], "text": by_id[r["id"]].text,
             "relevance": r["relevance"], "reason": r["reason"]}
            for r in picked
        ],
    }

What it returns is not the answer itself but the supporting passages and "why each was chosen." Even if you generate an answer downstream, choosing precise evidence first reduces hallucination later. For how retrieval misses quietly erode downstream quality, "Measuring silent recall decay in RAG" has a closer record.

The threshold 0.55 is a starting point; prepare a few dozen questions with known answers and nudge it toward a value that abstains neither too often nor too rarely.

Cost and Precision Trade-off

The second stage adds one Claude call per query. Since candidates are batched, the number of calls does not grow, but input tokens scale with the total size of the candidates. Holding a rough estimate makes the adoption decision concrete.

If 20 candidates average 300 tokens, that is about 6,000 tokens of body alone. Add the question and tool scaffolding and a query lands around 6,000 to 7,000 input tokens, with scoring output around 800 tokens for 20 items at ~40 tokens each. These are design estimates, not measured figures, but they are enough to grasp the order of magnitude. Prices change, so check Anthropic's current rates on the pricing page.

For myself, I settled on defaulting to Haiku 4.5 and raising only the hard categories where abstention keeps happening to Sonnet 5. The second stage is easier to sustain on a personal budget as "smart only when needed" than "always smart."

Summary

What embedding search returns is topical closeness, and what we want is answerability. The two are close, yet genuinely different. Simply splitting the roles into a first stage that gathers roughly and a second stage that reselects by answerability changes how search feels, plainly.

As a next step, widen your current search's top_n to 20 and slot this rerank function in behind it. With an abstain threshold in place, the count of confidently wrong answers on unsure searches should be the first thing to fall. I am still mid-tuning myself, but the situations where this small move of adding a second stage pays off are more common than I expected. Thank you for reading.