When the RAG Started Being Confidently Wrong — Field Notes on Measuring Retrieval Misses With Groundedness

Put Claude in the grader's seat to measure groundedness

Grading groundedness by hand doesn't last. So I stood up Claude as a grader in a call separate from generation, judging whether the answer is supported by the retrieved documents. The grading prompt is short, and the verdict comes back structured.

// eval/groundedness.ts
import Anthropic from '@anthropic-ai/sdk';
const client = new Anthropic();
 
type Verdict = { grounded: boolean; supported_claims: number; unsupported_claims: number };
 
export async function scoreGroundedness(
  answer: string,
  retrievedDocs: string[],
): Promise<Verdict> {
  const context = retrievedDocs.map((d, i) => `[doc ${i + 1}]\n${d}`).join('\n\n');
  const res = await client.messages.create({
    model: 'claude-sonnet-5',
    max_tokens: 300,
    // Grading is a separate process from generation. Do not fill gaps with outside knowledge.
    system:
      'Judge, using only the given documents, whether each claim in the answer is supported. ' +
      'Claims absent from the documents count as unsupported. Return JSON only.',
    messages: [
      {
        role: 'user',
        content:
          `# Retrieved documents\n${context}\n\n# Answer\n${answer}\n\n` +
          'Reply as {"grounded": boolean, "supported_claims": number, "unsupported_claims": number}.',
      },
    ],
  });
  const text = res.content[0].type === 'text' ? res.content[0].text : '{}';
  return JSON.parse(text.slice(text.indexOf('{'), text.lastIndexOf('}') + 1));
}

The key is telling the grader not to fill gaps with outside knowledge. Leave that ambiguous and Claude will defend the answer with its own general knowledge, and groundedness reads too lenient. Taking the verdict as JSON — down to counts of supported and unsupported claims — lets you later trace which answers drifted, and by how much.

For retrieval hit rate, I fixed a dataset of about 120 items — each a question paired with the ID of the document that holds its answer — and mechanically counted whether the correct ID appeared in the results. No special machinery. It just rides along in the daily batch.

Make groundedness one daily number

End the measurement as a one-off investigation and the decay quietly returns. I fixed groundedness and retrieval hit rate as two numbers I check on the dashboard each morning. As an indie developer handing daily work across several sites to a model, I find instruments like these are what let me keep the overnight runs going without worry. I hand even AdMob-revenue support copy to the model, so when answer correctness breaks, it hits real operations directly.

Following the numbers, I worked down from the top causes.

The biggest lever was re-chunking. With every document I added, long heading-sized chunks had accumulated, each holding several topics at once. Their embeddings averaged out into vectors that were vaguely near every question and decisive for none. Just cutting to around 512 tokens raised the retrieval hit rate by 14 points.

Threshold drift is easy to miss too. When I swapped the embedding model for a new version, I reused the old cosine-similarity threshold as-is. The score distributions differ between versions, so the cutoff effectively tightened and started rejecting the answer document.

// retrieve.ts — calibrate the threshold from the distribution (don't reuse a fixed value)
export function calibrateThreshold(scores: number[], keepRatio = 0.3): number {
  const sorted = [...scores].sort((a, b) => b - a);
  const idx = Math.floor(sorted.length * keepRatio);
  return sorted[idx] ?? 0; // the position that keeps the top 30% becomes the dynamic threshold
}

Don't reuse a fixed threshold after an embedding-model update. That's the kind of lesson that only sticks once it has stung you.

Bake "if it isn't grounded, don't answer" into generation

Fixing retrieval brings groundedness up a little behind the hit rate. The last push was on the generation side: stop Claude from patching holes with general knowledge when the retrieved docs don't hold the answer. In the prompt, make it say "I don't know" outright when there's no support.

// system prompt in generate.ts (excerpt)
const system = `You answer only from the retrieved documents.
- State only claims that the documents support.
- If the documents do not contain the answer, do not guess; say "the provided materials contain no relevant statement."
- End each claim with the [doc n] it relied on.`;

Requiring a [doc n] on each claim stabilizes the groundedness grading and makes answers easier for users to verify. A RAG that can honestly say "I don't know" is far more trustworthy than one that lies fluently. I've come to prefer the honesty of drawing the boundary correctly over the fluency of filling the gap.

The recovery order on one page

If I met the same situation again, this is the sequence I'd follow.

1. Don't trust how the answer looks. Treat fluency and correctness as separate things.
2. Doubt it as "retrieval hit rate" and "groundedness." Isolate retrieval-side vs generation-side first.
3. Put Claude in the grader's seat and make groundedness one daily number.
4. Work down from chunk bloat, threshold drift, and embedding updates.
5. Calibrate the threshold from the distribution. Don't reuse a fixed value after an update.
6. Bake "if it isn't grounded, don't answer" into generation, and require a [doc n] on each claim.

A RAG isn't done when you build it; it earns trust only when you keep measuring whether it still grabs the answer. There's nothing flashy about it, but this quiet instrument is what steadily illuminates the decay hidden behind fluent prose. I hope it helps in your own build. Thank you for reading.