One Friday night, right after I bumped my support chatbot from Sonnet 4.5 to 4.6, the confirmation phrase "could you share your order number?" silently turned into "could you share your inquiry number?" in roughly 30% of conversations. The prompt itself had not changed by a single character. Two full days passed before a bug report tipped me off. This is the textbook example of a prompt that fails silently.
Unit tests caught nothing. The output format was valid. The JSON parsed. Only the business-critical word "order number" had been quietly swapped for something close enough to look fine.
This article walks through a golden-dataset evaluation pipeline that catches exactly this class of regression in CI — with full code you can adapt today. It is the same pattern I run across four production sites, and I'll include a cost design that keeps the bill under fifty dollars a month.
Why Unit Tests Cannot Protect LLM Quality on Their Own
Traditional unit tests guarantee "for input X, output Y is returned." LLMs are non-deterministic by design, so that contract simply does not hold.
Three moments in particular consistently break prompts in my experience.
The first is bumping the model version. Newer models usually win on average benchmarks, yet they can quietly underperform on specific tasks. A common one is reduced fidelity to "be brief" instructions.
The second is rewording a single sentence in your prompt to make it "clearer". Changing "answer politely" to "answer kindly" can shift the level of formality across thousands of conversations.
The third is changing the surrounding code for tool use or RAG. Even when the prompt is untouched, reordering or trimming retrieval results moves the final output quality just enough to matter.
What unites all three: tests stay green while the user experience erodes. Golden-dataset testing is the discipline for catching that drift quantitatively.
A Minimal Golden Dataset — Start with 50 Items
The phrase "golden dataset" sounds like thousands of labeled rows. In practice 30 to 100 items is enough. For a new project I start with 50.
The trick is that the dataset should compress the actual input distribution your prompt sees in production. For a support chatbot, pull the top 10 categories from the last month of logs (shipping delays, refund requests, how-to questions, and so on) and sample 5 items per category.
JSONL is the practical format. It plays well with Git, is easy to append to, and diffs cleanly.
{"id": "support_001", "category": "shipping_delay", "input": "My order has not arrived yet.", "expected_intent": "shipping_inquiry", "expected_keywords": ["order number", "tracking", "check"], "must_not_include": ["refund", "cancel"]}
{"id": "support_002", "category": "refund", "input": "The size doesn't fit, I'd like to return it.", "expected_intent": "refund_request", "expected_keywords": ["return policy", "refund", "process"], "must_not_include": []}
{"id": "support_003", "category": "usage", "input": "I can't log in to the app.", "expected_intent": "technical_support", "expected_keywords": ["password", "reset", "account"], "must_not_include": ["refund"]}Pairing expected_keywords with must_not_include is the part most people miss. The first lets you express "at least N of these terms must appear," the second lets you assert "these must never appear." Keyword-based matching tolerates wording variation while still catching meaning drift, which strict equality cannot.
Rule-Based Evaluation — Start with the Deterministic Layer
A solid evaluation pipeline is two-tiered: rule-based first, LLM-as-a-Judge second. The cheap layer rejects what it can; the expensive layer only sees what survives.
# evaluator/rule_based.py
import json
from typing import TypedDict
class GoldenItem(TypedDict):
id: str
category: str
input: str
expected_keywords: list[str]
must_not_include: list[str]
class RuleResult(TypedDict):
id: str
keyword_score: float # 0.0 to 1.0
blocklist_passed: bool
overall_passed: bool
failure_reasons: list[str]
def evaluate_rule(item: GoldenItem, output: str, min_keyword_ratio: float = 0.5) -> RuleResult:
"""Rule-based evaluation: expected-keyword ratio + blocklist enforcement."""
output_lower = output.lower()
# 1. What share of expected keywords actually appear?
matched = [kw for kw in item["expected_keywords"] if kw.lower() in output_lower]
keyword_score = len(matched) / max(len(item["expected_keywords"]), 1)
# 2. Any blocklist hit fails the item
blocklist_violations = [kw for kw in item["must_not_include"] if kw.lower() in output_lower]
blocklist_passed = len(blocklist_violations) == 0
failure_reasons = []
if keyword_score < min_keyword_ratio:
missing = set(item["expected_keywords"]) - set(matched)
failure_reasons.append(f"keyword_ratio {keyword_score:.2f} < {min_keyword_ratio} (missing: {missing})")
if not blocklist_passed:
failure_reasons.append(f"blocklist_violation: {blocklist_violations}")
return {
"id": item["id"],
"keyword_score": keyword_score,
"blocklist_passed": blocklist_passed,
"overall_passed": keyword_score >= min_keyword_ratio and blocklist_passed,
"failure_reasons": failure_reasons,
}
# Expected output:
# {'id': 'support_001', 'keyword_score': 1.0, 'blocklist_passed': True, 'overall_passed': True, 'failure_reasons': []}The non-obvious choice here is min_keyword_ratio=0.5. Demanding 100% match treats normal phrasing variation as failure. In my experience 0.5 to 0.6 is the practical sweet spot.
LLM-as-a-Judge — Scoring the Outputs Rules Cannot
Quality dimensions like politeness, factuality, and contextual relevance resist rule-based scoring. This is where you bring in Claude as a judge.
There is a serious pitfall, though: the judge model itself drifts when its version changes. I once "updated" my judge from Sonnet 4.6 to Sonnet 4.6 (same major version, different minor revision) and watched my evaluation scores drop from 0.78 to 0.71 — without changing anything else.
Two countermeasures matter. First, always log the exact judge model and version into every result row. Second, periodically compute Cohen's Kappa between the judge and a human rater (often you).
# evaluator/llm_judge.py
import os
import anthropic
from typing import TypedDict
JUDGE_MODEL = "claude-sonnet-4-6" # always log this in your eval output
class JudgeResult(TypedDict):
id: str
politeness_score: int # 1 to 5
factual_score: int # 1 to 5
relevance_score: int # 1 to 5
judge_reasoning: str
judge_model: str
JUDGE_SYSTEM = """You are an evaluator of customer-support replies.
Score the response on three axes from 1 to 5 and return JSON only.
- politeness: 1=rude / 5=very polite
- factual: 1=contains factual error / 5=no factual errors
- relevance: 1=off-topic / 5=fully captures the user's intent
Be strict. Anchor 3 as 'average'; when in doubt, score lower."""
JUDGE_USER_TEMPLATE = """User input:
{user_input}
Response to evaluate:
{model_output}
Return only JSON in the form {{"politeness": N, "factual": N, "relevance": N, "reasoning": "..."}}."""
def judge(item_id: str, user_input: str, model_output: str) -> JudgeResult:
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
try:
msg = client.messages.create(
model=JUDGE_MODEL,
max_tokens=400,
system=JUDGE_SYSTEM,
messages=[{
"role": "user",
"content": JUDGE_USER_TEMPLATE.format(
user_input=user_input, model_output=model_output
)
}],
)
text = msg.content[0].text.strip()
# strip ```json ... ``` if present
if text.startswith("```"):
text = text.split("```")[1].lstrip("json").strip()
import json
parsed = json.loads(text)
return {
"id": item_id,
"politeness_score": int(parsed["politeness"]),
"factual_score": int(parsed["factual"]),
"relevance_score": int(parsed["relevance"]),
"judge_reasoning": parsed.get("reasoning", ""),
"judge_model": JUDGE_MODEL,
}
except (anthropic.APIError, json.JSONDecodeError, KeyError) as e:
# If the judge itself fails, mark as N/A — do not fail the whole suite
return {
"id": item_id, "politeness_score": -1, "factual_score": -1,
"relevance_score": -1, "judge_reasoning": f"judge_error: {e}",
"judge_model": JUDGE_MODEL,
}
# Expected output:
# {'id': 'support_001', 'politeness_score': 5, 'factual_score': 5, 'relevance_score': 4, 'judge_reasoning': 'Polite, factually clean, order-number flow is clear.', 'judge_model': 'claude-sonnet-4-6'}Why is judge_error not a hard failure? A failed judge call says nothing about the candidate model's quality. Conflating the two means a transient Anthropic-side hiccup turns your CI red. Sentinel -1 values let you count and report N/A separately in the summary without breaking the suite.
Use Cohen's Kappa to Keep the Judge Honest
Cohen's Kappa measures how well two raters (your LLM judge vs. you) actually agree, beyond chance. Once a month, randomly sample 20 items from your golden set and rate them yourself, then compare.
# evaluator/kappa.py
from sklearn.metrics import cohen_kappa_score
from typing import Sequence
def compute_kappa(judge_scores: Sequence[int], human_scores: Sequence[int]) -> dict:
"""Agreement on an ordinal 1-5 scale (politeness, etc.)."""
# ordinal scale -> use quadratic weights
kappa = cohen_kappa_score(human_scores, judge_scores, weights="quadratic")
diffs = [abs(j - h) for j, h in zip(judge_scores, human_scores)]
return {
"kappa": kappa,
"mean_abs_diff": sum(diffs) / len(diffs),
"exact_agreement_rate": sum(1 for d in diffs if d == 0) / len(diffs),
"n_samples": len(diffs),
}
# Expected output:
# {'kappa': 0.72, 'mean_abs_diff': 0.4, 'exact_agreement_rate': 0.65, 'n_samples': 20}
# Treat kappa >= 0.6 as 'usable', >= 0.8 as 'strong agreement'.Decision rule: a kappa under 0.6 is your signal to revise the judge prompt. In my experience the most common cause is using a single ambiguous axis ("overall quality"). Splitting it into politeness, factual, and relevance typically restores agreement.
Designing the Pipeline to Stay Under $50 a Month
"Won't running this in CI for every PR cost a fortune?" is the first concern people raise. Here is how I keep mine cheap.
First, only run the full eval on merges to main. PR builds run rule-based checks only; LLM-as-a-Judge runs on the merge build, in batch.
Second, consider Haiku 4.5 as the judge. I benchmarked it side-by-side with Sonnet 4.6 on my support-chat dataset and the kappa difference was within 0.05, so I switched. Cost dropped roughly 5x.
Third, always use prompt caching. The judge's system prompt is fixed, so cache hit rate exceeds 90%.
The math for my setup:
- 50 golden items × 3 axes = 150 evaluation calls per run
- 5 main merges per day × 30 days = 150 runs
- 22,500 evaluation calls per month
- Haiku 4.5 (input $0.80 / output $4.00 per 1M, effective input $0.10 with cache) at ~600 input + 80 output tokens per call
- Monthly: ($0.10 × 0.6 + $4.00 × 0.08) / 1000 × 22500 ≈ about $8
Add the once-a-month 20-item Sonnet 4.6 run for the kappa check and you still land well under $15.
Wiring It into CI — A Minimal GitHub Actions Workflow
# .github/workflows/prompt-eval.yml
name: Prompt Regression Eval
on:
push:
branches: [main]
pull_request:
paths:
- 'prompts/**'
- 'evaluator/**'
jobs:
eval:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
- run: pip install -r evaluator/requirements.txt
# PRs only run the cheap, deterministic layer
- name: Rule-based eval (PR)
if: github.event_name == 'pull_request'
run: python evaluator/run.py --mode rule_only --golden golden_dataset.jsonl
# main merges run the full pipeline including LLM judge
- name: Full eval (main)
if: github.event_name == 'push'
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: python evaluator/run.py --mode full --golden golden_dataset.jsonl --output eval_results.json
- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: eval-results
path: eval_results.json
# Optional Slack notification on regression
- name: Notify on regression
if: failure() && github.event_name == 'push'
run: |
curl -X POST -H 'Content-type: application/json' \
--data "{\"text\":\"⚠️ Prompt eval regression detected on main. See: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\"}" \
${{ secrets.SLACK_WEBHOOK_URL }}Set timeout-minutes: 15 without exception. A hung judge call or a runaway retry loop will otherwise jam your CI for hours and tank developer velocity.
Five Pitfalls You Will Almost Certainly Hit
These are the traps I personally fell into, in the order they bit me.
(1) Building the golden dataset once and forgetting it. Production input distributions shift constantly. Add 5 to 10 fresh items from production logs each month and prune anything older than six months that no longer reflects reality.
(2) Locking the threshold to your initial score. "We scored 0.85, so 0.85 is the bar" is dangerous. Judge variance alone can drop the next run to 0.83 and turn the whole repository red. Use initial × 0.95 as your starting threshold and refine empirically over three months.
(3) Asking for "1 to 5" and getting 4.7 back. Claude is flexible enough to return decimals even when you ask for integers. Either coerce with int(parsed["politeness"]) or enforce structure via Tool Use.
(4) Reporting only "average score: 0.78" when something fails. That tells you nothing about what regressed. Always emit per-item-ID diffs against the previous run. Persisting eval_results.json to S3 (or any durable store) lets you diff today's run against previous_run.json.
(5) Discarding the judge's reasoning. Without the judge_reasoning field, you cannot iterate on the judge prompt later. It costs a few hundred KB per month — keep it.
A Monthly Review to Keep the System from Rotting
The most important habit is to not treat the pipeline as one-and-done. I block 30 minutes on the first of every month, even when working solo.
I look at the score trend on a dashboard (or a spreadsheet — it does not matter), and I check four things: which cases are trending downward and why, whether Cohen's Kappa is still above 0.6, what new production cases deserve to be added to the golden set, and whether the judge prompt itself needs tightening.
Skip this review and the dataset goes stale within three months. In my own logs, months with the review had roughly half the user-reported issues compared to months I skipped it. The pipeline is something you grow, not something you ship.
A Concrete Walkthrough — Building Your First 30 Items in 30 Minutes
If you have never assembled a golden dataset before, the abstract guidance above can feel daunting. Here is the exact sequence I follow on a fresh project.
I open my support log database and run two queries. The first pulls conversations from the last month where the user sent more than three messages back-to-back without the assistant resolving the request — these are conversations where the bot was probably struggling to understand. The second pulls conversations where a human operator had to take over. Together, these two queries surface the cases where my prompt is most likely failing in production right now.
I copy 15 of each into a working spreadsheet and read them. I am looking for two things: the user's actual intent (which I will record as expected_intent), and three to five terms the ideal response should contain (expected_keywords). I also note any terms that would be wrong — for example, if the user asked about shipping but the response mentioned a refund, that is a clear must_not_include.
The whole exercise takes about 30 minutes for 30 items. The mistake I see teams make is trying to make this perfect. The dataset improves over time, and your first version exists primarily to give you a baseline to compare against. Ship it imperfect, refine it monthly.
The format I use is JSONL because Git diffs read cleanly and appending an item is trivial. CSV breaks the moment a user message contains a newline, which they do constantly in chat. JSON arrays are fine but require rewriting the whole file to add one entry. JSONL gets out of your way.
Edge Cases — When the Standard Recipe Breaks
The two-tier rule-based-then-LLM design covers maybe 80% of teams. Here are the situations where you will need to deviate.
Highly subjective creative tasks. If your product asks Claude to generate marketing copy or fictional dialogue, keyword matching is essentially useless and even the LLM judge struggles. For these I recommend pairwise comparison instead: have the judge compare the new prompt's output to the baseline prompt's output and pick a winner. This sidesteps the absolute-scoring problem entirely. The downside is that you lose interpretability — you know things got worse without knowing exactly why.
Multi-turn conversations. A single-turn evaluation misses the most important failure mode in chatbots: the assistant correctly answers turn 1 but loses context by turn 4. To evaluate multi-turn behavior, store the full conversation history in your golden item and replay it deterministically. The expected output is a function of the entire prior conversation, not just the last user message. This costs more (longer prompts, more tokens) but it catches bugs the single-turn approach simply cannot see.
Outputs with structured side effects. When Claude is calling tools — say, querying a database or sending an email — you need to evaluate the side effects, not just the textual response. The pattern that has worked for me is to mock the tool layer in evaluation mode and assert against the captured tool calls. Your expected_tool_calls field complements expected_keywords, and a single golden item can verify both layers.
Multilingual products. If your product serves users in multiple languages, your golden dataset needs items in each language, in roughly the same proportions as your real traffic. A single English-only dataset will silently miss regressions that affect Japanese or German users. Maintain one combined dataset with a language field, and report scores both globally and per language so a 20% drop in Japanese quality cannot hide behind 80% English traffic.
Domains with confidential data. If the inputs in your dataset contain customer PII, do not commit the dataset to your code repository. I keep these in encrypted S3 buckets that the CI pulls at runtime using a scoped IAM role, with a small public-safe sample committed for documentation. The added operational complexity is worth it; the alternative is a leaked customer record in your Git history forever.
Comparison with Off-the-Shelf Evaluation Frameworks
You might be wondering whether to roll your own pipeline or adopt something like Promptfoo, Inspect AI, or LangSmith. Here is my honest take.
For a single product with a single prompt and fewer than 100 items, the homegrown approach in this article is simpler than learning any of those frameworks. You will spend less time fighting tool conventions and more time iterating on your actual evaluation. The 200 lines of Python you write here will outlive any framework you adopt today.
For multi-team organizations evaluating ten or more prompts, dedicated frameworks start paying off. They give you a shared vocabulary, a UI for non-engineers to inspect failures, and built-in dataset versioning. The switching cost is real but bounded.
The pattern I keep seeing succeed is start homegrown, migrate when team size demands it. The discipline you learn from building this yourself transfers cleanly to any framework, while teams that adopt a framework first often misuse it because they never internalized the underlying ideas.
A Real Regression I Caught — Anatomy of a Win
Let me walk through a specific regression my pipeline caught last quarter, because abstract guidance is far less convincing than a concrete story.
I had been iterating on the system prompt for an internal documentation chatbot. A small change — adding the sentence "If the user's question is ambiguous, ask one clarifying question before answering" — seemed obviously beneficial. It shipped to staging on a Wednesday afternoon.
The Wednesday-night main-merge build ran the full eval and the politeness score dropped from a steady 4.6 average to 4.1, with five specific items flagged. I opened the judge reasoning logs for those items and the pattern was immediate: the bot was now asking clarifying questions even when the user's intent was perfectly clear, which read as condescending rather than helpful.
Without the pipeline, I would not have noticed for at least a week. The bot still answered every question. JSON parsed. No errors logged. Just a slow erosion of how it felt to interact with the bot. Catching it in the next CI run cost me 30 minutes of investigation; catching it from a customer complaint would have cost a week of slow attrition.
The fix was to constrain the new instruction: "If the user's question is ambiguous and you cannot give a useful answer with reasonable assumptions, ask one clarifying question." That single qualifier restored the politeness score to 4.5 and avoided the over-cautious behavior. The next eval run confirmed it green, and I shipped on Thursday morning.
This is the loop you are building. Catch the regression in CI, inspect the per-item failures, hypothesize a fix, run it again. The pipeline does not replace judgment — it gives your judgment a faster feedback loop.
What This Approach Does Not Solve
It would be dishonest to pitch golden-dataset testing as a complete solution. Here are its limits, so you can plan for what it leaves uncovered.
It does not detect novel failure modes. By definition, a golden dataset can only catch regressions in patterns you have already encoded. A brand-new category of input — say, a flood of users asking about a feature you just launched — will land in production untested. Pair this pipeline with production sampling: log a random 1% of real conversations, run the same evaluation against them weekly, and feed surprising failures back into the dataset.
It does not measure user satisfaction directly. A response can score 5 across all three judge axes and still be the wrong shape for a particular user — too long when they wanted brief, too cautious when they wanted a recommendation. To close that gap, instrument your product to capture explicit thumbs-up and thumbs-down signals and reconcile them against your eval scores quarterly. Big divergences mean your judge is measuring the wrong thing.
It does not catch latency or cost regressions. A prompt change that makes the model think harder might keep accuracy steady while doubling p95 latency. Add timing and token-count assertions to the same pipeline. The simplest version is just assert tokens_used < baseline * 1.2 per item and assert latency_ms < 5000. Free, deterministic, and catches the boring regressions that compound into real problems.
It does not protect against safety regressions. Toxicity, jailbreaks, and policy violations need their own dedicated red-team dataset that you maintain separately. The same pipeline can run it, but the items belong in a different file and the failure handling is different — a single hit is an emergency, not a trend.
Getting Team Buy-In Without Drama
The technical pipeline is the easy part. The harder challenge is convincing your team that an extra 90 seconds on every main-merge build is worth it.
The argument that has worked for me is to run the pipeline silently for two weeks first, with results posted to a private dashboard, before turning on the failure-blocking step. After two weeks you will have at least one or two real regressions caught, and you can show the team a concrete "this is what we would have shipped without it." Abstract arguments lose; specific examples win.
When you do flip the switch on blocking failures, set the threshold loose initially. A pipeline that fires false alarms in its first week will be turned off by a frustrated developer at 2 AM, and you will not get a second chance to make a first impression. Better to start at "block only if score drops more than 10%" and tighten over a few months as everyone trusts the signal.
Document the override path. There will be times when the team genuinely needs to ship a change despite a regression — usually because the regression is in a niche case while the change unblocks a major customer. Make this a conscious one-line decision (SKIP_EVAL=true with a required reason in the commit message) rather than something developers learn by figuring out which env var disables CI. The override is not a bug in the system; it is a feature that prevents the system from being routed around.
What to Do Next
Start by writing 30 golden items. Aiming for "the perfect dataset" is how this never gets done. Pull 15 conversations where the user retried, plus 15 where a human had to step in, from your last month of logs. That is a 30-minute task you can do tomorrow morning.
Then put the rule-based layer into CI first. The LLM judge can wait until the cheap layer is humming. Trying to ship the full stack on day one is how teams burn out and shelve the whole effort.
For complementary reading, see Building an LLM Evaluation Pipeline with the Claude API, A Testing Strategy for the Claude API — Unit, Integration and E2E for AI Quality, and Production Prompt Registry, Versioning and A/B Testing on the Claude API. Together they form a triangle of evaluation, versioning, and testing that is hard to beat.