Catching Claude Quality Regressions With an Eval Harness

Building a golden dataset, minimally

The foundation of an eval is a golden dataset. You don't need perfect expected outputs. It's far more practical to attach a "pass condition" — a rubric — to each input. I made just 40 cases in the first week, then added one each time an incident or complaint appeared, growing it to 200 over six months.

# golden.jsonl — one case per line. "expected" is a rubric, not a string match.
# {"id": "cat-012", "category": "refund", "input": "...", "rubric": ["classified as refund", "states a one-sentence reason", "no PII in output"]}
 
import json
from pathlib import Path
 
def load_golden(path: str) -> list[dict]:
    cases = []
    for line in Path(path).read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if line:
            cases.append(json.loads(line))
    return cases

The key is to break each rubric into items you can judge objectively as YES/NO. Vague criteria like "should be polite" produce wobbly judgments. Driving items down to a grain where humans wouldn't disagree — "states a reason in one sentence," "uses no category name outside the allowed list" — was the single biggest lever for the judge-agreement rate below.

The layer that runs the candidate prompt

The thing under test — your system prompt plus how you assemble user input — should travel the same code path as production. Writing a separate implementation just for evals invites the worst kind of divergence: it passes in eval and breaks in prod. Import the production function and run it directly.

import anthropic
 
client = anthropic.Anthropic()
CANDIDATE_MODEL = "claude-sonnet-4-6"
 
def run_candidate(system_prompt: str, user_input: str) -> str:
    resp = client.messages.create(
        model=CANDIDATE_MODEL,
        max_tokens=1024,
        temperature=0,            # fixed for reproducible evals
        system=system_prompt,
        messages=[{"role": "user", "content": user_input}],
    )
    return resp.content[0].text

Taming LLM-as-judge drift with rubrics

Grading 200 cases by hand will break your spirit. Using Claude itself as the judge is the realistic option, but left alone a judge drifts both lenient and harsh. My first judge agreed with human grading only 68% of the time — not enough to trust.

Three changes lifted agreement to 91%. First, I stopped asking for a free-form score and had it return true/false per rubric item as structured output. Second, I required the judge to write a reason for each item and to emit the reason before the verdict (let it conclude first and it rationalizes leniently afterward). Third, I pinned the judge to a stronger model than the candidate — Opus — keeping candidate and judge on separate lineages.

JUDGE_MODEL = "claude-opus-4-6"
 
def judge(user_input: str, output: str, rubric: list[str]) -> dict:
    rubric_lines = "\n".join(f"{i+1}. {r}" for i, r in enumerate(rubric))
    judge_system = (
        "You are a strict grader. For each rubric item, first write a "
        "one-sentence reason, then decide true/false. When in doubt, choose "
        "false. Output only the specified JSON."
    )
    judge_user = (
        f"# Input\n{user_input}\n\n# Candidate output\n{output}\n\n"
        f"# Rubric\n{rubric_lines}\n\n"
        '# Format\n{"items":[{"reason":"...","pass":true}], "notes":"..."}'
    )
    resp = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=1024,
        temperature=0,
        system=judge_system,
        messages=[{"role": "user", "content": judge_user}],
    )
    data = json.loads(resp.content[0].text)
    items = data["items"]
    passed = sum(1 for it in items if it["pass"])
    return {"score": passed / len(items), "items": items, "notes": data.get("notes", "")}

Using a stronger model as judge is a debatable call. It costs more, but if the judge drifts the entire eval becomes meaningless, so this is one place I prefer not to economize.

Comparing against a baseline to flag regressions

A score alone doesn't tell you whether things are good or bad. Save the score of the last passing commit as baseline.json and judge by the delta. Looking not just at the average but at the stratified minimum and the rise in worst cases is the countermeasure for the opening incident.

import statistics
 
def evaluate(system_prompt: str, cases: list[dict]) -> dict:
    per_case, per_category = [], {}
    for c in cases:
        out = run_candidate(system_prompt, c["input"])
        r = judge(c["input"], out, c["rubric"])
        per_case.append({"id": c["id"], "category": c.get("category", "-"), "score": r["score"]})
        per_category.setdefault(c.get("category", "-"), []).append(r["score"])
    scores = [p["score"] for p in per_case]
    return {
        "mean": round(statistics.mean(scores), 4),
        "p10": round(sorted(scores)[max(0, len(scores) // 10 - 1)], 4),
        "fails": [p for p in per_case if p["score"] < 0.6],
        "by_category": {k: round(statistics.mean(v), 4) for k, v in per_category.items()},
    }
 
def detect_regression(baseline: dict, candidate: dict) -> list[str]:
    alerts = []
    if candidate["mean"] < baseline["mean"] - 0.02:          # mean dropped >= 2pt
        alerts.append(f"mean drop: {baseline['mean']} -> {candidate['mean']}")
    if candidate["p10"] < baseline["p10"] - 0.05:            # bottom 10% worsened
        alerts.append(f"bottom band worse: p10 {baseline['p10']} -> {candidate['p10']}")
    for cat, sc in candidate["by_category"].items():
        base = baseline["by_category"].get(cat)
        if base is not None and sc < base - 0.05:            # category degraded
            alerts.append(f"category '{cat}' degraded: {base} -> {sc}")
    return alerts

Thresholds like -0.02 and -0.05 have no universal correct value. I started loose (mean -0.05) and tightened only after the judge agreement rose and measurement stabilized. Tightening thresholds while measurement is still noisy buries you in false regression alerts. Tightening in stages was the right operational answer.

Wiring it into CI to stop a bad push

Finally, make this a CI step that returns a non-zero exit code on regression. Run it only on PRs that touch the prompt file or the model spec, and both cost and frequency stay low.

import sys
 
def main():
    cases = load_golden("eval/golden.jsonl")
    system_prompt = Path("prompts/classify.txt").read_text(encoding="utf-8")
    candidate = evaluate(system_prompt, cases)
    baseline = json.loads(Path("eval/baseline.json").read_text(encoding="utf-8"))
    alerts = detect_regression(baseline, candidate)
    print(json.dumps(candidate, ensure_ascii=False, indent=2))
    if alerts:
        print("Regression detected:", file=sys.stderr)
        for a in alerts:
            print("  - " + a, file=sys.stderr)
        sys.exit(1)
    Path("eval/candidate.json").write_text(json.dumps(candidate, ensure_ascii=False))
    print("No regression. Wrote a baseline-promotion candidate to candidate.json.")
 
if __name__ == "__main__":
    main()

Once it passes and a human reviews, promote candidate.json to baseline.json. I deliberately avoid auto-updating the baseline: it prevents the "boiled frog" effect where tiny degradations accumulate and drag the standard itself down. Always keeping one human judgment in front of a baseline update is a line I think is worth holding.

The numbers, and how to decide

In my setup, one eval — 200 golden cases run on Sonnet and graded by Opus — takes 4–6 minutes and costs roughly $0.5–0.8. Run per PR, that's negligible. Since adopting it, I've stopped three "one category quietly degraded" cases before they shipped — none of which my local spot-checks would have caught.

If judge cost worries you, a two-stage approach also worked: grade everything with a cheaper pass, then re-judge with Opus only the items flagged false. But two stages add complexity, so I'd start with straightforward full grading and add the second stage only once cost becomes a real problem.

In 1997, at sixteen, when I first got online, I was captivated by watching code written by strangers around the world run on my own screen. What hasn't changed since is the joy of something I built continuing to work as intended. A Claude-powered feature is the same: just having a foundation that measures quality on every change lets you keep making changes with confidence.

Start with 40 golden cases and a single judge like this one. You don't need to build a perfect eval foundation in one go. Add one case each time something breaks, and six months from now you'll have an asset quietly guarding your production quality.