Don't Trust the Confidence Score: Per-Class Calibration and Abstain Routing for Vision Classification
Overall accuracy looked fine while individual categories quietly collapsed. Here is how I calibrated Claude Vision's self-reported confidence per class and routed abstentions to a human queue.
When the RAG Started Being Confidently Wrong — Field Notes on Measuring Retrieval Misses With Groundedness
In a Claude API RAG, the answers stay fluent while the facts drift. Often the cause is a silent recall decay on the retrieval side, missing the document that holds the answer. Field notes on measuring groundedness and retrieval hit rate and walking the system back, with working code and real numbers.
Stop Claude API Prompt Regressions with Golden-Dataset Testing
A complete production guide to catching the silent quality drift that hits Claude API prompts when models or prompts change — using golden datasets and LLM-as-a-Judge wired into CI.
Building LLM Evaluation Pipelines with Claude API — Claude-as-Judge, Prompt A/B Testing, and Quality Scoring Patterns
Designing and implementing LLM evaluation pipelines on the Claude API — Claude-as-Judge, prompt A/B testing, quality scoring, and regression testing for production applications.