◉CLAUDE LABJP
●MODEL — Claude Fable 5.1 and Claude Mythos 5.1 landed on September 1. They are the same underlying model; only the strength of the safeguards differs●PRICING — Per-token rates hold at $10/$50 per MTok. What changed is cache reads, cut 75% to $0.25 per MTok●COST — How much that saves depends on your workload: roughly 25% for typical use, up to about 45% for context-heavy agentic work. Worth measuring your own split before quoting a number●BENCH — Terminal-Bench-Science 0.1 climbs from 24.7% on Fable 5 to 52.6%. Anthropic also states a standard error of 3.5-4.5 points, which is worth remembering before reading small gaps as real●SAFEGUARDS — Sharper cyber safeguards cut interventions in Claude Code sessions by roughly 60% on average. Finding vulnerabilities is now allowed; developing exploits still is not●API — New API accounts created from today can no longer edit prior context while preserving Claude's thinking transcript. It is an anti-distillation measure, and existing accounts are unaffected for now●MODEL — Claude Fable 5.1 and Claude Mythos 5.1 landed on September 1. They are the same underlying model; only the strength of the safeguards differs●PRICING — Per-token rates hold at $10/$50 per MTok. What changed is cache reads, cut 75% to $0.25 per MTok●COST — How much that saves depends on your workload: roughly 25% for typical use, up to about 45% for context-heavy agentic work. Worth measuring your own split before quoting a number●BENCH — Terminal-Bench-Science 0.1 climbs from 24.7% on Fable 5 to 52.6%. Anthropic also states a standard error of 3.5-4.5 points, which is worth remembering before reading small gaps as real●SAFEGUARDS — Sharper cyber safeguards cut interventions in Claude Code sessions by roughly 60% on average. Finding vulnerabilities is now allowed; developing exploits still is not●API — New API accounts created from today can no longer edit prior context while preserving Claude's thinking transcript. It is an anti-distillation measure, and existing accounts are unaffected for now
TAG

evaluation

4 articles
← Back to all tags
Related:
Claude API2claude-api2prompt-engineering2production2Vision1Classification1Indie Dev1RAG1vector search1retrieval augmented generation1groundedness1TypeScript1
⬡ API & SDK/2026-07-10Advanced

Don't Trust the Confidence Score: Per-Class Calibration and Abstain Routing for Vision Classification

Overall accuracy looked fine while individual categories quietly collapsed. Here is how I calibrated Claude Vision's self-reported confidence per class and routed abstentions to a human queue.

⬡ API & SDK/2026-07-09Advanced

When the RAG Started Being Confidently Wrong — Field Notes on Measuring Retrieval Misses With Groundedness

In a Claude API RAG, the answers stay fluent while the facts drift. Often the cause is a silent recall decay on the retrieval side, missing the document that holds the answer. Field notes on measuring groundedness and retrieval hit rate and walking the system back, with working code and real numbers.

⬡ API & SDK/2026-05-01Advanced

Stop Claude API Prompt Regressions with Golden-Dataset Testing

A complete production guide to catching the silent quality drift that hits Claude API prompts when models or prompts change — using golden datasets and LLM-as-a-Judge wired into CI.

⬡ API & SDK/2026-03-27Advanced

Building LLM Evaluation Pipelines with Claude API — Claude-as-Judge, Prompt A/B Testing, and Quality Scoring Patterns

Designing and implementing LLM evaluation pipelines on the Claude API — Claude-as-Judge, prompt A/B testing, quality scoring, and regression testing for production applications.