Don't Trust the Confidence Score: Per-Class Calibration and Abstain Routing for Vision Classification

Before: A Flat 0.80 Cutoff

Here is what I shipped first.

# before.py - one global threshold
THRESHOLD = 0.80
 
def route(prediction: dict) -> str:
    """prediction = {"category": "animal", "confidence": 0.87}"""
    if prediction["confidence"] >= THRESHOLD:
        return "auto_accept"
    return "manual_review"

Two unstated assumptions hide in those four lines.

First, that confidence carries the same meaning across classes. Second, that precision is a single number worth talking about. Neither held up once I measured.

Running route() over the 412 images and computing precision only on what was auto-accepted:

In hindsight, "animal" was simple to explain. The model confidently calls a plush toy or a cartoon character an animal. Honestly, a human would half-agree. The category definition was the vague part, not the model.

Before touching calibration, I tightened the definition. Adding "photographic living creatures only; exclude plush toys, illustrations, and characters" to the prompt lifted animal precision from 0.742 to 0.861 on its own.

Calibration is not a tool for papering over a bad prompt. Fix definitional vagueness with definitions, then use thresholds to cut the errors that remain. Do it in the other order and you will spend weeks nudging a threshold that was never the problem.

After: Searching for a Per-Class Threshold

With definitions tightened, find the lowest threshold per class that still meets the precision target. The word "lowest" is the whole point. Raising a threshold raises precision but drops coverage, and every dropped image costs you human minutes.

# calibrate.py - per-class threshold search
from collections import defaultdict
from dataclasses import dataclass
 
@dataclass
class Sample:
    predicted: str
    truth: str
    confidence: float
 
MIN_SUPPORT = 20           # lower bound for a "thick" class
FALLBACK_THRESHOLD = 0.93  # shared line for thin classes
GRID = [round(0.50 + 0.01 * i, 2) for i in range(50)]  # 0.50 to 0.99
 
def _precision_at(samples: list[Sample], t: float) -> tuple[float, int]:
    accepted = [s for s in samples if s.confidence >= t]
    if not accepted:
        return 0.0, 0
    hits = sum(1 for s in accepted if s.predicted == s.truth)
    return hits / len(accepted), len(accepted)
 
def calibrate(
    samples: list[Sample],
    target_precision: float = 0.95,
    min_coverage: int = 5,
) -> dict[str, float]:
    """Return the lowest threshold per predicted class that meets the target."""
    by_class: dict[str, list[Sample]] = defaultdict(list)
    for s in samples:
        by_class[s.predicted].append(s)
 
    thresholds: dict[str, float] = {}
    for cls, group in by_class.items():
        if len(group) < MIN_SUPPORT:
            thresholds[cls] = FALLBACK_THRESHOLD
            continue
 
        chosen = None
        for t in GRID:  # ascending scan; stop at the first point that qualifies
            prec, n = _precision_at(group, t)
            if n >= min_coverage and prec >= target_precision:
                chosen = t
                break
        # A class that never reaches the target gives up on auto-acceptance
        thresholds[cls] = chosen if chosen is not None else 1.01
 
    return thresholds

The line returning 1.01 is, to me, the most important one in the file.

When a class simply cannot reach the target, you could pin it at 0.99 and tell yourself "nothing will pass anyway." But then the boundary depends on floating-point luck, and one day a 0.995 prediction slips through. Declaring unreachability with a value that cannot be exceeded keeps the intent in the code where the next reader will find it.

I use the same instinct when validating structured output, which I described in schema validation and repair loops for structured output. Never express an impossible state as a value that merely happens not to occur.

After calibration:

"Nature" is accurate well below 0.80. "Animal" needs 0.94 before it clears the bar. A flat 0.80 was wasting human review on the former and shipping errors from the latter.

Across the whole set, coverage fell from 83% to 76% while precision on auto-accepted images rose from 0.918 to 0.957. Seven points of coverage bought roughly half the errors.

Where Abstentions Go

Anything under the line lands in manual_review. Make that a "look at it later" folder and it will fill up and die. I know because that is exactly what I did first.

What I run now converts the reason for abstaining directly into a work instruction, and stacks it in a prioritized queue.

# route.py - routing with a reason attached
from typing import Literal, TypedDict
 
class Prediction(TypedDict):
    category: str
    confidence: float
 
class Decision(TypedDict):
    action: Literal["auto_accept", "manual_review"]
    reason: str
    priority: int  # lower runs first
 
def route(pred: Prediction, thresholds: dict[str, float]) -> Decision:
    cls = pred["category"]
    t = thresholds.get(cls, FALLBACK_THRESHOLD)
    c = pred["confidence"]
 
    if c >= t:
        return {"action": "auto_accept", "reason": "above_threshold", "priority": 9}
 
    if t > 1.0:
        # Unreachable class: calibration cannot save it. The definition needs work.
        return {"action": "manual_review", "reason": "class_not_calibratable", "priority": 1}
 
    if t - c <= 0.05:
        # So close a human decides in seconds
        return {"action": "manual_review", "reason": "near_threshold", "priority": 2}
 
    return {"action": "manual_review", "reason": "low_confidence", "priority": 5}

Anything landing at priority: 1 is not classification work. It is homework about the category definition. Mix it into the general queue and you lose the signal that says "this category makes me hesitate every single time." Keeping the reason codes separate means I can pull only class_not_calibratable on a weekend and rewrite the definition sentence in the prompt.

In production, near_threshold accounted for 38% of all abstentions. Those decisions are fast — batched together they take three or four seconds each. Splitting by priority alone changed how heavy the review work felt.

Calibration Rots Quietly

The thing I most underestimated: thresholds are perishable goods.

Swap the model. Edit one line of the prompt. Let the incoming image mix shift. Each of those moves the confidence distribution. Leave the threshold sitting on the old distribution and precision drifts down without a sound.

So I freeze the calibration set and re-run the same 412 images weekly.

# drift_check.py - detect calibration decay
def check_drift(
    samples: list[Sample],
    thresholds: dict[str, float],
    target_precision: float = 0.95,
    tolerance: float = 0.02,
) -> list[str]:
    """Hold thresholds fixed and re-measure precision today."""
    alerts = []
    by_class: dict[str, list[Sample]] = defaultdict(list)
    for s in samples:
        by_class[s.predicted].append(s)
 
    for cls, group in by_class.items():
        t = thresholds.get(cls, FALLBACK_THRESHOLD)
        prec, n = _precision_at(group, t)
        if n == 0:
            alerts.append(f"{cls}: zero auto-accepts. Threshold may be too high.")
            continue
        if prec < target_precision - tolerance:
            alerts.append(
                f"{cls}: precision {prec:.3f} < {target_precision - tolerance:.3f} "
                f"(n={n}, threshold={t}) -> recalibration needed"
            )
    return alerts

I deliberately do not recalibrate automatically. If thresholds move on their own, you lose the moment where you would have asked why precision fell — a model change, or a shift in what people are uploading. Emit the alert, confirm the cause, then recalibrate by hand. That small friction pays for itself.

Treating a frozen validation set as a regression test is the same idea I use in golden dataset regression testing for prompts and the LLM evaluation pipeline. Classifier calibration belongs on that same shelf; it is easier to manage as one more form of evaluation.

It earned its keep the week I updated the model. "Space" tripped the alert at precision 0.911. Digging in, the new model was more eager to call gradient wallpapers "space," and its confidence distribution had shifted about 0.03 higher overall. Raising the threshold from 0.93 to 0.96 settled it. Without the alert, several hundred images would have shipped under the wrong category.

When This Is Worth the Trouble

Honestly, sometimes it is not.

In my case, wallpaper categories are something users tap directly. Open "Animal" and find abstract art, and that is a dent in the app's credibility. Trading away seven points of coverage was not a hard decision. Had these been internal analytics tags, I would have left the flat threshold alone.

Where to Start

Choosing a threshold turned out to be an exercise in saying out loud where I draw the line between accuracy and effort. Staring at the numbers long enough, it becomes clear which category's mistakes I can live with and which I cannot.

If you are facing the same problem, build a validation set of 200 to 400 rows with three columns: prediction, ground truth, confidence. Then plot _precision_at() per class. Which classes your flat threshold was quietly sacrificing is usually obvious at a glance.

Calibration does not make the model smarter. It measures the model's habits and translates them into your decisions. Once I framed it that way, the work stopped feeling like a chore.

Thank you for reading. If you find an unexpected class sinking in your own validation set, start by suspecting the definition sentence in your prompt.