Which Model Ran Last Night's Unattended Session? Building Model Attribution and Default-Drift Detection After the Sonnet 5 Switch

Extracting the Actual Model From Run Output — Defensively, From Two Sources

Next, record the model that actually ran, on every execution. The result object from headless runs (claude -p --output-format json) carries usage information, but its exact field layout can shift between versions. I deliberately avoid betting on a single field and probe candidates in order. The session transcript (JSONL) is my second source, since assistant events carry the model ID.

// extract-models.ts — defensively extract model IDs from run output
// Source 1: the result object from `claude -p --output-format json`
// Source 2: the session transcript JSONL (assistant events)
import { readFileSync } from "node:fs";
 
export function modelsFromResultJson(raw: string): string[] {
  const found = new Set<string>();
  try {
    const r = JSON.parse(raw);
    // Candidate 1: per-model usage breakdown (keys are model IDs)
    if (r && typeof r.modelUsage === "object" && r.modelUsage !== null) {
      for (const k of Object.keys(r.modelUsage)) found.add(k);
    }
    // Candidate 2: a flat `model` field
    if (typeof r.model === "string") found.add(r.model);
  } catch {
    // Not JSON — defer to the transcript, but never swallow silently
    console.error("Failed to parse result JSON; falling back to transcript");
  }
  return [...found];
}
 
export function modelsFromTranscript(path: string): string[] {
  const found = new Set<string>();
  for (const line of readFileSync(path, "utf8").split("\n")) {
    if (!line.trim()) continue;
    try {
      const e = JSON.parse(line);
      const m = e?.message?.model; // model ID on assistant events
      if (typeof m === "string" && m.startsWith("claude-")) found.add(m);
    } catch { /* skip corrupt lines (partial-write protection) */ }
  }
  return [...found];
}
 
// Expected output:
//   from result JSON → ["claude-sonnet-5"]
//   from transcript  → ["claude-sonnet-5", "claude-haiku-4-5"]  // subagents can mix in a second model

Two implementation details matter here. First, the return type is an array, not a string. Sessions that spawn subagents or background tasks can legitimately involve more than one model, and a "single primary model" assumption silently drops that information. Second, when the two sources disagree, I record both rather than throwing. The extractor's job is evidence, not arbitration — a human can reconcile later.

An Append-Only Run Record That Survives Crashes

Each extracted model list gets appended, one line per run, to a per-lineage record file. Unattended environments must assume the process can die mid-write, so I write to a temp file and rename it into place. Readers never see a half-written line.

// run-ledger.ts — atomic append to the run record
import { readFileSync, writeFileSync, renameSync, existsSync } from "node:fs";
 
export interface RunRecord {
  ts: string;          // ISO 8601, normalized to one timezone
  lineage: string;     // task lineage ID, e.g. "site-a-premium-fri"
  models: string[];    // model IDs actually observed
  source: "result" | "transcript" | "both";
  exitCode: number;
}
 
export function appendRun(path: string, rec: RunRecord): void {
  const prev = existsSync(path) ? readFileSync(path, "utf8") : "";
  const next = prev + JSON.stringify(rec) + "\n";
  const tmp = `${path}.${process.pid}.tmp`; // PID-suffixed temp name avoids collisions
  writeFileSync(tmp, next, "utf8");
  renameSync(tmp, path); // rename within one filesystem is atomic
}
 
// Expected output (one line):
// {"ts":"2026-07-02T04:15:09+09:00","lineage":"site-a-premium-fri",
//  "models":["claude-sonnet-5"],"source":"both","exitCode":0}

The essential habit: write the record on success, not just on failure. A failure-only log captures nothing about events that succeed while changing — which is exactly what a default switch is.

Matching Observations Against Expectations — Pin Versus Follow

With records accumulating, each run compares its observation against a per-lineage policy. I kept the policy space to exactly two options: pin (this lineage must run on this model; anything else is a fault) and follow-default (ride the default, but tell me the moment it moves). Verdicts are three-valued — ok, drift, unknown — and unknown (no model observed at all) fails closed into an investigation queue.

// drift-check.ts — compare policy against observed models
export type Policy =
  | { kind: "pin"; model: string }
  | { kind: "follow-default" };
 
export type Verdict = "ok" | "drift" | "unknown";
 
export function judge(
  policy: Policy,
  observed: string[],
  previous: string[] | null
): { verdict: Verdict; note: string } {
  if (observed.length === 0) {
    return { verdict: "unknown", note: "No model observed — suspect extractor breakage or an output-format change" };
  }
  if (policy.kind === "pin") {
    return observed.includes(policy.model)
      ? { verdict: "ok", note: `Pinned as expected: ${policy.model}` }
      : { verdict: "drift", note: `Pin ${policy.model} violated; observed ${observed.join(",")}` };
  }
  // follow-default: alert (but don't halt) when the primary model moves
  if (previous && previous[0] && observed[0] !== previous[0]) {
    return { verdict: "drift", note: `Default moved: ${previous[0]} → ${observed[0]}` };
  }
  return { verdict: "ok", note: "Following default" };
}
 
// Expected behavior:
//   pin claude-opus-4-8, observed ["claude-sonnet-5"] → drift (alert + quarantine today's artifacts)
//   follow-default, previous ["claude-opus-4-x"] → observed ["claude-sonnet-5"] → drift (notify only)

Treating drift differently per policy is the operational core. A drift under pin means broken config or a stray environment variable, so that lineage's artifacts get quarantined before publication. A drift under follow-default is expected life, so nothing halts — but the notification always fires and that day's artifacts get one extra review pass. On July 2, all three of my unpinned lineages emitted exactly that one-line notice, and "Default moved: → claude-sonnet-5" turned what would have been days of confused diffing into a calm observation exercise.

What a Week of Observation Showed

I deliberately left the three lineages on follow-default for a week and compared against the pre-switch records. These are my local observations, not benchmarks, but the direction was unambiguous.

On balance a clear improvement — but the length variance nearly tripped a downstream quality-gate threshold. Without the model trail, I would have spent days combing prompt diffs for a cause that wasn't there. One recorded line replaced that entire investigation.

Pin or Follow? The Three Axes I Used to Decide

Finally, decide per lineage whether to switch to pinning. I used three axes.

1. Price is time-limited. Sonnet 5's intro pricing ends August 31; from September 1 it returns to $3 input / $15 output per million tokens. Make permanent decisions at standard pricing and treat the intro window as an observation bonus, not a baseline.
2. Pinning transfers deprecation ownership to you. The moment you pin, tracking that model ID's lifecycle becomes your job. Pinning a lineage that has no tracking in place is a slow-motion outage. The allowlist approach from Stopping Auto-Upgrades From Draining Your Credits — Pinning Execution Models with enforceAvailableModels transfers directly.
3. Match stability requirements to the lineage. Lineages that generate published artifacts lean pin; lineages whose output passes through human review lean follow. And since behavior can shift even under an unchanged model ID, pairing this with the startup canary from When the Same Model Name Starts Behaving Differently — Catching Drift with a Boot-Time Canary closes the remaining gap.

My outcome: the two lineages that write published artifacts are now pinned explicitly to claude-sonnet-5 — pinning to the same model as the default still matters, because it moves the decision from the platform's hands into mine — and the third stays on follow-default with notifications. If you're on a team account, note that organization-level model restrictions are rolling out as well; add "admin restriction" as a fourth channel in your inventory table.

The First Step

Start tonight: from your very next run, record the model name — one line, from either the result JSON or the transcript. The three modules above can wait for the weekend; the evidence trail can't be backfilled. This won't be the last default-model change, and the position you want to be in next time is the quiet one: "every lineage, we can say exactly which model it ran on."