Choosing Dynamic Workflow Size and Effort From a Ledger, Not a Hunch

Pulling Telemetry Onto Your Own Machine

Claude Code exports metrics over OpenTelemetry. That sounds like standing up a monitoring stack, but if all you want is evidence for a decision, the console exporter is enough. I first tried to bring up Prometheus and burned half a day on setup alone. I would not recommend it now.

# For ~/.claude/settings.json, put these under the env block.
# For a single trial session, the shell is plenty.
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=console
export OTEL_LOGS_EXPORTER=console
export OTEL_METRIC_EXPORT_INTERVAL=10000   # default is 60s; short runs get lost
 
# Capture each run to its own file (it interleaves with stdout otherwise)
mkdir -p ~/.wf-ledger/raw
claude -p "$(cat ~/.wf-ledger/task.md)" 2> ~/.wf-ledger/raw/$(date +%s).otel.jsonl

Leave OTEL_METRIC_EXPORT_INTERVAL at its default and any run shorter than 60 seconds finishes before the flush, leaving you with zero metrics. My first three attempts produced empty files for exactly this reason. Drop it to 10 seconds and nothing slips through.

The two metrics worth collecting are claude_code.token.usage, which carries a type attribute of input / output / cacheRead / cacheCreation, and claude_code.cost.usage. Note the word estimate on that second one. It is a number for comparing conditions against each other, not for reconciling against an invoice.

A Sweep Harness That Runs One Task Everywhere

Change one variable at a time and repeat the same task. It is unglamorous, but without it you never escape "large feels better."

#!/usr/bin/env bash
# ~/.wf-ledger/sweep.sh — sweep one task across size x effort
set -euo pipefail
 
TASK_FILE="${1:?usage: sweep.sh <task.md> [repeats]}"
REPEATS="${2:-3}"
LEDGER="$HOME/.wf-ledger"
mkdir -p "$LEDGER/raw"
 
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=console
export OTEL_METRIC_EXPORT_INTERVAL=10000
 
for size in small medium large; do
  for effort in high xhigh; do
    for i in $(seq 1 "$REPEATS"); do
      run_id="${size}-${effort}-${i}-$(date +%s)"
      start=$(date +%s%3N)
 
      # Write size into settings before launching; the value matches /config.
      python3 - "$size" <<'PY'
import json, pathlib, sys
p = pathlib.Path.home() / ".claude" / "settings.json"
cfg = json.loads(p.read_text()) if p.exists() else {}
cfg["dynamicWorkflowSize"] = sys.argv[1]
p.write_text(json.dumps(cfg, indent=2))
PY
 
      claude -p "$(cat "$TASK_FILE")" \
        --output-format json \
        > "$LEDGER/raw/${run_id}.result.json" \
        2> "$LEDGER/raw/${run_id}.otel.jsonl" || echo "run failed: $run_id" >&2
 
      end=$(date +%s%3N)
      echo "{\"run_id\":\"${run_id}\",\"size\":\"${size}\",\"effort\":\"${effort}\",\"elapsed_ms\":$((end - start))}" \
        >> "$LEDGER/runs.jsonl"
      echo "done ${run_id} ($(( (end - start) / 1000 ))s)" >&2
    done
  done
done

Three repeats, because one tells you nothing. Identical conditions still wobble: across three runs of the same setting I saw output tokens vary by as much as 18%. Back when I judged from a single run, I was reading that wobble as a difference between conditions.

Settings key names can differ between versions. Set the value once through /config, open ~/.claude/settings.json, and copy whatever key actually appears — that is safer than trusting my spelling of it.

It Only Becomes Comparable Once It Is Folded

Raw OTel output is unreadable as-is. Fold it into three columns: tokens, estimated cost, wall time.

#!/usr/bin/env node
// ~/.wf-ledger/ledger.mjs — fold raw/*.otel.jsonl into a per-condition ledger
import { readFileSync, readdirSync } from "node:fs";
import { homedir } from "node:os";
import { join } from "node:path";
 
const DIR = join(homedir(), ".wf-ledger");
const RAW = join(DIR, "raw");
 
// The console exporter spreads one metric across several lines,
// so scan by block rather than by line.
function extractMetrics(text) {
  const out = [];
  const re = /"name"\s*:\s*"(claude_code\.[a-z._]+)"[\s\S]*?"value"\s*:\s*([\d.]+)([\s\S]*?"attributes"\s*:\s*\{([\s\S]*?)\})?/g;
  let m;
  while ((m = re.exec(text)) !== null) {
    out.push({ name: m[1], value: Number(m[2]), attrs: m[4] ?? "" });
  }
  return out;
}
 
const runs = readFileSync(join(DIR, "runs.jsonl"), "utf8")
  .trim().split("\n").map((l) => JSON.parse(l));
 
const rows = runs.map((run) => {
  const file = readdirSync(RAW).find((f) => f.startsWith(run.run_id) && f.endsWith(".otel.jsonl"));
  if (!file) return { ...run, tokens: null, cost: null, note: "otel missing" };
 
  const text = readFileSync(join(RAW, file), "utf8");
  const metrics = extractMetrics(text);
 
  const tokens = metrics
    .filter((x) => x.name === "claude_code.token.usage")
    .filter((x) => !/cacheRead/.test(x.attrs))   // cache reads are priced differently
    .reduce((s, x) => s + x.value, 0);
  const cost = metrics
    .filter((x) => x.name === "claude_code.cost.usage")
    .reduce((s, x) => s + x.value, 0);
 
  const result = JSON.parse(readFileSync(join(RAW, `${run.run_id}.result.json`), "utf8"));
  return { ...run, tokens, cost: Number(cost.toFixed(4)), passed: gradeOutput(result) };
});
 
// Grade against your own acceptance criteria. Replace this wholesale.
function gradeOutput(result) {
  const text = typeof result === "string" ? result : (result.result ?? JSON.stringify(result));
  const hasAllSections = ["## Context", "## Verification", "## Conclusion"].every((h) => text.includes(h));
  const hasCitations = (text.match(/https?:\/\//g) ?? []).length >= 3;
  return hasAllSections && hasCitations;
}
 
const grouped = {};
for (const r of rows) {
  const key = `${r.size}/${r.effort}`;
  (grouped[key] ??= []).push(r);
}
 
const avg = (xs) => xs.reduce((a, b) => a + b, 0) / xs.length;
console.table(
  Object.entries(grouped).map(([key, rs]) => ({
    condition: key,
    runs: rs.length,
    tokens_avg: Math.round(avg(rs.map((r) => r.tokens ?? 0))),
    cost_avg: Number(avg(rs.map((r) => r.cost ?? 0)).toFixed(4)),
    elapsed_s: Math.round(avg(rs.map((r) => r.elapsed_ms)) / 1000),
    pass_rate: `${Math.round((rs.filter((r) => r.passed).length / rs.length) * 100)}%`,
  })),
);

gradeOutput is the load-bearing part. Without a pass column this ledger is just a cost table — and since cheaper always wins a cost table, your decision collapses onto "the cheapest condition." Decide your acceptance criteria first, then express them in a form a machine can check. That is where the time is well spent.

Mine were crude: does the research output carry all three headings, and does it cite at least three source URLs. Crude, but a crude column still changes the decision.

The Table That Came Out, and What I Changed

Here is the sweep for one task — article research for Dolice Labs. Three-run averages, from my environment. These are not numbers to carry home; the procedure is.

What moved the pass rate on this task was effort, not size. Leaving size at small and raising effort to xhigh reached 100%, at roughly one sixth the output tokens of large / xhigh. For a week, what I had been buying with large was a crowd of subagents all headed to the same destination.

A cross-site dead-link audit across all four blogs inverted the result. small misses links; medium and above hold steady. The search surface is genuinely wide there.

I keep this table in ~/.wf-ledger/policy.md and have my unattended runs read it. When in doubt I recommend starting at small / xhigh: insufficient width shows up in the pass rate, whereas insufficient depth quietly erodes quality without ever failing a check. I have not stopped letting ultracode make the call — I just prefer to hold my own answer before I delegate the question.

Where the Reading Goes Wrong

Even with a ledger in hand I misread it several times. Four of them, in the order I hit them.

Summing cache-read tokens

claude_code.token.usage with a cacheRead type is priced differently, so a naive sum inflates what size increases appear to cost. Split by the type attribute — that is why the script above filters cacheRead out.

Treating the estimate as billing

The docs call claude_code.cost.usage an estimate. Fine for comparing conditions, useless for monthly reconciliation. Console usage remains the source of truth, and confusing the two will quietly skew your production budgeting.

Judging on wall time alone

large stretches both time and tokens, but in unattended runs time is not necessarily cost. Nobody is waiting on my nightly jobs, so 14 minutes is fine; the spend is what hurts. In an interactive session the calculus flips. The same table yields different conclusions depending on when the run happens.

Relaxing the threshold after the fact

The nastiest one. When the cheap condition fails, the temptation to soften gradeOutput arrives immediately. The workaround is blunt: set the bar before the sweep and do not touch it once results are in. I moved my thresholds into a separate file and keep it read-only for the duration of a sweep. Obvious in principle; surprisingly slippery once your own wallet is involved.

Spend Thirty Minutes Measuring First

I am glad the dials exist. But what arrived is a choice, not an answer — the default used to absorb this decision, and now it sits with you.

The work is small. Pick one representative task, set OTEL_METRICS_EXPORTER=console, sweep size and effort, and add one pass column. That alone makes you a better decision-maker than I was during my week of vaguely choosing large. Thirty minutes, start to finish.

Setting names and attributes shift between releases, so confirm against the primary sources — the Claude Code monitoring docs and the workflows docs — before copying anything here. For the authoring side, building workflows yourself, I covered that in writing your own Dynamic Workflow.

As an indie developer I never once questioned "bigger is better" until I measured it. If this spares even one person the same detour, it was worth writing.