Allocating the 200K Context Window in Claude API — Budgeting System, Tools, Memory, and History in Production

Implementation: BudgetAllocator

This is the core of the TypeScript module, lifted from the version I run in production. It works on Cloudflare Workers and Node alike.

type Category = "system" | "tools" | "memory" | "history" | "reserve";
 
interface BudgetPolicy {
  base: Record<Category, number>;
  max: Record<Category, number>;
  totalBudget: number; // 192_000 for Sonnet 4.6
}
 
const DEFAULT_POLICY: BudgetPolicy = {
  base: { system: 5000, tools: 15000, memory: 30000, history: 40000, reserve: 24000 },
  max:  { system: 6000, tools: 22000, memory: 60000, history: 80000, reserve: 24000 },
  totalBudget: 192_000,
};
 
export class BudgetAllocator {
  constructor(private policy: BudgetPolicy = DEFAULT_POLICY) {}
 
  // Fast path: return the static budget unchanged
  staticAllocate(): Record<Category, number> {
    return { ...this.policy.base };
  }
 
  // Look at actual demand and reallocate the elastic envelopes
  dynamicAllocate(
    requested: Partial<Record<Category, number>>,
  ): Record<Category, number> {
    const out = { ...this.policy.base };
    const { base, max, totalBudget } = this.policy;
 
    // 1) Lock in system and tools first (up to their caps)
    for (const k of ["system", "tools"] as const) {
      out[k] = Math.min(requested[k] ?? base[k], max[k]);
    }
 
    // 2) Memory and history share the elastic remainder
    const fixed = out.system + out.tools + base.reserve;
    const flexBudget = totalBudget - fixed;
    const memReq = Math.min(requested.memory ?? base.memory, max.memory);
    const histReq = Math.min(requested.history ?? base.history, max.history);
 
    if (memReq + histReq <= flexBudget) {
      out.memory = memReq;
      out.history = histReq;
    } else {
      // On contention, history wins (cutting recent context hurts UX more)
      out.history = Math.min(histReq, flexBudget - base.memory);
      out.memory = flexBudget - out.history;
    }
    return out;
  }
}

Two things matter here. First, system and tools are resolved before anything else, because they rarely change across requests and locking them in simplifies every downstream decision. Second, when memory and history compete for the elastic remainder, history wins. Losing recent conversation context degrades the user experience visibly; memory loss tends to degrade quality gradually, and the model adapts.

The Tool-Definitions Blind Spot

This was the actual culprit behind the March incident. I had three MCP servers wired up, and the combined tool definitions were 18,200 tokens — every request. The Anthropic Console shows input_tokens as one number, so to break it down you have to measure it yourself.

import Anthropic from "@anthropic-ai/sdk";
 
async function measureToolTokens(client: Anthropic, tools: Anthropic.Tool[]) {
  const probe = await client.messages.countTokens({
    model: "claude-sonnet-4-6",
    system: "probe",
    tools,
    messages: [{ role: "user", content: "x" }],
  });
  const baseline = await client.messages.countTokens({
    model: "claude-sonnet-4-6",
    system: "probe",
    messages: [{ role: "user", content: "x" }],
  });
  return probe.input_tokens - baseline.input_tokens;
}

Pair a countTokens call with and without tools, and the delta is your tool overhead. In my case the Slack MCP carried verbose descriptions that ran 700–1,200 tokens per tool. Trimming descriptions to essentials and pruning unused enum values brought the total from 18,200 down to 9,400 tokens. That single change recovered 23% of monthly input cost.

Splitting the Memory Tier Across Two Models

The 30,000-token memory envelope is not homogeneous. In my wallpaper app it holds three kinds of recall, and each has a different precision requirement.

1. Summaries of the last five sessions (medium precision, extractive) — generated by Haiku 4.5
2. Structured preference tags (high precision, already structured) — fetched directly from KV, no model required
3. Long-term episodes (low precision, thematic only) — abstracted by Sonnet 4.6

Haiku 4.5 took the medium tier because its input pricing runs roughly a quarter of Sonnet 4.6 and the quality delta on summarization tasks was not perceptible in user testing. The long-term abstraction layer stayed on Sonnet 4.6 because missed meaning there is fatal, and prompt_caching keeps the recomputation cost in check.

After this split, monthly API spend dropped from ¥86,000 to ¥38,000. DAU was flat, Day 7 retention moved +1.8%, and I have not yet seen evidence of quality regression.

Production Gotchas I Hit

Three potholes that only became visible after I shipped.

(1) Prompt-caching hit rate breaks budget assumptions

I had carved envelopes assuming system and tools would land in cache. With a five-minute TTL, low-frequency users missed cache on every request, and cache_creation_input_tokens showed up as its own line item I only spotted at billing time. Switching to a one-hour TTL alongside the default lifted hit rate from 41% to 78%.

(2) Dynamic tool sets corrupt history coherence

If a tool used in a past turn disappears from the current request, the assistant turn that referenced tool_use is left dangling. I now treat the tool set as immutable for the duration of a session. Tool pruning happens only at session boundaries.

(3) Hard-coding max_tokens at 8,000 truncates good responses

For requests that expect long-form explanations, an 8,000 ceiling clips the tail. I keep the headroom budgeted, but I borrow from reserve to lift max_tokens dynamically up to 16,000 when the demand profile justifies it, and I repay that borrow by shrinking the history envelope on the same request.

Recommended Allocations by Scenario

There is no universal answer. Here are the three profiles I actually run, which I recommend as starting points.

Profile A: Short-form support bot (DAU 200–500)

• system 4,000 / tools 6,000 / memory 10,000 / history 20,000 / reserve 152,000
• Few tools, heavy on history. A thick reserve absorbs occasional long answers. Recommended when tool variety is low.

Profile B: Multi-agent coding assistant

• system 8,000 / tools 25,000 / memory 50,000 / history 70,000 / reserve 39,000
• Tool definitions are heavy and codebase search lands in memory. I recommend enabling prompt_caching here unconditionally.

Profile C: Personalized chat (the wallpaper-app pattern)

• system 5,000 / tools 9,000 / memory 30,000 / history 40,000 / reserve 108,000
• Within memory, 18,000 goes to Haiku 4.5 summaries and 12,000 to KV-backed structured preferences. Recommended when the workload is highly stateful and per-user personalized.

Across all three profiles, the constant is leaving reserve alone. The day you trim it is the day an unexpected MCP server or a retrieval-heavy query overflows the window.

The Indie Developer's Pragmatic Take

The apps I run survive on a small economic loop of AdMob revenue and modest in-app purchases. Unpredictable API costs swing gross margin sharply. At the same time, trimming context too aggressively degrades the experience, drives down reviews, and ultimately lowers AdMob eCPM through reduced session quality.

The two-tier discipline — fix the static budget, reallocate dynamically — is what reconciles those pressures. It mirrors how I work as an artist: 17 international art awards have not changed the fact that the work which holds up is the work built on a skeleton I refuse to compromise on. Code is no different. Define the skeleton first, then improvise within it.

If you take one action from this article, measure the token cost of your system and tools with countTokens today. In my experience, most of the slack hides there, not in your conversation history. Thank you for reading.