Hierarchical Chat History Summarization with Claude API: A 3-Tier Design That Cut Tokens by 70%

Tier 1: Keeping the Recent Window Verbatim

The hard part of Tier 1 was not the code; it was choosing the threshold. I originally fixed it at "last 10 turns" and discovered that for fast-paced users that was redundant, while for slow-paced users 10 turns was not enough. I now budget Tier 1 by tokens instead of turn count.

// src/history/tier1.ts
import type { Message } from "@anthropic-ai/sdk/resources/messages";
 
export interface Tier1Result {
  recent: Message[];
  overflow: Message[];
  tokenEstimate: number;
}
 
const TIER1_TOKEN_BUDGET = 3_000;
const TIER1_MIN_TURNS = 4;
 
// Mixed JA/EN content: 1 token ~= 0.6 characters as a rough heuristic.
function estimateTokens(message: Message): number {
  const content = typeof message.content === "string"
    ? message.content
    : message.content.map((c) => "text" in c ? c.text : "").join("");
  return Math.ceil(content.length / 0.6);
}
 
export function partitionTier1(history: Message[]): Tier1Result {
  const recent: Message[] = [];
  let used = 0;
 
  for (let i = history.length - 1; i >= 0; i--) {
    const cost = estimateTokens(history[i]);
    if (used + cost > TIER1_TOKEN_BUDGET && recent.length >= TIER1_MIN_TURNS) {
      return {
        recent: recent.reverse(),
        overflow: history.slice(0, i + 1),
        tokenEstimate: used,
      };
    }
    recent.push(history[i]);
    used += cost;
  }
 
  return { recent: recent.reverse(), overflow: [], tokenEstimate: used };
}

The Minimum-Turns Guard

TIER1_MIN_TURNS guarantees that no matter how token-heavy the recent turns are, at least four turns are kept. My app exposes Continue and Refine buttons that explicitly reference the last assistant turn, and those are tapped more than 8,000 times a day. If Tier 1 ever cut them out, the reviews would not be kind.

Why a Cheap Token Estimator Is Fine

A precise count is available via messages.count_tokens, but calling it on every partitioning step adds cost and latency. The 0.6-characters-per-token heuristic is within 4% of the real count in my logs, which is precise enough for partitioning decisions even if it would be too rough for billing analysis.

Tier 2: Compressing Episodes to ~200 Tokens with Haiku

Tier 2 takes the overflow from Tier 1, slices it into six-turn episodes, and hands each chunk to Haiku 4.5. The prompt trick that mattered most was isolation: do not let the parent chat's system prompt leak into the summarizer, or the summary becomes "a description of the app" rather than a record of the actual conversation.

// src/history/tier2.ts
import Anthropic from "@anthropic-ai/sdk";
import type { Message } from "@anthropic-ai/sdk/resources/messages";
 
const anthropic = new Anthropic();
 
const SUMMARIZE_SYSTEM = `You compress a chunk of a chat conversation.
Output a single paragraph in the user's language, ~200 tokens, written in the third person.
Preserve: explicit preferences, rejected ideas, decisions, in-progress requests.
Drop: greetings, filler, repeated acknowledgements.
Never invent facts; if uncertain, omit.`;
 
export interface EpisodeSummary {
  range: [number, number];
  summary: string;
  createdAt: number;
}
 
export async function summarizeEpisode(
  episode: Message[],
  baseIndex: number,
): Promise<EpisodeSummary> {
  const transcript = episode
    .map((m) => `[${m.role}] ${flattenContent(m)}`)
    .join("\n");
 
  const res = await anthropic.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 320,
    system: SUMMARIZE_SYSTEM,
    messages: [{ role: "user", content: transcript }],
  });
 
  const text = res.content
    .filter((c): c is { type: "text"; text: string } => c.type === "text")
    .map((c) => c.text).join("\n").trim();
 
  return {
    range: [baseIndex, baseIndex + episode.length - 1],
    summary: text,
    createdAt: Date.now(),
  };
}
 
function flattenContent(m: Message): string {
  if (typeof m.content === "string") return m.content;
  return m.content
    .map((c) => "text" in c ? c.text : "[non-text content]")
    .join(" ");
}

Fixed-Size Episode Boundaries Work Surprisingly Well

Semantic boundaries — slicing where the conversation actually shifts topic — would in theory yield better summaries. In practice, a fixed six-turn window has been fine. The cost of an extra Claude call to find boundaries always outweighed the marginal quality gain, and the summarizer is good enough at preserving meaning across arbitrary cuts.

Always Preserve Speaker Attribution

A subtle detail that mattered: keep [user] and [assistant] tags in the transcript you pass to the summarizer. Without that, "the user prefers blue" and "the assistant suggested blue" eventually blend into the same statement, and Tier 3 later distills a wrong preference.

A Trap: Language Drift in the Summary

If the system prompt is in English, Haiku will frequently return the summary in English even when the conversation was Japanese. That cascades into Tier 3 distillation noise. Instructing Haiku explicitly to "output a paragraph in the user's language" while keeping the system prompt itself in English fixed it.

Tier 3: Distilling Entities and Intent into JSON with Sonnet

Once at least three episode summaries exist, I run Tier 3 to update the long-range memory snapshot. The point here is structure over prose, so I use tool_use with a strict schema to force JSON output.

// src/history/tier3.ts
import Anthropic from "@anthropic-ai/sdk";
 
const anthropic = new Anthropic();
 
const memoryTool = {
  name: "record_long_term_memory",
  description: "Record the user's stable preferences, rejected ideas, and ongoing intents.",
  input_schema: {
    type: "object",
    properties: {
      preferences: {
        type: "array",
        items: { type: "string" },
        description: "Stable preferences (color palette, mood, motif, style).",
      },
      rejected: {
        type: "array",
        items: { type: "string" },
        description: "Ideas the user has explicitly turned down.",
      },
      ongoing_intent: {
        type: "string",
        description: "The user's current goal in 1-2 sentences.",
      },
      anchors: {
        type: "array",
        items: { type: "string" },
        description: "Named entities worth remembering (artist, location, technique).",
      },
    },
    required: ["preferences", "rejected", "ongoing_intent", "anchors"],
  },
} as const;
 
export interface LongTermMemory {
  preferences: string[];
  rejected: string[];
  ongoing_intent: string;
  anchors: string[];
  updatedAt: number;
}
 
export async function distillLongTermMemory(
  episodeSummaries: string[],
  previous: LongTermMemory | null,
): Promise<LongTermMemory> {
  const prior = previous
    ? `Previous memory snapshot:\n${JSON.stringify(previous, null, 2)}\n\n`
    : "";
  const merged = episodeSummaries
    .map((s, i) => `Episode ${i + 1}:\n${s}`)
    .join("\n\n");
 
  const res = await anthropic.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 800,
    tools: [memoryTool],
    tool_choice: { type: "tool", name: "record_long_term_memory" },
    messages: [{
      role: "user",
      content: `${prior}New episode summaries:\n${merged}\n\nMerge prior memory with the new evidence. Drop superseded preferences. Keep the JSON small.`,
    }],
  });
 
  const toolUse = res.content.find((c) => c.type === "tool_use");
  if (!toolUse || toolUse.type !== "tool_use") {
    throw new Error("Tier3: tool_use not returned");
  }
 
  const input = toolUse.input as Omit<LongTermMemory, "updatedAt">;
  return { ...input, updatedAt: Date.now() };
}

Force tool_choice to Stop Free-Text Drift

Without forcing tool_choice, Sonnet occasionally returns prose instead of using the tool. Making the tool mandatory and listing every field in required guarantees a parseable response and removes a whole class of edge cases from the downstream code. Distillation runs roughly once every 30 turns at a cost of about $0.012 per call, which is negligible.

Merge, Don't Overwrite

My first implementation rebuilt long-term memory from scratch every Tier 3 cycle. That caused stable preferences to flicker — a single new episode could overwrite three episodes of consistent evidence. Passing the previous snapshot as context and instructing Sonnet to "merge prior memory with the new evidence" fixed that, and the stability of the memory snapshots is much better as a result.

Running Compression Asynchronously

The biggest risk of hierarchical summarization is that the compression itself slows down the user. If you call Tier 2 or Tier 3 synchronously before responding, the user pays one to three extra seconds of latency. In an indie app, that shows up in reviews very quickly.

// src/history/orchestrator.ts
import { partitionTier1 } from "./tier1";
import type { LongTermMemory, EpisodeSummary } from "./types";
 
export interface ChatState {
  history: Message[];
  episodes: EpisodeSummary[];
  longTermMemory: LongTermMemory | null;
}
 
const EPISODE_SIZE = 6;
const DISTILL_TRIGGER = 3;
 
export async function buildPromptMessages(state: ChatState): Promise<Message[]> {
  const { recent } = partitionTier1(state.history);
 
  const memorySystem = state.longTermMemory
    ? formatMemorySystemBlock(state.longTermMemory)
    : "";
  const episodicContext = state.episodes
    .map((e, i) => `<episode index="${i}">${e.summary}</episode>`)
    .join("\n");
 
  return [
    { role: "user", content: `${memorySystem}\n${episodicContext}` },
    ...recent,
  ];
}
 
export function shouldCompressTier2(state: ChatState): boolean {
  const { overflow } = partitionTier1(state.history);
  const summarized = state.episodes.length * EPISODE_SIZE;
  return overflow.length - summarized >= EPISODE_SIZE;
}
 
export function shouldCompressTier3(state: ChatState): boolean {
  const unsummarizedEpisodes = state.episodes.length
    - (state.longTermMemory
        ? Math.floor(state.episodes.length / DISTILL_TRIGGER) * DISTILL_TRIGGER
        : 0);
  return unsummarizedEpisodes >= DISTILL_TRIGGER;
}

Push Work to Cloudflare Queues

shouldCompressTier2 and shouldCompressTier3 are pure decisions. The actual Claude calls go to Cloudflare Queues from a c.executionCtx.waitUntil block, so the user-facing Worker returns immediately. By the next message, the summaries and updated long-term memory are already in KV.

What Happens When Compression Is Late

Compression failures need an explicit policy. Mine is: if compression has not finished by the time the user sends the next message, fall back to Tier 1 only and accept temporarily blowing the token budget rather than blocking the user. The Worker logs a "compression skipped" event, and an alert fires if the daily skip rate exceeds 5%, which is my signal that the queue is backed up.

Seven Weeks of Production Data

Why Perceived Quality Improved

The most interesting result was not the cost reduction; it was the drop in "forgot what I said" complaints. Tier 3 was actively preserving entities like "no red or gold" across 30+ turns, and naive trimming could never have done that. From a user perspective the assistant simply seemed to listen better.

A Secondary Latency Win

Time-to-first-token also improved. Requests under 3,000 input tokens landed at about 1,500 ms in my logs, while requests above 6,000 sat at about 2,400 ms. Because compression runs asynchronously, that latency cliff disappeared on the synchronous path. AdMob's ad rendering is sensitive to surrounding response speeds, so faster session pacing translated into slightly higher impressions per session as a side effect.

Prompt Caching on Top of the Three Tiers

If you layer cache_control on top of this architecture, put the long-term memory block in its own cached chunk. The episode list changes too often for caching to help, but the Tier 3 JSON is stable across multiple turns.

const cachedSystem = [
  {
    type: "text",
    text: formatMemorySystemBlock(state.longTermMemory),
    cache_control: { type: "ephemeral" },
  },
];
 
await anthropic.messages.create({
  model: "claude-sonnet-4-6",
  system: cachedSystem,
  messages: [...episodicAsUser, ...recent],
  max_tokens: 1_024,
});

Episode List Is Not a Good Cache Candidate

Each Tier 2 run appends a new summary, so the episode list invalidates frequently. Putting it under cache_control would just thrash the cache. The Tier 3 JSON is different: my cache hit rate over seven weeks was 71%, and a hit costs roughly one tenth of an uncached input token. That is what brings the API bill down by another tens of dollars per month.

Tier 3 Invalidates the Cache

Every Tier 3 run rewrites the long-term memory block, which immediately invalidates the cache for one turn afterward. I treat that one-turn cost as an accepted overhead. Just be careful not to over-trigger Tier 3 — every distillation pays a re-warm cost, and if DISTILL_TRIGGER is too aggressive, you can lose more on cache thrashing than you gain in fresh memory.

When Hierarchical Summarization Is Not the Right Tool

Three patterns where I would not reach for this design:

1. Short Stateless Q&A

If your product is a one- to three-turn support FAQ assistant, the hierarchy overhead exceeds the actual API spend. A simple ring buffer of the last N turns is fine. My Tier 2 only fires after 12 turns; if most sessions never get there, you are just paying for plumbing.

2. Legally or Medically Sensitive Conversations

Summarization is probabilistic about meaning. In domains where the exact wording of past statements matters — medical history, legal advice — you should keep the raw transcript. My wallpaper app is firmly in a subjective taste domain where this is not a concern, but as soon as you cross into compliance territory, the safer move is to keep history intact.

3. Multi-User Collaborative Sessions

I experimented with a meeting-style chat where multiple users share one session. The Tier 3 schema explodes because every user has independent preferences, and merging them coherently is hard. Per-user state with separate Tier 3 memories worked far better than one shared state.

The Practical Indie-Developer Test

Hierarchical summarization pays off when your sessions grow linearly, when long-running sessions are core to revenue, and when users expect the assistant to remember the conversation. In my situation — solo developer running multiple AdMob-monetized apps with in-app AI features — it dropped API cost by 70% while raising perceived quality, and it is now a permanent part of every chat integration I ship.

What I Want to Try Next

The next iteration I am sketching is a semantic episode boundary detector — picking cuts where the conversation actually shifts topic — and embedding the Tier 3 JSON as a vector so it can be retrieved across multiple sessions. Sharing long-term memory across my app catalog so a user who taught one wallpaper app their color palette does not need to re-teach the next one is the long-term goal. Thanks for reading.