Hierarchical Chat History Summarization with Claude API: A 3-Tier Design That Cut Tokens by 70%

Tier 1: Keeping the Recent Window Verbatim

The hard part of Tier 1 was not the code; it was choosing the threshold. I originally fixed it at "last 10 turns" and discovered that for fast-paced users that was redundant, while for slow-paced users 10 turns was not enough. I now budget Tier 1 by tokens instead of turn count.

// src/history/tier1.ts
import type { Message } from "@anthropic-ai/sdk/resources/messages";
 
export interface Tier1Result {
  recent: Message[];
  overflow: Message[];
  tokenEstimate: number;
}
 
const TIER1_TOKEN_BUDGET = 3_000;
const TIER1_MIN_TURNS = 4;
 
// Mixed JA/EN content: 1 token ~= 0.6 characters as a rough heuristic.
function estimateTokens(message: Message): number {
  const content = typeof message.content === "string"
    ? message.content
    : message.content.map((c) => "text" in c ? c.text : "").join("");
  return Math.ceil(content.length / 0.6);
}
 
export function partitionTier1(history: Message[]): Tier1Result {
  const recent: Message[] = [];
  let used = 0;
 
  for (let i = history.length - 1; i >= 0; i--) {
    const cost = estimateTokens(history[i]);
    if (used + cost > TIER1_TOKEN_BUDGET && recent.length >= TIER1_MIN_TURNS) {
      return {
        recent: recent.reverse(),
        overflow: history.slice(0, i + 1),
        tokenEstimate: used,
      };
    }
    recent.push(history[i]);
    used += cost;
  }
 
  return { recent: recent.reverse(), overflow: [], tokenEstimate: used };
}

The Minimum-Turns Guard

TIER1_MIN_TURNS guarantees that no matter how token-heavy the recent turns are, at least four turns are kept. My app exposes Continue and Refine buttons that explicitly reference the last assistant turn, and those are tapped more than 8,000 times a day. If Tier 1 ever cut them out, the reviews would not be kind.

Why a Cheap Token Estimator Is Fine

A precise count is available via messages.count_tokens, but calling it on every partitioning step adds cost and latency. The 0.6-characters-per-token heuristic is within 4% of the real count in my logs, which is precise enough for partitioning decisions even if it would be too rough for billing analysis.

Tier 2: Compressing Episodes to ~200 Tokens with Haiku

Tier 2 takes the overflow from Tier 1, slices it into six-turn episodes, and hands each chunk to Haiku 4.5. The prompt trick that mattered most was isolation: do not let the parent chat's system prompt leak into the summarizer, or the summary becomes "a description of the app" rather than a record of the actual conversation.

// src/history/tier2.ts
import Anthropic from "@anthropic-ai/sdk";
import type { Message } from "@anthropic-ai/sdk/resources/messages";
 
const anthropic = new Anthropic();
 
const SUMMARIZE_SYSTEM = `You compress a chunk of a chat conversation.
Output a single paragraph in the user's language, ~200 tokens, written in the third person.
Preserve: explicit preferences, rejected ideas, decisions, in-progress requests.
Drop: greetings, filler, repeated acknowledgements.
Never invent facts; if uncertain, omit.`;
 
export interface EpisodeSummary {
  range: [number, number];
  summary: string;
  createdAt: number;
}
 
export async function summarizeEpisode(
  episode: Message[],
  baseIndex: number,
): Promise<EpisodeSummary> {
  const transcript = episode
    .map((m) => `[${m.role}] ${flattenContent(m)}`)
    .join("\n");
 
  const res = await anthropic.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 320,
    system: SUMMARIZE_SYSTEM,
    messages: [{ role: "user", content: transcript }],
  });
 
  const text = res.content
    .filter((c): c is { type: "text"; text: string } => c.type === "text")
    .map((c) => c.text).join("\n").trim();
 
  return {
    range: [baseIndex, baseIndex + episode.length - 1],
    summary: text,
    createdAt: Date.now(),
  };
}
 
function flattenContent(m: Message): string {
  if (typeof m.content === "string") return m.content;
  return m.content
    .map((c) => "text" in c ? c.text : "[non-text content]")
    .join(" ");
}

Fixed-Size Episode Boundaries Work Surprisingly Well

Semantic boundaries — slicing where the conversation actually shifts topic — would in theory yield better summaries. In practice, a fixed six-turn window has been fine. The cost of an extra Claude call to find boundaries always outweighed the marginal quality gain, and the summarizer is good enough at preserving meaning across arbitrary cuts.

Always Preserve Speaker Attribution

A subtle detail that mattered: keep [user] and [assistant] tags in the transcript you pass to the summarizer. Without that, "the user prefers blue" and "the assistant suggested blue" eventually blend into the same statement, and Tier 3 later distills a wrong preference.

A Trap: Language Drift in the Summary

If the system prompt is in English, Haiku will frequently return the summary in English even when the conversation was Japanese. That cascades into Tier 3 distillation noise. Instructing Haiku explicitly to "output a paragraph in the user's language" while keeping the system prompt itself in English fixed it.

Tier 3: Distilling Entities and Intent into JSON with Sonnet

Once at least three episode summaries exist, I run Tier 3 to update the long-range memory snapshot. The point here is structure over prose, so I use tool_use with a strict schema to force JSON output.

// src/history/tier3.ts
import Anthropic from "@anthropic-ai/sdk";
 
const anthropic = new Anthropic();
 
const memoryTool = {
  name: "record_long_term_memory",
  description: "Record the user's stable preferences, rejected ideas, and ongoing intents.",
  input_schema: {
    type: "object",
    properties: {
      preferences: {
        type: "array",
        items: { type: "string" },
        description: "Stable preferences (color palette, mood, motif, style).",
      },
      rejected: {
        type: "array",
        items: { type: "string" },
        description: "Ideas the user has explicitly turned down.",
      },
      ongoing_intent: {
        type: "string",
        description: "The user's current goal in 1-2 sentences.",
      },
      anchors: {
        type: "array",
        items: { type: "string" },
        description: "Named entities worth remembering (artist, location, technique).",
      },
    },
    required: ["preferences", "rejected", "ongoing_intent", "anchors"],
  },
} as const;
 
export interface LongTermMemory {
  preferences: string[];
  rejected: string[];
  ongoing_intent: string;
  anchors: string[];
  updatedAt: number;
}
 
export async function distillLongTermMemory(
  episodeSummaries: string[],
  previous: LongTermMemory | null,
): Promise<LongTermMemory> {
  const prior = previous
    ? `Previous memory snapshot:\n${JSON.stringify(previous, null, 2)}\n\n`
    : "";
  const merged = episodeSummaries
    .map((s, i) => `Episode ${i + 1}:\n${s}`)
    .join("\n\n");
 
  const res = await anthropic.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 800,
    tools: [memoryTool],
    tool_choice: { type: "tool", name: "record_long_term_memory" },
    messages: [{
      role: "user",
      content: `${prior}New episode summaries:\n${merged}\n\nMerge prior memory with the new evidence. Drop superseded preferences. Keep the JSON small.`,
    }],
  });
 
  const toolUse = res.content.find((c) => c.type === "tool_use");
  if (!toolUse || toolUse.type !== "tool_use") {
    throw new Error("Tier3: tool_use not returned");
  }
 
  const input = toolUse.input as Omit<LongTermMemory, "updatedAt">;
  return { ...input, updatedAt: Date.now() };
}

Force tool_choice to Stop Free-Text Drift

Without forcing tool_choice, Sonnet occasionally returns prose instead of using the tool. Making the tool mandatory and listing every field in required guarantees a parseable response and removes a whole class of edge cases from the downstream code. Distillation runs roughly once every 30 turns at a cost of about $0.012 per call, which is negligible.

Merge, Don't Overwrite

My first implementation rebuilt long-term memory from scratch every Tier 3 cycle. That caused stable preferences to flicker — a single new episode could overwrite three episodes of consistent evidence. Passing the previous snapshot as context and instructing Sonnet to "merge prior memory with the new evidence" fixed that, and the stability of the memory snapshots is much better as a result.

Running Compression Asynchronously

The biggest risk of hierarchical summarization is that the compression itself slows down the user. If you call Tier 2 or Tier 3 synchronously before responding, the user pays one to three extra seconds of latency. In an indie app, that shows up in reviews very quickly.

// src/history/orchestrator.ts
import { partitionTier1 } from "./tier1";
import type { LongTermMemory, EpisodeSummary } from "./types";
 
export interface ChatState {
  history: Message[];
  episodes: EpisodeSummary[];
  longTermMemory: LongTermMemory | null;
}
 
const EPISODE_SIZE = 6;
const DISTILL_TRIGGER = 3;
 
export async function buildPromptMessages(state: ChatState): Promise<Message[]> {
  const { recent } = partitionTier1(state.history);
 
  const memorySystem = state.longTermMemory
    ? formatMemorySystemBlock(state.longTermMemory)
    : "";
  const episodicContext = state.episodes
    .map((e, i) => `<episode index="${i}">${e.summary}</episode>`)
    .join("\n");
 
  return [
    { role: "user", content: `${memorySystem}\n${episodicContext}` },
    ...recent,
  ];
}
 
export function shouldCompressTier2(state: ChatState): boolean {
  const { overflow } = partitionTier1(state.history);
  const summarized = state.episodes.length * EPISODE_SIZE;
  return overflow.length - summarized >= EPISODE_SIZE;
}
 
export function shouldCompressTier3(state: ChatState): boolean {
  const unsummarizedEpisodes = state.episodes.length
    - (state.longTermMemory
        ? Math.floor(state.episodes.length / DISTILL_TRIGGER) * DISTILL_TRIGGER
        : 0);
  return unsummarizedEpisodes >= DISTILL_TRIGGER;
}

Push Work to Cloudflare Queues

shouldCompressTier2 and shouldCompressTier3 are pure decisions. The actual Claude calls go to Cloudflare Queues from a c.executionCtx.waitUntil block, so the user-facing Worker returns immediately. By the next message, the summaries and updated long-term memory are already in KV.

What Happens When Compression Is Late

Compression failures need an explicit policy. Mine is: if compression has not finished by the time the user sends the next message, fall back to Tier 1 only and accept temporarily blowing the token budget rather than blocking the user. The Worker logs a "compression skipped" event, and an alert fires if the daily skip rate exceeds 5%, which is my signal that the queue is backed up.

Measuring What Compression Dropped — With Questions, Not Eyeballs

For about two weeks I watched only the compression ratio. Tokens fell the way I wanted, the reviews stayed calm, and I assumed the design was working.

Then a message arrived: the assistant had started ignoring a color constraint the user gave at the very beginning. I opened the logs. Tier 3 still had it under rejected. Tier 2 was where it vanished — and the turn in question sat exactly on a six-turn episode boundary.

A summary's quality is not something you read and judge. It is something you probe and measure. Read with your eyes, a summary that quietly lost its meaning at the boundary still looks like a well-written summary. That is precisely why two weeks went by.

Probe only for facts that exist nowhere but the raw history

The method is plain. Pull facts out of the raw transcript that appear only in older turns, write a question for each, and decide which words a correct answer must contain. Then hand the model nothing but the compressed messages, ask the question, and check for those words.

Don't reach for an LLM judge first. Pick questions that a literal match can settle, and fall back to Haiku only when wording drift makes the call ambiguous. Judging cost stays roughly constant, which is what makes it cheap enough to run nightly.

// src/history/recall-probe.ts
import Anthropic from "@anthropic-ai/sdk";
import { buildPromptMessages } from "./orchestrator";
import type { ChatState } from "./orchestrator";
 
export interface Probe {
  /** Index in the raw history where this fact was stated */
  statedAtTurn: number;
  question: string;
  /** A correct answer contains at least one of these */
  mustInclude: string[];
}
 
export interface ProbeResult extends Probe {
  answer: string;
  passed: boolean;
  judgedBy: "literal" | "model";
}
 
const anthropic = new Anthropic();
const UNKNOWN = "UNKNOWN";
 
export async function runProbe(state: ChatState, probe: Probe): Promise<ProbeResult> {
  // Build the compressed messages through the production path.
  // Reimplementing this for the test makes the measurement meaningless.
  const compressed = await buildPromptMessages(state);
 
  const res = await anthropic.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 200,
    system: `Answer only from the conversation context provided. If it is not there, reply exactly: ${UNKNOWN}`,
    messages: [...compressed, { role: "user", content: probe.question }],
  });
 
  const answer = res.content.map((b) => ("text" in b ? b.text : "")).join("").trim();
 
  const literal = probe.mustInclude.some(
    (w) => answer.toLowerCase().includes(w.toLowerCase()),
  );
  if (literal || answer === UNKNOWN) {
    return { ...probe, answer, passed: literal, judgedBy: "literal" };
  }
  return { ...probe, answer, passed: await judgeByModel(probe, answer), judgedBy: "model" };
}
 
async function judgeByModel(probe: Probe, answer: string): Promise<boolean> {
  const res = await anthropic.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 8,
    system: "Reply with exactly YES or NO.",
    messages: [{
      role: "user",
      content: `Does the answer below convey any of ${JSON.stringify(probe.mustInclude)}?\n\nAnswer: ${answer}`,
    }],
  });
  return res.content.some(
    (b) => "text" in b && b.text.trim().toUpperCase().startsWith("YES"),
  );
}

The one rule I'd hold to: never rebuild buildPromptMessages for the harness. A history assembled by a second code path is not the history your users get. I cut that corner once, watched every probe pass including the ones that had no business passing, and lost half a day to it.

The losses are not spread evenly

I built 240 questions from 42 sessions and grouped the results by where the fact had originally been stated.

The average is 86.3%, which reads perfectly respectable. The distribution does not. Only the boundary figure, 63.5%, sits clearly apart from the other three.

The cause was not summary prompt quality. Slicing mechanically every six turns can split the turn where a user states a constraint from the turn where the model acknowledges it. Haiku, seeing only one half, treats an unanchored remark as unimportant and drops it — which, as summarization, is correct behavior.

One turn of overlap at each seam

The fix was a single turn of overlap in how episodes get sliced.

const EPISODE_SIZE = 6;
const EPISODE_OVERLAP = 1;
 
export function sliceEpisodes(overflow: Message[], alreadySummarized: number): Message[][] {
  const episodes: Message[][] = [];
  let start = alreadySummarized;
 
  while (overflow.length - start >= EPISODE_SIZE) {
    // Carry the last turn of the previous episode so a statement
    // and its acknowledgement never land on opposite sides of a seam.
    const from = Math.max(0, start - EPISODE_OVERLAP);
    episodes.push(overflow.slice(from, start + EPISODE_SIZE));
    start += EPISODE_SIZE;
  }
  return episodes;
}

Re-running the same 240 probes moved boundary recall from 63.5% to 86.5%. The price is about 17% more input tokens per summarization call — $1.9 a month at my volume. Episode count itself doesn't change, so this was an easy trade to make.

I tried two turns of overlap as well. Boundary recall only reached 88.4% while input grew 34%. One turn of overlap is enough; past that you're paying for duplicated summary.

How many questions before you trust the number

There's nothing theoretical behind 240. My rule of thumb is at least 40 probes per position bucket. At 40, a ten-point move is coarse enough to separate from noise. In the low twenties a single wrong answer swings the figure five points, and I could never tell whether a threshold change had helped or hurt.

Writing the questions is front-loaded work. I read ten real sessions and marked only the places where I myself thought this should be remembered, then grew the set from that shape. One nightly run, recall by position printed next to last night's — that alone is what made it obvious which threshold I'd broken the day before.

Seven Weeks of Production Data

Why Perceived Quality Improved

The most interesting result was not the cost reduction; it was the drop in "forgot what I said" complaints. Tier 3 was actively preserving entities like "no red or gold" across 30+ turns, and naive trimming could never have done that. When I decide what to change in an app, I keep "tried" and "set aside" in separate notes for the same reason. Give the assistant that shape and, from the user's side, it simply seems to listen better.

A Secondary Latency Win

Time-to-first-token also improved. Requests under 3,000 input tokens landed at about 1,500 ms in my logs, while requests above 6,000 sat at about 2,400 ms. Because compression runs asynchronously, that latency cliff disappeared on the synchronous path. AdMob's ad rendering is sensitive to surrounding response speeds, so faster session pacing translated into slightly higher impressions per session as a side effect.

Prompt Caching on Top of the Three Tiers

If you layer cache_control on top of this architecture, put the long-term memory block in its own cached chunk. The episode list changes too often for caching to help, but the Tier 3 JSON is stable across multiple turns.

const cachedSystem = [
  {
    type: "text",
    text: formatMemorySystemBlock(state.longTermMemory),
    cache_control: { type: "ephemeral" },
  },
];
 
await anthropic.messages.create({
  model: "claude-sonnet-4-6",
  system: cachedSystem,
  messages: [...episodicAsUser, ...recent],
  max_tokens: 1_024,
});

Episode List Is Not a Good Cache Candidate

Each Tier 2 run appends a new summary, so the episode list invalidates frequently. Putting it under cache_control would just thrash the cache. The Tier 3 JSON is different: my cache hit rate over seven weeks was 71%, and a hit costs roughly one tenth of an uncached input token. That is what brings the API bill down by another tens of dollars per month.

Tier 3 Invalidates the Cache

Every Tier 3 run rewrites the long-term memory block, which immediately invalidates the cache for one turn afterward. I treat that one-turn cost as an accepted overhead. Just be careful not to over-trigger Tier 3 — every distillation pays a re-warm cost, and if DISTILL_TRIGGER is too aggressive, you can lose more on cache thrashing than you gain in fresh memory.

When Hierarchical Summarization Is Not the Right Tool

Three patterns where I would not reach for this design:

1. Short Stateless Q&A

If your product is a one- to three-turn support FAQ assistant, the hierarchy overhead exceeds the actual API spend. A simple ring buffer of the last N turns is fine. My Tier 2 only fires after 12 turns; if most sessions never get there, you are just paying for plumbing.

2. Legally or Medically Sensitive Conversations

Summarization is probabilistic about meaning. In domains where the exact wording of past statements matters — medical history, legal advice — you should keep the raw transcript. My wallpaper app is firmly in a subjective taste domain where this is not a concern, but as soon as you cross into compliance territory, the safer move is to keep history intact.

3. Multi-User Collaborative Sessions

I experimented with a meeting-style chat where multiple users share one session. The Tier 3 schema explodes because every user has independent preferences, and merging them coherently is hard. Per-user state with separate Tier 3 memories worked far better than one shared state.

The Practical Indie-Developer Test

Hierarchical summarization pays off when your sessions grow linearly, when long-running sessions are core to revenue, and when users expect the assistant to remember the conversation. In my situation — solo developer running multiple AdMob-monetized apps with in-app AI features — it dropped API cost by 70% while raising perceived quality, and it is now a permanent part of every chat integration I ship.

Measure Before You Compress

The next iteration I am sketching is a semantic episode boundary detector — cutting where the conversation actually changes subject — and embedding the Tier 3 JSON as a vector so it can be retrieved across sessions. There is one step I would put ahead of all of it, though.

If you're weighing history compression right now, write forty probes before you implement Tier 1. Knowing your recall rate without compression is what lets you notice the day it drops. I did this in the opposite order and spent two weeks not knowing. Don't shrink what you haven't measured. That is the one rule I keep, even in a week with a deadline in it.

If this saves someone an evening of guessing, it was worth writing down.