Building Production Conversational AI with Claude API — Context Management, Long-Term Memory, and Safety Filters

You built a chatbot with Claude API. The first five turns were perfect — accurate answers, natural context awareness, smooth conversation flow. Then around turn ten, things start breaking. The AI forgets what you told it three turns ago, repeats explanations it already gave, and token consumption balloons to tens of thousands per request.

Nearly every developer who builds a conversational system with Claude API hits this wall. The Messages API is stateless, so maintaining conversation continuity means sending the entire message history with every request. This naive approach works for short exchanges, but in production, you'll crash into three critical problems: context window exhaustion, loss of important information, and input/output safety risks.

This article walks through an integrated architecture refined across multiple production projects. This isn't an API tutorial — it digs into why each design decision was made and which alternative approaches were rejected.

The Three Memory Layers Your Production Chat System Needs

Just as human memory operates through short-term memory, episodic memory, and semantic memory, a production conversational system needs three distinct memory layers.

Layer 1: Working Memory (Recent Messages) — Keep the last N messages verbatim. This provides immediate conversational context, enabling pronoun resolution and references to recent topics. Too many messages and token costs explode; too few and context breaks.

Layer 2: Episodic Memory (Conversation Summaries) — Have Claude summarize older message groups into compressed form. Something like "The user asked about Python async processing and now understands how asyncio.gather works." This preserves conversational flow while dramatically cutting token usage.

Layer 3: Semantic Memory (User Attributes and Preferences) — Long-term information stored in a vector database. Technical skill level, preferred programming languages, past question patterns — all persisted across sessions to enable personalized responses.

The official documentation simply says "send all messages." In practice, you can't achieve both cost efficiency and good user experience without combining all three layers.

Implementing Sliding Window + Summary Buffer

Let's look at the implementation that integrates Layers 1 and 2. The key insight: don't just discard old messages — convert them to summaries before compression.

import Anthropic from "@anthropic-ai/sdk";
 
interface ConversationMemory {
  summary: string;           // Layer 2: accumulated conversation summary
  recentMessages: Array<{    // Layer 1: recent messages
    role: "user" | "assistant";
    content: string;
  }>;
  totalTurns: number;
}
 
const client = new Anthropic();
const MAX_RECENT_MESSAGES = 10; // Last 5 turns (user + assistant pairs)
const SUMMARY_TRIGGER = 8;     // Trigger summarization above this count
const MAX_RETRIES = 3;
 
async function chat(
  memory: ConversationMemory,
  userMessage: string,
  attempt = 0
): Promise<{ response: string; updatedMemory: ConversationMemory }> {
  memory.recentMessages.push({ role: "user", content: userMessage });
  memory.totalTurns++;
 
  // Compress old messages into summary when threshold is exceeded
  if (memory.recentMessages.length > MAX_RECENT_MESSAGES) {
    try {
      memory.summary = await compressToSummary(
        memory.summary,
        memory.recentMessages.slice(0, -SUMMARY_TRIGGER)
      );
      memory.recentMessages = memory.recentMessages.slice(-SUMMARY_TRIGGER);
    } catch (error) {
      // If summarization fails, truncate rather than crash the conversation
      console.error("Summary generation failed, truncating:", error);
      memory.recentMessages = memory.recentMessages.slice(-MAX_RECENT_MESSAGES);
    }
  }
 
  const systemPrompt = buildSystemPrompt(memory.summary);
 
  try {
    const response = await client.messages.create({
      model: "claude-sonnet-4-6",
      max_tokens: 2048,
      system: systemPrompt,
      messages: memory.recentMessages,
    });
 
    const assistantMessage =
      response.content[0].type === "text" ? response.content[0].text : "";
    memory.recentMessages.push({ role: "assistant", content: assistantMessage });
 
    return { response: assistantMessage, updatedMemory: memory };
  } catch (error) {
    if (error instanceof Anthropic.RateLimitError && attempt < MAX_RETRIES) {
      // Honor retry-after when present, otherwise exponential backoff
      const retryAfter = Number(error.headers?.["retry-after"]);
      const waitMs = Number.isFinite(retryAfter)
        ? retryAfter * 1000
        : 2 ** attempt * 1000;
      await new Promise((r) => setTimeout(r, waitMs));
 
      // Roll back the pushed user message and turn counter before retrying
      memory.recentMessages.pop();
      memory.totalTurns--;
      return chat(memory, userMessage, attempt + 1);
    }
    // Leave conversation state clean when handing the error back
    memory.recentMessages.pop();
    memory.totalTurns--;
    throw error;
  }
}
 
async function compressToSummary(
  existingSummary: string,
  messagesToCompress: Array<{ role: string; content: string }>
): Promise<string> {
  const conversationText = messagesToCompress
    .map((m) => `${m.role}: ${m.content}`)
    .join("\n");
 
  // Use Haiku for summarization — cost-effective with negligible quality difference
  const response = await client.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 500,
    messages: [
      {
        role: "user",
        content: `Summarize the following conversation history concisely while preserving key information (user intent, resolved issues, open topics, stated preferences).
 
Existing summary:
${existingSummary || "(none)"}
 
New conversation:
${conversationText}
 
Summary (under 200 words):`,
      },
    ],
  });
 
  return response.content[0].type === "text"
    ? response.content[0].text
    : existingSummary;
}
 
function buildSystemPrompt(summary: string): string {
  const base = `You are a technical assistant. Provide practical answers with code examples.`;
 
  if (!summary) return base;
  return `${base}
 
[Previous Conversation Summary]
${summary}
 
Use this summary for context but don't repeat its content.`;
}

Why use claude-haiku-4-5 for summarization? Summarization tasks have long inputs and short outputs — exactly the pattern where Haiku's low-cost, high-speed characteristics shine. Switching to Sonnet for summarization yields marginally better summaries at roughly 5x the cost, and the quality difference is imperceptible in production. Model routing is the single most important cost optimization lever for conversational systems.

Building Long-Term Memory with Vector Databases

Cross-session memory requires Layer 3: semantic memory. Here's an implementation using pgvector. The critical design decision is what to remember — storing everything increases retrieval noise and actually degrades response quality.

import Anthropic from "@anthropic-ai/sdk";
import pg from "pg";
 
const client = new Anthropic();
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
 
// Extract memories worth persisting from a conversation
async function extractMemories(
  conversation: Array<{ role: string; content: string }>
): Promise<Array<{ content: string; category: string; importance: number }>> {
  const conversationText = conversation
    .map((m) => `${m.role}: ${m.content}`)
    .join("\n");
 
  try {
    const response = await client.messages.create({
      model: "claude-haiku-4-5",
      max_tokens: 1000,
      messages: [
        {
          role: "user",
          content: `Extract information from this conversation that would be useful in future sessions. Return as a JSON array.
 
Extract:
- Preferences: languages, frameworks, coding style
- Facts: project names, tech stack, team size
- Context: current challenges, learning stage
 
Do NOT extract:
- Generic technical knowledge (searchable information)
- Greetings or small talk
- Temporary topics
 
Conversation:
${conversationText}
 
JSON array:
[{"content": "...", "category": "preference|fact|context", "importance": 0.0-1.0}]`,
        },
      ],
    });
 
    const text = response.content[0].type === "text" ? response.content[0].text : "[]";
    const jsonMatch = text.match(/\[[\s\S]*\]/);
    if (!jsonMatch) return [];
 
    const parsed = JSON.parse(jsonMatch[0]);
    return parsed.filter(
      (m: { importance: number }) => m.importance >= 0.5
    );
  } catch (error) {
    console.error("Memory extraction failed:", error);
    return []; // Extraction failure doesn't break the conversation
  }
}
 
// Retrieve relevant memories via vector similarity search
async function retrieveRelevantMemories(
  userId: string,
  query: string,
  limit: number = 5
): Promise<string[]> {
  try {
    const embedding = await generateEmbedding(query);
 
    // Composite scoring: similarity * importance for balanced ranking
    const result = await pool.query(
      `SELECT content, importance,
              1 - (embedding <=> $1::vector) AS similarity
       FROM user_memories
       WHERE user_id = $2
         AND created_at > NOW() - INTERVAL '90 days'
       ORDER BY (1 - (embedding <=> $1::vector)) * importance DESC
       LIMIT $3`,
      [JSON.stringify(embedding), userId, limit]
    );
 
    return result.rows.map((row: { content: string }) => row.content);
  } catch (error) {
    console.error("Memory retrieval failed:", error);
    return []; // Graceful degradation — conversation continues without memories
  }
}
 
async function generateEmbedding(text: string): Promise<number[]> {
  const response = await fetch("https://api.openai.com/v1/embeddings", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "text-embedding-3-small",
      input: text,
    }),
  });
 
  if (!response.ok) {
    throw new Error(`Embedding API error: ${response.status}`);
  }
 
  const data = await response.json();
  return data.data[0].embedding;
}

The similarity * importance composite scoring deserves attention. Pure vector similarity treats "likes Python" and "currently leading a Kubernetes migration" with equal weight. Using importance as a ranking factor rather than a filter balances relevance against significance — you always get the most relevant results, but critical context floats to the top.

The created_at > NOW() - INTERVAL '90 days' clause implements automatic memory decay. Users' tech stacks and interests evolve, and stale memories pollute search results more than they help.

Designing for Persona and Tone Consistency

As conversations grow longer, AI responses tend to drift in tone — starting formal and gradually becoming casual, or using terminology inconsistently. The root cause is usually system prompt design.

The approach that works in production is persona layering: splitting the system prompt into three tiers — base personality, response rules, and contextual adaptation.

interface PersonaConfig {
  basePersonality: string;
  responseRules: string[];
  adaptiveRules: Record<string, string>;
}
 
function buildLayeredSystemPrompt(
  persona: PersonaConfig,
  userLevel: "beginner" | "intermediate" | "advanced",
  conversationMood: "casual" | "professional" | "urgent"
): string {
  // Tier 1: Base personality (immutable)
  let prompt = `# Your Persona\n${persona.basePersonality}\n\n`;
 
  // Tier 2: Response rules (always applied)
  prompt += `# Response Rules\n`;
  for (const rule of persona.responseRules) {
    prompt += `- ${rule}\n`;
  }
 
  // Tier 3: Contextual adaptation (dynamic)
  prompt += `\n# Current Context\n`;
  prompt += `- User's technical level: ${userLevel}\n`;
 
  if (userLevel === "beginner") {
    prompt += `- Always explain jargon in plain language before using it\n`;
    prompt += `- Add line-by-line comments to code examples\n`;
  } else if (userLevel === "advanced") {
    prompt += `- Skip prerequisite explanations, focus on the core\n`;
    prompt += `- Address performance implications and edge cases\n`;
  }
 
  if (persona.adaptiveRules[conversationMood]) {
    prompt += `- ${persona.adaptiveRules[conversationMood]}\n`;
  }
 
  return prompt;
}
 
const techAdvisor: PersonaConfig = {
  basePersonality:
    "Experienced senior engineer. Prioritizes accuracy while being candid about real-world trade-offs. Honest about uncertainty.",
  responseRules: [
    "Always include error handling in code examples",
    "Frame suggestions as options, not directives",
    "Never present more than 3 choices at once (prevent decision fatigue)",
    "When correcting a misconception, acknowledge the user's reasoning first",
  ],
  adaptiveRules: {
    casual: "Conversational tone, occasionally share personal experience",
    professional: "Concise and structured responses",
    urgent: "Lead with the shortest fix, detail comes after",
  },
};

Why three tiers? A monolithic system prompt requires rewriting everything whenever you adjust for context, which introduces unintended personality drift. Separating layers lets you lock the base personality while dynamically tuning only situational behavior.

Multi-Layer Input Validation and Output Filtering

Safety is the most frequently overlooked aspect of production chat systems. You need to handle prompt injection attempts, accidental PII exposure, and potentially harmful content.

import Anthropic from "@anthropic-ai/sdk";
 
interface SafetyCheckResult {
  safe: boolean;
  reason?: string;
  sanitizedInput?: string;
}
 
const client = new Anthropic();
 
async function validateInput(userInput: string): Promise<SafetyCheckResult> {
  // Stage 1: Regex-based fast filter (zero API cost)
  const patterns = {
    creditCard: /\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b/,
    injection:
      /ignore\s+(all\s+)?previous\s+instructions|system\s*prompt|you\s+are\s+now/i,
  };
 
  if (patterns.creditCard.test(userInput)) {
    return {
      safe: false,
      reason: "Credit card number detected. Please don't enter personal financial information.",
    };
  }
 
  if (patterns.injection.test(userInput)) {
    // Neutralize rather than block — preserve legitimate user intent
    const sanitized = userInput.replace(patterns.injection, "[filtered]");
    return { safe: true, sanitizedInput: sanitized };
  }
 
  // Stage 2: Semantic check via Claude Haiku (only when needed)
  if (userInput.length > 500 || containsSuspiciousPatterns(userInput)) {
    try {
      const response = await client.messages.create({
        model: "claude-haiku-4-5",
        max_tokens: 100,
        messages: [
          {
            role: "user",
            content: `Classify this user input for safety. Reply only "safe" or "unsafe: [reason]".
 
Input: "${userInput.slice(0, 1000)}"`,
          },
        ],
      });
 
      const result =
        response.content[0].type === "text" ? response.content[0].text : "safe";
 
      if (result.startsWith("unsafe")) {
        return { safe: false, reason: result.replace("unsafe: ", "") };
      }
    } catch (error) {
      // Safety check failure falls through — availability over perfect filtering
      console.error("Safety check failed:", error);
    }
  }
 
  return { safe: true, sanitizedInput: userInput };
}
 
function containsSuspiciousPatterns(input: string): boolean {
  const keywords = [
    "jailbreak", "bypass", "override", "roleplay as",
    "pretend you", "act as if", "DAN mode",
  ];
  const lower = input.toLowerCase();
  return keywords.some((kw) => lower.includes(kw));
}
 
function sanitizeOutput(output: string): string {
  let sanitized = output;
  // Strip API keys and tokens
  sanitized = sanitized.replace(
    /\b(sk-[a-zA-Z0-9]{20,}|ghp_[a-zA-Z0-9]{36}|AKIA[A-Z0-9]{16})\b/g,
    "[REDACTED]"
  );
  // Strip internal file paths
  sanitized = sanitized.replace(
    /\/(?:home|Users|var|etc)\/[^\s"']+/g,
    "[PATH_REDACTED]"
  );
  return sanitized;
}

The two-stage filtering design is deliberate. Regex catches obvious patterns at zero latency on every request. Semantic classification via Claude Haiku triggers only for long inputs or suspicious patterns, keeping safety check costs minimal.

A common mistake in safety design is the "block everything or allow everything" binary. Neutralizing suspected injection attempts rather than blocking them preserves the experience for legitimate users while defusing the attack.

Actually Calculating the Token Curve for All Three Approaches

"Adding a summary buffer makes things cheaper" appears in every article on this topic. What's usually missing: at which turn does the benefit kick in, and does it still hold after you account for the cost of generating the summaries? Skip that question and you may end up shipping a summary buffer into a product whose conversations never exceed three turns — pure added cost, zero benefit.

So I worked it out. The script below hits no API at all. It computes input-token volume per turn deterministically, so copying it and running it locally produces exactly the numbers shown afterward.

# ctx.py — compare input-token growth across naive / window / window+summary
AVG_USER, AVG_ASSISTANT = 120, 420   # average tokens per message
SYSTEM_BASE = 350                    # base system prompt
SUMMARY_MAX = 300                    # summary token ceiling
WINDOW = 8                           # messages retained (= 4 turns)
 
def naive(turn):
    return SYSTEM_BASE + turn * (AVG_USER + AVG_ASSISTANT)
 
def window_only(turn):
    kept = min(turn * 2, WINDOW)
    return SYSTEM_BASE + kept // 2 * (AVG_USER + AVG_ASSISTANT) + (kept % 2) * AVG_USER
 
def window_summary(turn):
    return window_only(turn) + (SUMMARY_MAX if turn * 2 > WINDOW else 0)
 
print(f"{'turn':>5} {'naive':>9} {'window':>9} {'win+sum':>9}")
cum = {'naive': 0, 'win': 0, 'ws': 0}
for t in range(1, 51):
    n, w, sm = naive(t), window_only(t), window_summary(t)
    cum['naive'] += n; cum['win'] += w; cum['ws'] += sm
    if t in (1, 5, 10, 20, 30, 50):
        print(f"{t:>5} {n:>9,} {w:>9,} {sm:>9,}")
 
print(f"\n50-turn totals: naive={cum['naive']:,} / window={cum['win']:,} / win+sum={cum['ws']:,}")
print(f"win+sum reduction: {(1 - cum['ws'] / cum['naive']) * 100:.1f}%")
 
# Summaries fire every 4 turns once the window overflows
sum_calls = sum(1 for t in range(1, 51) if t * 2 > WINDOW and t % 4 == 0)
SUM_IN, SUM_OUT = 2200, 300
print(f"summary calls: {sum_calls} / extra tokens in={sum_calls * SUM_IN:,} out={sum_calls * SUM_OUT:,}")
# Effective reduction with Haiku input priced at 1/12 of Sonnet
equiv = cum['ws'] + sum_calls * SUM_IN / 12
print(f"effective reduction incl. Haiku cost: {(1 - equiv / cum['naive']) * 100:.1f}%")

Output from my run (Python 3.11):

 turn     naive    window   win+sum
    1       890       890       890
    5     3,050     2,510     2,810
   10     5,750     2,510     2,810
   20    11,150     2,510     2,810
   30    16,550     2,510     2,810
   50    27,350     2,510     2,810

50-turn totals: naive=706,000 / window=122,260 / win+sum=136,060
win+sum reduction: 80.7%
summary calls: 11 / extra tokens in=24,200 out=3,300
effective reduction incl. Haiku cost: 80.4%

Three things stand out.

The summary buffer only starts paying off at turn five. Through turn four every message still fits inside the window, no summary is generated, and the curve is identical to naive. The flip side matters more: if your average conversation runs shorter than four turns, a summary buffer is pure liability. Check your conversation-length distribution before writing any of this code. For an indie developer, implementation time is the scarcest resource in the whole system — knowing which layers won't pay off is worth more than the tokens they would have saved.

Summarization cost is noise. Across 50 turns the summary API fires only 11 times, and pricing Haiku input against Sonnet moves the reduction from 80.7% to 80.4%. The cost of summarizing is not a reason to hesitate.

Window-only versus window+summary differs by 11% (122,260 against 136,060). That 11% buys one thing: remembering what happened ten turns ago. Drop the summary layer and you save a little over a tenth — at the cost of users receiving the same explanation twice. I think that tenth is worth paying, though for something like FAQ routing, where turns are largely context-independent, window-only is genuinely enough.

One caveat: this model holds average token counts fixed. Real token counts depend on actual content, so treat the output as a tool for seeing the relative shape of the three curves rather than as absolute figures.

Three More Ways to Cut Token Costs

The calculation above covers context design alone. Layering the following three optimizations on top widens the gap further.

Optimization 1: Prompt Caching — The system prompt and long-term memory section barely change during a conversation, making them ideal cache candidates.

const response = await client.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 2048,
  system: [
    {
      type: "text",
      text: systemPromptBase, // Base personality + rules (stable)
      cache_control: { type: "ephemeral" },
    },
    {
      type: "text",
      text: dynamicContext, // Summary + long-term memories (changes per turn)
    },
  ],
  messages: memory.recentMessages,
});
// Cache hits reduce input token costs by 90%

Optimization 2: Model Routing — Not every response needs Sonnet. Simple confirmations ("Yes, that's correct") and summarization work fine with Haiku. Classifying input complexity and routing to the appropriate model cuts average costs by 40%. The classification itself runs on Haiku at negligible cost.

Optimization 3: Progressive Summary Compression — When generating summaries via the sliding window, periodically re-compress the summaries themselves. By turn 50, early summaries condense to a few sentences containing only the user's fundamental goal and the core problem being solved. This caps system prompt token usage at a constant level regardless of conversation length.

Seven Production Pitfalls You'll Hit

Problems that only surface in real-world operation, with concrete mitigations.

1. Silent Context Degradation — The API throws an error when you exceed the context window, but just below the limit, the model disproportionately attends to recent messages and underweights the system prompt. Keep context usage below 70% of the window as a rule of thumb.

2. Summarization Drift — Each summarization pass loses subtle but important details. "User is stuck on React 18 Suspense" degrades to "User is having React issues." Mitigation: store key entities and topics as structured data alongside the narrative summary.

3. Hallucinated Memories — Claude may infer information during conversation that gets incorrectly saved as long-term memory. If the model guesses a user preference, storing that guess as fact is dangerous. Constrain your extraction prompt to only capture explicitly stated facts.

4. Over-Aggressive Safety Filters — Overly strict filters block legitimate technical discussions about security vulnerabilities or exploit mitigation. For technical chatbots, maintain a domain-specific allowlist so security-related terms in valid questions aren't flagged.

5. Mid-Stream Error Recovery — When network errors hit during SSE streaming, users see incomplete responses. Implement client-side completion detection with automatic retry for truncated streams.

6. Multi-Turn Tool Use State — Managing tool execution results across conversation turns is surprisingly complex when combined with summarization. Tool result messages have a specific structure that breaks when summarized. Exclude tool results from summarization and retain only their textual representation.

7. Timezone Misalignment — "Earlier" and "yesterday" need the user's timezone for correct resolution. Without it, you'll give UTC-based answers that are 9 hours off for users in Japan or 5 hours off for East Coast US users.

Monitoring Metrics and the Quality Feedback Loop

Four metrics are enough to run this in production.

Conversation continuation rate — the share of users who get past ten turns. A decline points at context management, not model quality.
Summary fidelity score — sample periodically and check whether summaries preserve the important information from the original exchange.
Safety filter trigger rate — too high means over-blocking, too low means you're missing things.
Token cost per turn — if this converges to a flat line as conversations lengthen, your context management is working.

For A/B testing, the levers worth pulling are summarization timing (8 messages versus 12), long-term memory retrieval count (3 versus 5 versus 7), and the model-routing threshold. Summarization timing moved the needle most: triggering at 8 messages gave the best balance of cost against conversation quality.

Bringing It All Together: The Complete Production Chat Loop

Here's the integrated flow that combines all components into a production-ready conversation loop.

async function productionChatLoop(
  userId: string,
  memory: ConversationMemory,
  userInput: string
): Promise<{ response: string; updatedMemory: ConversationMemory }> {
  // 1. Input safety validation
  const inputCheck = await validateInput(userInput);
  if (!inputCheck.safe) {
    return {
      response: `I can't process this input. ${inputCheck.reason}`,
      updatedMemory: memory,
    };
  }
  const safeInput = inputCheck.sanitizedInput || userInput;
 
  // 2. Retrieve relevant long-term memories
  const longTermMemories = await retrieveRelevantMemories(userId, safeInput);
 
  // 3. Build context-aware system prompt
  const systemPrompt = buildContextualSystemPrompt(
    memory.summary,
    longTermMemories
  );
 
  // 4. Execute conversation (sliding window + summary buffer)
  const result = await chat(memory, safeInput);
 
  // 5. Output safety filtering
  result.response = sanitizeOutput(result.response);
 
  // 6. Async memory extraction (don't block the response)
  extractMemories(memory.recentMessages.slice(-4)).then((memories) => {
    for (const mem of memories) {
      saveToVectorDB(userId, mem).catch(console.error);
    }
  });
 
  return result;
}

For conversation analytics patterns, see the logging strategies covered in Claude API Chatbot Building Fundamentals. For deeper coverage of vector database memory persistence, check out Production Memory Implementation with pgvector. If you want to go deeper on the input-side threat model, Production Patterns for Prompt Injection Defense picks up where this leaves off.

Your next step is straightforward: implement the sliding window + summary buffer code and verify that summaries generate correctly across a 10-turn conversation. From there, layer in long-term memory, then integrate safety filters. Building all three layers simultaneously makes debugging nearly impossible — take it one layer at a time.