Building Production Conversational AI with Claude API — Context Management, Long-Term Memory, and Safety Filters

You built a chatbot with Claude API. The first five turns were perfect — accurate answers, natural context awareness, smooth conversation flow. Then around turn ten, things start breaking. The AI forgets what you told it three turns ago, repeats explanations it already gave, and token consumption balloons to tens of thousands per request.

Nearly every developer who builds a conversational system with Claude API hits this wall. The Messages API is stateless, so maintaining conversation continuity means sending the entire message history with every request. This naive approach works for short exchanges, but in production, you'll crash into three critical problems: context window exhaustion, loss of important information, and input/output safety risks.

This article walks through an integrated architecture refined across multiple production projects. This isn't an API tutorial — it digs into why each design decision was made and which alternative approaches were rejected.

The Three Memory Layers Your Production Chat System Needs

Just as human memory operates through short-term memory, episodic memory, and semantic memory, a production conversational system needs three distinct memory layers.

Layer 1: Working Memory (Recent Messages) — Keep the last N messages verbatim. This provides immediate conversational context, enabling pronoun resolution and references to recent topics. Too many messages and token costs explode; too few and context breaks.

Layer 2: Episodic Memory (Conversation Summaries) — Have Claude summarize older message groups into compressed form. Something like "The user asked about Python async processing and now understands how asyncio.gather works." This preserves conversational flow while dramatically cutting token usage.

Layer 3: Semantic Memory (User Attributes and Preferences) — Long-term information stored in a vector database. Technical skill level, preferred programming languages, past question patterns — all persisted across sessions to enable personalized responses.

The official documentation simply says "send all messages." In practice, you can't achieve both cost efficiency and good user experience without combining all three layers.

Implementing Sliding Window + Summary Buffer

Let's look at the implementation that integrates Layers 1 and 2. The key insight: don't just discard old messages — convert them to summaries before compression.

import Anthropic from "@anthropic-ai/sdk";
 
interface ConversationMemory {
  summary: string;           // Layer 2: accumulated conversation summary
  recentMessages: Array<{    // Layer 1: recent messages
    role: "user" | "assistant";
    content: string;
  }>;
  totalTurns: number;
}
 
const client = new Anthropic();
const MAX_RECENT_MESSAGES = 10; // Last 5 turns (user + assistant pairs)
const SUMMARY_TRIGGER = 8;     // Trigger summarization above this count
 
async function chat(
  memory: ConversationMemory,
  userMessage: string
): Promise<{ response: string; updatedMemory: ConversationMemory }> {
  memory.recentMessages.push({ role: "user", content: userMessage });
  memory.totalTurns++;
 
  // Compress old messages into summary when threshold is exceeded
  if (memory.recentMessages.length > MAX_RECENT_MESSAGES) {
    try {
      memory.summary = await compressToSummary(
        memory.summary,
        memory.recentMessages.slice(0, -SUMMARY_TRIGGER)
      );
      memory.recentMessages = memory.recentMessages.slice(-SUMMARY_TRIGGER);
    } catch (error) {
      // If summarization fails, truncate rather than crash the conversation
      console.error("Summary generation failed, truncating:", error);
      memory.recentMessages = memory.recentMessages.slice(-MAX_RECENT_MESSAGES);
    }
  }
 
  const systemPrompt = buildSystemPrompt(memory.summary);
 
  try {
    const response = await client.messages.create({
      model: "claude-sonnet-4-6",
      max_tokens: 2048,
      system: systemPrompt,
      messages: memory.recentMessages,
    });
 
    const assistantMessage =
      response.content[0].type === "text" ? response.content[0].text : "";
    memory.recentMessages.push({ role: "assistant", content: assistantMessage });
 
    return { response: assistantMessage, updatedMemory: memory };
  } catch (error) {
    if (error instanceof Anthropic.RateLimitError) {
      // Exponential backoff on rate limits
      await new Promise((r) => setTimeout(r, 2000));
      return chat(
        { ...memory, recentMessages: memory.recentMessages.slice(0, -1) },
        userMessage
      );
    }
    throw error;
  }
}
 
async function compressToSummary(
  existingSummary: string,
  messagesToCompress: Array<{ role: string; content: string }>
): Promise<string> {
  const conversationText = messagesToCompress
    .map((m) => `${m.role}: ${m.content}`)
    .join("\n");
 
  // Use Haiku for summarization — cost-effective with negligible quality difference
  const response = await client.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 500,
    messages: [
      {
        role: "user",
        content: `Summarize the following conversation history concisely while preserving key information (user intent, resolved issues, open topics, stated preferences).
 
Existing summary:
${existingSummary || "(none)"}
 
New conversation:
${conversationText}
 
Summary (under 200 words):`,
      },
    ],
  });
 
  return response.content[0].type === "text"
    ? response.content[0].text
    : existingSummary;
}
 
function buildSystemPrompt(summary: string): string {
  const base = `You are a technical assistant. Provide practical answers with code examples.`;
 
  if (\!summary) return base;
  return `${base}
 
[Previous Conversation Summary]
${summary}
 
Use this summary for context but don't repeat its content.`;
}

Why use claude-haiku-4-5 for summarization? Summarization tasks have long inputs and short outputs — exactly the pattern where Haiku's low-cost, high-speed characteristics shine. Switching to Sonnet for summarization yields marginally better summaries at roughly 5x the cost, and the quality difference is imperceptible in production. Model routing is the single most important cost optimization lever for conversational systems.

Building Long-Term Memory with Vector Databases

Cross-session memory requires Layer 3: semantic memory. Here's an implementation using pgvector. The critical design decision is what to remember — storing everything increases retrieval noise and actually degrades response quality.

import Anthropic from "@anthropic-ai/sdk";
import pg from "pg";
 
const client = new Anthropic();
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
 
// Extract memories worth persisting from a conversation
async function extractMemories(
  conversation: Array<{ role: string; content: string }>
): Promise<Array<{ content: string; category: string; importance: number }>> {
  const conversationText = conversation
    .map((m) => `${m.role}: ${m.content}`)
    .join("\n");
 
  try {
    const response = await client.messages.create({
      model: "claude-haiku-4-5",
      max_tokens: 1000,
      messages: [
        {
          role: "user",
          content: `Extract information from this conversation that would be useful in future sessions. Return as a JSON array.
 
Extract:
- Preferences: languages, frameworks, coding style
- Facts: project names, tech stack, team size
- Context: current challenges, learning stage
 
Do NOT extract:
- Generic technical knowledge (searchable information)
- Greetings or small talk
- Temporary topics
 
Conversation:
${conversationText}
 
JSON array:
[{"content": "...", "category": "preference|fact|context", "importance": 0.0-1.0}]`,
        },
      ],
    });
 
    const text = response.content[0].type === "text" ? response.content[0].text : "[]";
    const jsonMatch = text.match(/\[[\s\S]*\]/);
    if (\!jsonMatch) return [];
 
    const parsed = JSON.parse(jsonMatch[0]);
    return parsed.filter(
      (m: { importance: number }) => m.importance >= 0.5
    );
  } catch (error) {
    console.error("Memory extraction failed:", error);
    return []; // Extraction failure doesn't break the conversation
  }
}
 
// Retrieve relevant memories via vector similarity search
async function retrieveRelevantMemories(
  userId: string,
  query: string,
  limit: number = 5
): Promise<string[]> {
  try {
    const embedding = await generateEmbedding(query);
 
    // Composite scoring: similarity * importance for balanced ranking
    const result = await pool.query(
      `SELECT content, importance,
              1 - (embedding <=> $1::vector) AS similarity
       FROM user_memories
       WHERE user_id = $2
         AND created_at > NOW() - INTERVAL '90 days'
       ORDER BY (1 - (embedding <=> $1::vector)) * importance DESC
       LIMIT $3`,
      [JSON.stringify(embedding), userId, limit]
    );
 
    return result.rows.map((row: { content: string }) => row.content);
  } catch (error) {
    console.error("Memory retrieval failed:", error);
    return []; // Graceful degradation — conversation continues without memories
  }
}
 
async function generateEmbedding(text: string): Promise<number[]> {
  const response = await fetch("https://api.openai.com/v1/embeddings", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "text-embedding-3-small",
      input: text,
    }),
  });
 
  if (\!response.ok) {
    throw new Error(`Embedding API error: ${response.status}`);
  }
 
  const data = await response.json();
  return data.data[0].embedding;
}

The similarity * importance composite scoring deserves attention. Pure vector similarity treats "likes Python" and "currently leading a Kubernetes migration" with equal weight. Using importance as a ranking factor rather than a filter balances relevance against significance — you always get the most relevant results, but critical context floats to the top.

The created_at > NOW() - INTERVAL '90 days' clause implements automatic memory decay. Users' tech stacks and interests evolve, and stale memories pollute search results more than they help.

Designing for Persona and Tone Consistency

As conversations grow longer, AI responses tend to drift in tone — starting formal and gradually becoming casual, or using terminology inconsistently. The root cause is usually system prompt design.

The approach that works in production is persona layering: splitting the system prompt into three tiers — base personality, response rules, and contextual adaptation.

interface PersonaConfig {
  basePersonality: string;
  responseRules: string[];
  adaptiveRules: Record<string, string>;
}
 
function buildLayeredSystemPrompt(
  persona: PersonaConfig,
  userLevel: "beginner" | "intermediate" | "advanced",
  conversationMood: "casual" | "professional" | "urgent"
): string {
  // Tier 1: Base personality (immutable)
  let prompt = `# Your Persona\n${persona.basePersonality}\n\n`;
 
  // Tier 2: Response rules (always applied)
  prompt += `# Response Rules\n`;
  for (const rule of persona.responseRules) {
    prompt += `- ${rule}\n`;
  }
 
  // Tier 3: Contextual adaptation (dynamic)
  prompt += `\n# Current Context\n`;
  prompt += `- User's technical level: ${userLevel}\n`;
 
  if (userLevel === "beginner") {
    prompt += `- Always explain jargon in plain language before using it\n`;
    prompt += `- Add line-by-line comments to code examples\n`;
  } else if (userLevel === "advanced") {
    prompt += `- Skip prerequisite explanations, focus on the core\n`;
    prompt += `- Address performance implications and edge cases\n`;
  }
 
  if (persona.adaptiveRules[conversationMood]) {
    prompt += `- ${persona.adaptiveRules[conversationMood]}\n`;
  }
 
  return prompt;
}
 
const techAdvisor: PersonaConfig = {
  basePersonality:
    "Experienced senior engineer. Prioritizes accuracy while being candid about real-world trade-offs. Honest about uncertainty.",
  responseRules: [
    "Always include error handling in code examples",
    "Frame suggestions as options, not directives",
    "Never present more than 3 choices at once (prevent decision fatigue)",
    "When correcting a misconception, acknowledge the user's reasoning first",
  ],
  adaptiveRules: {
    casual: "Conversational tone, occasionally share personal experience",
    professional: "Concise and structured responses",
    urgent: "Lead with the shortest fix, detail comes after",
  },
};

Why three tiers? A monolithic system prompt requires rewriting everything whenever you adjust for context, which introduces unintended personality drift. Separating layers lets you lock the base personality while dynamically tuning only situational behavior.

Multi-Layer Input Validation and Output Filtering

Safety is the most frequently overlooked aspect of production chat systems. You need to handle prompt injection attempts, accidental PII exposure, and potentially harmful content.

import Anthropic from "@anthropic-ai/sdk";
 
interface SafetyCheckResult {
  safe: boolean;
  reason?: string;
  sanitizedInput?: string;
}
 
const client = new Anthropic();
 
async function validateInput(userInput: string): Promise<SafetyCheckResult> {
  // Stage 1: Regex-based fast filter (zero API cost)
  const patterns = {
    creditCard: /\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b/,
    injection:
      /ignore\s+(all\s+)?previous\s+instructions|system\s*prompt|you\s+are\s+now/i,
  };
 
  if (patterns.creditCard.test(userInput)) {
    return {
      safe: false,
      reason: "Credit card number detected. Please don't enter personal financial information.",
    };
  }
 
  if (patterns.injection.test(userInput)) {
    // Neutralize rather than block — preserve legitimate user intent
    const sanitized = userInput.replace(patterns.injection, "[filtered]");
    return { safe: true, sanitizedInput: sanitized };
  }
 
  // Stage 2: Semantic check via Claude Haiku (only when needed)
  if (userInput.length > 500 || containsSuspiciousPatterns(userInput)) {
    try {
      const response = await client.messages.create({
        model: "claude-haiku-4-5",
        max_tokens: 100,
        messages: [
          {
            role: "user",
            content: `Classify this user input for safety. Reply only "safe" or "unsafe: [reason]".
 
Input: "${userInput.slice(0, 1000)}"`,
          },
        ],
      });
 
      const result =
        response.content[0].type === "text" ? response.content[0].text : "safe";
 
      if (result.startsWith("unsafe")) {
        return { safe: false, reason: result.replace("unsafe: ", "") };
      }
    } catch (error) {
      // Safety check failure falls through — availability over perfect filtering
      console.error("Safety check failed:", error);
    }
  }
 
  return { safe: true, sanitizedInput: userInput };
}
 
function containsSuspiciousPatterns(input: string): boolean {
  const keywords = [
    "jailbreak", "bypass", "override", "roleplay as",
    "pretend you", "act as if", "DAN mode",
  ];
  const lower = input.toLowerCase();
  return keywords.some((kw) => lower.includes(kw));
}
 
function sanitizeOutput(output: string): string {
  let sanitized = output;
  // Strip API keys and tokens
  sanitized = sanitized.replace(
    /\b(sk-[a-zA-Z0-9]{20,}|ghp_[a-zA-Z0-9]{36}|AKIA[A-Z0-9]{16})\b/g,
    "[REDACTED]"
  );
  // Strip internal file paths
  sanitized = sanitized.replace(
    /\/(?:home|Users|var|etc)\/[^\s"']+/g,
    "[PATH_REDACTED]"
  );
  return sanitized;
}

The two-stage filtering design is deliberate. Regex catches obvious patterns at zero latency on every request. Semantic classification via Claude Haiku triggers only for long inputs or suspicious patterns, keeping safety check costs minimal.

A common mistake in safety design is the "block everything or allow everything" binary. Neutralizing suspected injection attempts rather than blocking them preserves the experience for legitimate users while defusing the attack.

Cutting Monthly Token Costs by 70%

Production chat system costs grow alarmingly if left unchecked. One project's 2 million monthly turns ran up $8,000/month before optimization. Three changes brought it down to $2,400.

Optimization 1: Prompt Caching — The system prompt and long-term memory section barely change during a conversation, making them ideal cache candidates.

const response = await client.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 2048,
  system: [
    {
      type: "text",
      text: systemPromptBase, // Base personality + rules (stable)
      cache_control: { type: "ephemeral" },
    },
    {
      type: "text",
      text: dynamicContext, // Summary + long-term memories (changes per turn)
    },
  ],
  messages: memory.recentMessages,
});
// Cache hits reduce input token costs by 90%

Optimization 2: Model Routing — Not every response needs Sonnet. Simple confirmations ("Yes, that's correct") and summarization work fine with Haiku. Classifying input complexity and routing to the appropriate model cuts average costs by 40%. The classification itself runs on Haiku at negligible cost.

Optimization 3: Progressive Summary Compression — When generating summaries via the sliding window, periodically re-compress the summaries themselves. By turn 50, early summaries condense to a few sentences containing only the user's fundamental goal and the core problem being solved. This caps system prompt token usage at a constant level regardless of conversation length.

Seven Production Pitfalls You'll Hit

Problems that only surface in real-world operation, with concrete mitigations.

1. Silent Context Degradation — The API throws an error when you exceed the context window, but just below the limit, the model disproportionately attends to recent messages and underweights the system prompt. Keep context usage below 70% of the window as a rule of thumb.

2. Summarization Drift — Each summarization pass loses subtle but important details. "User is stuck on React 18 Suspense" degrades to "User is having React issues." Mitigation: store key entities and topics as structured data alongside the narrative summary.

3. Hallucinated Memories — Claude may infer information during conversation that gets incorrectly saved as long-term memory. If the model guesses a user preference, storing that guess as fact is dangerous. Constrain your extraction prompt to only capture explicitly stated facts.

4. Over-Aggressive Safety Filters — Overly strict filters block legitimate technical discussions about security vulnerabilities or exploit mitigation. For technical chatbots, maintain a domain-specific allowlist so security-related terms in valid questions aren't flagged.

5. Mid-Stream Error Recovery — When network errors hit during SSE streaming, users see incomplete responses. Implement client-side completion detection with automatic retry for truncated streams.

6. Multi-Turn Tool Use State — Managing tool execution results across conversation turns is surprisingly complex when combined with summarization. Tool result messages have a specific structure that breaks when summarized. Exclude tool results from summarization and retain only their textual representation.

7. Timezone Misalignment — "Earlier" and "yesterday" need the user's timezone for correct resolution. Without it, you'll give UTC-based answers that are 9 hours off for users in Japan or 5 hours off for East Coast US users.

Bringing It All Together: The Complete Production Chat Loop

Here's the integrated flow that combines all components into a production-ready conversation loop.

async function productionChatLoop(
  userId: string,
  memory: ConversationMemory,
  userInput: string
): Promise<{ response: string; updatedMemory: ConversationMemory }> {
  // 1. Input safety validation
  const inputCheck = await validateInput(userInput);
  if (\!inputCheck.safe) {
    return {
      response: `I can't process this input. ${inputCheck.reason}`,
      updatedMemory: memory,
    };
  }
  const safeInput = inputCheck.sanitizedInput || userInput;
 
  // 2. Retrieve relevant long-term memories
  const longTermMemories = await retrieveRelevantMemories(userId, safeInput);
 
  // 3. Build context-aware system prompt
  const systemPrompt = buildContextualSystemPrompt(
    memory.summary,
    longTermMemories
  );
 
  // 4. Execute conversation (sliding window + summary buffer)
  const result = await chat(memory, safeInput);
 
  // 5. Output safety filtering
  result.response = sanitizeOutput(result.response);
 
  // 6. Async memory extraction (don't block the response)
  extractMemories(memory.recentMessages.slice(-4)).then((memories) => {
    for (const mem of memories) {
      saveToVectorDB(userId, mem).catch(console.error);
    }
  });
 
  return result;
}

For conversation analytics patterns, see the logging strategies covered in Claude API Chatbot Building Fundamentals. For deeper coverage of vector database memory persistence, check out Production Memory Implementation with pgvector. Authentication and API security fundamentals are covered in the Claude API Production Security Guide.

Your next step is straightforward: implement the sliding window + summary buffer code and verify that summaries generate correctly across a 10-turn conversation. From there, layer in long-term memory, then integrate safety filters. Building all three layers simultaneously makes debugging nearly impossible — take it one layer at a time.