Combining Haiku 4.5, Streaming, and Prompt Caching to Cut Costs in a Personal App — An Implementation Record

About six months after integrating Claude API into my personal app, something felt off. The feature was working, but at the end of each month, the API bill was climbing faster than the user count.

I sat down with the request logs and traced the problem to three overlapping issues: the model wasn't right for the task; streaming was introducing instability in certain edge cases rather than improving the experience; and the same system prompt was being re-sent with every single request.

I've been building iOS and Android apps independently since 2014, and one thing that experience teaches you is that the difference between a sustainable product and an expensive one often comes down to how you use your tools, not just which tools you use. That's what this article is about.

Why Three Techniques Were Needed Together

Before combining them, I tried each approach individually. That's where the real learning happened.

Haiku 4.5 alone delivered on the promise of being faster and cheaper than Sonnet. But switching models without updating prompts introduced inconsistency. Instructions that Sonnet followed reliably were occasionally ignored by Haiku — not randomly, but in a pattern I had to dig into.

Streaming alone improved the perceived responsiveness of the app. But it introduced new failure modes: dropped connections, partial response rendering, and edge cases in the UI that hadn't existed with the simple request-response model. Streaming looks simple on the surface but has hidden complexity in implementation.

Prompt caching alone didn't hit the theoretical savings I expected. The reason, as I later discovered, was a misunderstanding of where to place the cache breakpoint. Placing it incorrectly doesn't just fail to save money — it adds cache-write costs on top.

Combining all three required understanding how they interact. Here's the full sequence.

Environment and App Profile

The app is a personalized content app for iOS and Android. Users enter text, Claude generates a response, and the result is displayed in-app. The request profile looks like this:

• System prompt: ~800 tokens (app-specific instructions, mostly static)
• User input: 50–200 tokens (changes every request)
• Expected output: 100–400 tokens

Backend: Node.js + TypeScript. Claude integration: the official Anthropic SDK. Monthly active requests: tens of thousands.

import Anthropic from "@anthropic-ai/sdk";
 
const client = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

Step 1: Migrating to Haiku 4.5 and Prompt Redesign

What Went Wrong with the Initial Migration

Haiku 4.5 occasionally failed to follow instructions that Sonnet handled without issue. After analyzing request logs, I identified the pattern: instructions placed in the second half of a long system prompt were being deprioritized.

The fix was to restructure the system prompt, moving the most critical instructions to the top and reducing the total density. I also found that Haiku responds better to instructions written in flowing sentences rather than bullet-point lists.

// Before: overpacked system prompt with mixed priorities
const systemPromptV1 = `
You are the assistant for [App Name].
- Respond to user input
- Always respond in Japanese
- Use a friendly tone
- Keep responses under 200 characters
- Avoid technical jargon
- Include 1-2 emoji
- Avoid negative statements
- Include encouraging content
... (continues)
`;
 
// After: critical instructions consolidated at the top
const systemPromptV2 = `
You are the assistant for [App Name].
Respond to user input in Japanese, within 200 characters, in an upbeat and encouraging tone.
 
Required:
- Always respond in Japanese
- Keep responses to 100-150 characters
- Include 1-2 emoji
- Maintain a positive tone
`;

Instruction compliance stabilized noticeably after this change.

Model Configuration

const response = await client.messages.create({
  model: "claude-haiku-4-5-20251001",
  max_tokens: 512,
  system: systemPromptV2,
  messages: [
    {
      role: "user",
      content: userInput,
    },
  ],
});

Step 2: Streaming Implementation and Its Hidden Issues

When Streaming Helps vs. When It Doesn't

Streaming improves experience when output is long and users are consciously waiting. For short output — under 100 characters — streaming can actually feel worse. Tokens appearing one by one look more natural when the output is substantial; for brief responses, it looks choppy.

Since most of my app's responses are under 200 characters, I added conditional routing:

type ContentRequest = {
  userInput: string;
  expectedLength: "short" | "long";
};
 
async function generateContent(req: ContentRequest): Promise<string> {
  if (req.expectedLength === "short") {
    return await generateWithoutStreaming(req.userInput);
  } else {
    return await generateWithStreaming(req.userInput);
  }
}

The Core Streaming Pattern

The most important part of streaming in a production mobile app isn't the happy path — it's handling disconnections and partial responses.

async function generateWithStreaming(userInput: string): Promise<string> {
  const chunks: string[] = [];
 
  try {
    const stream = await client.messages.stream({
      model: "claude-haiku-4-5-20251001",
      max_tokens: 1024,
      system: systemPromptV2,
      messages: [{ role: "user", content: userInput }],
    });
 
    for await (const chunk of stream) {
      if (
        chunk.type === "content_block_delta" &&
        chunk.delta.type === "text_delta"
      ) {
        chunks.push(chunk.delta.text);
        // Forward to UI via WebSocket/SSE here if needed
      }
    }
 
    const finalMessage = await stream.finalMessage();
 
    if (finalMessage.stop_reason !== "end_turn") {
      console.warn(
        `Unexpected stop_reason: ${finalMessage.stop_reason}`,
        { input: userInput }
      );
    }
 
    return chunks.join("");
  } catch (error) {
    if (error instanceof Anthropic.APIConnectionError) {
      console.error("Stream connection error:", error.message);
      throw error;
    }
    throw error;
  }
}

The stop_reason Problem I Didn't Expect

After deployment, logs showed stop_reason: "max_tokens" appearing sporadically. I had set max_tokens: 512, which should have been more than enough for 200-character Japanese responses.

The issue: Japanese characters can span multiple tokens. A "200-character" constraint in the system prompt doesn't map cleanly to a token budget. Responses of 400–500 characters were occasionally being generated, running into the token limit mid-output.

The fix was to reduce max_tokens to 256 and tighten the prompt instruction from "under 200 characters" to "approximately 100–150 characters." The buffer matters.

Step 3: Prompt Caching Design

Understanding the Caching Boundary Correctly

Three rules to internalize:

1. A cache entry is created when a request includes cache_control: { type: "ephemeral" } for the first time
2. The cache hits when all content before the cache breakpoint is identical in a subsequent request
3. Cache entries expire after 5 minutes (ephemeral)

My initial mistake was placing the cache breakpoint after the user input — the part that changes every request. This meant the cache never matched.

// Wrong: breakpoint placed after user input (which changes every request)
const badRequest = {
  model: "claude-haiku-4-5-20251001",
  max_tokens: 256,
  system: [{ type: "text", text: systemPromptV2 }],
  messages: [
    {
      role: "user",
      content: [
        {
          type: "text",
          text: userInput,
          cache_control: { type: "ephemeral" }, // ← breakpoint after variable content
        },
      ],
    },
  ],
};
 
// Correct: breakpoint placed on the fixed system prompt
const goodRequest = {
  model: "claude-haiku-4-5-20251001",
  max_tokens: 256,
  system: [
    {
      type: "text",
      text: systemPromptV2, // ← this is static
      cache_control: { type: "ephemeral" }, // ← breakpoint on fixed content
    },
  ],
  messages: [
    {
      role: "user",
      content: userInput, // ← this changes each request, which is fine
    },
  ],
};

Monitoring Cache Hit Rate

The API response includes token breakdown fields: cache_creation_input_tokens and cache_read_input_tokens. Tracking these tells you whether the cache is actually working.

interface UsageStats {
  inputTokens: number;
  outputTokens: number;
  cacheCreationTokens: number;
  cacheReadTokens: number;
}
 
function extractUsage(response: Anthropic.Message): UsageStats {
  return {
    inputTokens: response.usage.input_tokens,
    outputTokens: response.usage.output_tokens,
    cacheCreationTokens: response.usage.cache_creation_input_tokens ?? 0,
    cacheReadTokens: response.usage.cache_read_input_tokens ?? 0,
  };
}
 
function calculateCacheHitRate(stats: UsageStats[]): number {
  const cacheHits = stats.filter((s) => s.cacheReadTokens > 0).length;
  return (cacheHits / stats.length) * 100;
}

My initial cache hit rate was 30–40%. The 5-minute TTL meant that during quiet periods, the cache expired before the next request arrived.

Keeping the Cache Warm

The solution was to proactively refresh the cache before it expired:

async function warmUpCache(): Promise<void> {
  try {
    await client.messages.create({
      model: "claude-haiku-4-5-20251001",
      max_tokens: 1, // minimal tokens to keep cost low
      system: [
        {
          type: "text",
          text: systemPromptV2,
          cache_control: { type: "ephemeral" },
        },
      ],
      messages: [{ role: "user", content: "ping" }],
    });
    console.log("Cache warmed up at:", new Date().toISOString());
  } catch (error) {
    // Warmup failure is non-critical
    console.warn("Cache warmup failed:", error);
  }
}
 
// Refresh every 4 minutes (before the 5-minute TTL expires)
setInterval(warmUpCache, 4 * 60 * 1000);

After adding this, cache hit rate stabilized at 70–80%.

Step 4: The Combined Implementation

Unified Request Function

interface GenerateOptions {
  userInput: string;
  streaming: boolean;
  onStreamChunk?: (chunk: string) => void;
}
 
async function generateOptimized(
  options: GenerateOptions
): Promise<{ text: string; usage: UsageStats }> {
  const { userInput, streaming, onStreamChunk } = options;
 
  const systemConfig = [
    {
      type: "text" as const,
      text: systemPromptV2,
      cache_control: { type: "ephemeral" } as const,
    },
  ];
 
  if (streaming && onStreamChunk) {
    const chunks: string[] = [];
    const stream = await client.messages.stream({
      model: "claude-haiku-4-5-20251001",
      max_tokens: 256,
      system: systemConfig,
      messages: [{ role: "user", content: userInput }],
    });
 
    for await (const chunk of stream) {
      if (
        chunk.type === "content_block_delta" &&
        chunk.delta.type === "text_delta"
      ) {
        chunks.push(chunk.delta.text);
        onStreamChunk(chunk.delta.text);
      }
    }
 
    const finalMessage = await stream.finalMessage();
    return { text: chunks.join(""), usage: extractUsage(finalMessage) };
  } else {
    const response = await client.messages.create({
      model: "claude-haiku-4-5-20251001",
      max_tokens: 256,
      system: systemConfig,
      messages: [{ role: "user", content: userInput }],
    });
 
    const text =
      response.content[0].type === "text" ? response.content[0].text : "";
    return { text, usage: extractUsage(response) };
  }
}

Retry Handling

async function generateWithRetry(
  options: GenerateOptions,
  maxRetries = 3
): Promise<{ text: string; usage: UsageStats }> {
  let lastError: Error | null = null;
 
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await generateOptimized(options);
    } catch (error) {
      lastError = error as Error;
 
      if (error instanceof Anthropic.RateLimitError) {
        const waitMs = Math.pow(2, attempt) * 1000;
        console.warn(`Rate limit hit. Retrying in ${waitMs}ms...`);
        await new Promise((resolve) => setTimeout(resolve, waitMs));
        continue;
      }
 
      if (error instanceof Anthropic.APIConnectionError) {
        await new Promise((resolve) => setTimeout(resolve, 500));
        continue;
      }
 
      throw error;
    }
  }
 
  throw lastError ?? new Error("Max retries exceeded");
}

Before and After

I won't put a specific percentage on the cost reduction — API pricing changes, and your results will depend on your request volume and prompt structure. But the direction of change was clear.

The model switch to Haiku 4.5 had the largest single impact on cost. Streaming primarily improved user experience rather than cost. Prompt caching contributed meaningfully once the hit rate was stable above 70%.

The prompt restructuring that happened as a side effect of the Haiku migration — making instructions clearer and more concise — also improved output quality in a way that wasn't anticipated. Fewer edge cases in outputs meant fewer edge cases to handle in the UI.

Key Lessons

On Haiku 4.5: Front-load your critical instructions. If instruction compliance drops after switching from Sonnet, the issue is almost certainly prompt structure, not model capability.

On streaming: Don't apply it uniformly. Short outputs (<100 chars) are often better delivered all at once. Always log stop_reason — max_tokens hits in production are a warning sign that your token budget and prompt constraints are misaligned.

On prompt caching: Cache breakpoints belong on fixed content. Cache warmup is essential for apps with intermittent traffic. Without it, you'll create cache entries but rarely hit them.

Closing Note

The work of building apps independently — something I've been doing since 2014 — comes down to making decisions with limited information and iterating from there. The same applies to integrating AI.

None of the three techniques in this article are complicated on their own. What makes the difference is understanding how they interact, and being willing to instrument your system well enough to see what's actually happening.

If you're working on a personal app that uses Claude API and feeling like the costs are higher than they should be, start with logs before changing anything. The problem is usually not what you think it is at first.

Related Reading

• Fixing Claude API Streaming Disconnections
• Type-Safe Tool Calling with Claude API Streaming
• Claude Haiku 4.5 Complete Guide — When to Use the Lightweight Model