At 0:47 one March morning in 2026, my indie SaaS pager went off with Unexpected end of JSON input. A response from the Claude API had been piped into JSON.parse, and the closing brace simply was not there. The failure rate was eight requests out of about twelve hundred, or roughly 0.66%. That sounds like noise, until you notice one of those eight came from the billing flow and forced me to roll back a charge in the middle of the night.
The official "Use prefilled responses" page makes prefill sound airtight: stick a { in the assistant message and Claude will dutifully complete a JSON object. I believed that at first, too. But once you process thousands of requests a month, you start to see all the corners that simple framing does not cover. Responses get cut off when max_tokens is reached. A small fraction of calls suddenly take a detour into a tool_use block. Adding stop_sequence to keep things tidy ends up amputating nested structures. After running into every one of these in production, I converged on a four-layer pipeline that combines prefill with three defensive fallbacks. This article is the field journal of that design.
I have been building solo apps since 2014, and after eleven years of running an AdMob-driven app business with more than fifty million cumulative downloads, what hit me hardest about putting AI into a production flow was this: when your output is probabilistic, the system around it has to be deterministic. The four-layer defense below is one answer to that gap.
Three Production Failure Modes That Prefill Alone Cannot Cover
Prefilling, in the Claude API, means appending an assistant message to the messages array so the model continues from text you already supplied. Open with { "result": and Claude will, with high probability, finish a coherent JSON object. The trouble is that "high probability" and "100%" are very different animals when you are running production traffic.
Case A: silent truncation at max_tokens
In one of my products I needed to extract roughly thirty fields from a document. I had set max_tokens to 2,048. One day a particularly long input pushed the response past that ceiling, and Claude stopped mid-comma at token 2,049. The stop_reason was max_tokens. The opening { had landed because of prefill, but nothing closed it. JSON.parse had no chance.
Case B: collisions with tool_use blocks
When you enable tools and prefill { in the same call, Claude occasionally hesitates between continuing the JSON and emitting a tool_use block. The docs allude to this with a single sentence about behavior changes, but in practice I saw a handful of cases where Claude ignored the prefilled { and returned a tool_use instead. The downstream handler had been written assuming content[0].type === "text" and crashed with undefined.text. The frequency was below 0.1%, which is exactly the rate of bug you want to design out, not paper over.
Case C: stop_sequence cannibalizing the structure
Trying to be clever, I once added stop_sequences: ["}"] to terminate as soon as JSON closed. This is fine until any nested object appears. The first } from an inner object slammed the door before the outer one closed. That is the kind of self-inflicted wound that makes you respect your own future debugging more.
Accepting that prefill alone could not eliminate these three cases is what finally pushed me into a layered design.
The Four-Layer Pipeline at a Glance
The architecture has four independent layers, each able to fall back to the next:
- Layer 1 — Prefill locks the format by starting the JSON in the assistant message
- Layer 2 — Schema Validation checks the raw output with a strict Zod or Pydantic schema
- Layer 3 — Automatic Repair attempts mechanical fixes first, then asks Claude to repair if needed
- Layer 4 — Graceful Retreat returns a safe default and pages operations when everything else fails
The reason for separating these responsibilities is simple: deterministic recovery beats probabilistic recovery whenever you can use it. My grandfather, a temple carpenter, used to say you should always leave room for repair in whatever you build. That sentence comes back to me a lot when I am drawing pipelines like this.
Layer 1 — Anchoring the Format with Prefill
Here is the cleanest TypeScript version of Layer 1:
// extract.ts — Layer 1: prefill the JSON opening
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
export async function extractWithPrefill(input: string) {
const response = await client.messages.create({
model: "claude-sonnet-4-6",
max_tokens: 4096, // Wider than you think you need; protects against Case A
system: "You are a strict JSON extractor. Output must conform exactly to the requested schema.",
messages: [
{ role: "user", content: `Extract structured data from the following text:\n\n${input}` },
// The prefill: the assistant starts the JSON for us
{ role: "assistant", content: '{\n "result":' },
],
});
// The prefill string is NOT included in response.content; we have to prepend it ourselves
const block = response.content[0];
if (block.type !== "text") {
throw new Error(`Unexpected content block type: ${block.type}`); // Case B detector
}
const raw = '{\n "result":' + block.text;
return raw; // Don't JSON.parse yet — that's Layer 2's job
}Two pieces matter here. First, the prefill text is not included in the SDK response. The returned content is only the continuation, so you must prepend the prefill string yourself before any parsing. Second, set max_tokens generously. It feels wasteful, but token cost is cheaper than a failed extraction that requires manual triage at 2 AM.
I deliberately leave stop_sequences empty by default. Tempting as it is to stop on the first }, that strategy assumes flat JSON, and most real schemas have nested objects.
Layer 2 — Schema Validation with a Discriminating Failure Reason
Layer 2 takes the raw string, parses it, and validates against a schema. The catch is that you should distinguish two failure modes — a syntax break and a schema mismatch — because they want different recovery strategies.
// validate.ts — Layer 2: structural and syntactic validation
import { z } from "zod";
const ResultSchema = z.object({
result: z.object({
title: z.string().min(1),
tags: z.array(z.string()).max(20),
score: z.number().min(0).max(1),
}),
});
export type ValidatedResult = z.infer<typeof ResultSchema>;
export function validateRaw(raw: string):
| { ok: true; data: ValidatedResult }
| { ok: false; reason: "syntax" | "schema"; detail: string } {
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch (e) {
return { ok: false, reason: "syntax", detail: (e as Error).message };
}
const r = ResultSchema.safeParse(parsed);
if (!r.success) return { ok: false, reason: "schema", detail: r.error.message };
return { ok: true, data: r.data };
}A syntax error usually means truncation or an extra comma — fixable mechanically. A schema error means the structure exists but the values are wrong — that needs Claude's help. Conflating them wastes API calls and money.
Layer 3 — Mechanical Repair Saves 80% of Failures
This is the layer with the highest production payoff. In my own measurements, of the 0.66% of requests that reach Layer 3, roughly five out of every six are saved by mechanical repair alone before Claude is ever called again.
3a: Syntax Repair with json5 and Bracket Closing
Start with a more permissive parser. json5 accepts trailing commas, single quotes, and inline comments. If even that fails, count brackets and naively close what is missing.
// repair.ts — Layer 3a: syntax repair
import JSON5 from "json5";
export function repairSyntax(raw: string): string | null {
// (1) Try JSON5: tolerates trailing commas, comments, single quotes
try {
return JSON.stringify(JSON5.parse(raw));
} catch {}
// (2) Naive bracket closing — handles Case A truncation
const opens = (raw.match(/[{[]/g) || []).length;
const closes = (raw.match(/[}\]]/g) || []).length;
if (opens > closes) {
let candidate = raw;
// Close a dangling string literal first
const lastQuote = candidate.lastIndexOf('"');
const beforeQuote = candidate.slice(0, lastQuote);
const escaped = (beforeQuote.match(/\\"/g) || []).length;
const real = (beforeQuote.match(/"/g) || []).length - escaped;
if (real % 2 === 1) candidate += '"';
// Trim trailing comma
candidate = candidate.replace(/,\s*$/, "");
// Add the missing closing brackets
const need = opens - closes;
for (let i = 0; i < need; i++) candidate += "}";
try {
JSON.parse(candidate);
return candidate;
} catch {}
}
return null;
}This code looks unsophisticated, but it recovers more than 80% of Case A truncations in my logs. The json5 package is mature, well-maintained, and in six months of production traffic I have not had a single failure that traced back to the library itself.
3b: Asking Claude to Repair the JSON
When mechanical repair gives up, ask Claude itself. Keep the prompt short, deterministic, and free of pleasantries.
// repair.ts — Layer 3b: Claude-driven repair
export async function repairWithClaude(raw: string, schema: string): Promise<string | null> {
const response = await client.messages.create({
model: "claude-haiku-4-5", // Repair is cheap and Haiku does it cleanly
max_tokens: 4096,
system: "You repair broken JSON to strictly match the given schema. Output ONLY the corrected JSON, no prose, no commentary.",
messages: [
{
role: "user",
content: `Repair the following broken JSON so that it conforms to this schema.\n\n[Schema]\n${schema}\n\n[Broken JSON]\n${raw}`,
},
{ role: "assistant", content: "{" },
],
});
const block = response.content[0];
if (block.type !== "text") return null;
return "{" + block.text;
}I switched from Sonnet to Haiku 4.5 for repair because the task is mechanically simple: given a broken JSON, make it valid against a schema. Sonnet sometimes "improves" the data in ways that violate the original intent. Haiku tends to repair surgically. In my logs the success rate of Haiku-based repair sits at about 92%, at roughly 0.0008 USD per call as of May 2026.
Layer 4 — Graceful Retreat with a Distinct Operational Signal
For the sub-0.05% of requests that survive all three layers, you need a deliberate retreat. The principle is to separate what the user sees from what operations sees.
// run.ts — wiring up the four layers
import { extractWithPrefill } from "./extract";
import { validateRaw } from "./validate";
import { repairSyntax, repairWithClaude } from "./repair";
const SCHEMA_TEXT = JSON.stringify({
type: "object",
properties: { result: { type: "object", required: ["title", "tags", "score"] } },
required: ["result"],
});
export async function run(input: string) {
// Layer 1
const raw = await extractWithPrefill(input);
// Layer 2
let v = validateRaw(raw);
if (v.ok) return { ok: true, data: v.data, layer: 1 };
// Layer 3a — mechanical repair
if (v.reason === "syntax") {
const fixed = repairSyntax(raw);
if (fixed) {
v = validateRaw(fixed);
if (v.ok) return { ok: true, data: v.data, layer: "3a" };
}
}
// Layer 3b — Claude repair
const repaired = await repairWithClaude(raw, SCHEMA_TEXT);
if (repaired) {
const r2 = validateRaw(repaired);
if (r2.ok) return { ok: true, data: r2.data, layer: "3b" };
}
// Layer 4 — graceful retreat
await alertOps({ raw, lastDetail: v.ok ? "" : v.detail });
return { ok: false, layer: 4, fallback: defaultFallback() };
}
function defaultFallback() {
return { result: { title: "", tags: [], score: 0 } };
}
async function alertOps(payload: unknown) {
// For solo work, a Slack webhook plus a Sentry breadcrumb is enough
await fetch(process.env.SLACK_WEBHOOK_URL!, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ text: ":warning: JSON 4-layer fallback used" }),
});
}My rule of thumb is: return an empty-but-shaped result to the user so the application keeps moving, and ping operations immediately so the rare miss gets investigated. Before this layer existed, I was being woken up roughly two or three nights a week. After it landed, the wake-up rate dropped below one per month.
A Python Sketch for the Same Pipeline
For Python users, here is the equivalent of Layers 1 and 2 with pydantic and json_repair. I lean on json_repair for what would otherwise be Layer 3a in TypeScript.
# extract.py — Python version of Layer 1 + 2 with built-in repair
from anthropic import Anthropic
from pydantic import BaseModel, ValidationError, Field
import json
import json_repair
client = Anthropic()
class Inner(BaseModel):
title: str = Field(min_length=1)
tags: list[str] = Field(max_length=20)
score: float = Field(ge=0.0, le=1.0)
class Result(BaseModel):
result: Inner
def extract(input_text: str):
resp = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=4096,
system="You are a strict JSON extractor.",
messages=[
{"role": "user", "content": f"Extract structured data from:\n\n{input_text}"},
{"role": "assistant", "content": '{\n "result":'},
],
)
block = resp.content[0]
if block.type != "text":
raise RuntimeError(f"unexpected block: {block.type}")
raw = '{\n "result":' + block.text
try:
return Result.model_validate_json(raw)
except (json.JSONDecodeError, ValidationError):
# json_repair gives us a one-liner equivalent of Layer 3a
repaired = json_repair.repair_json(raw)
return Result.model_validate_json(repaired)The json_repair package on PyPI (GitHub link) is the single most underrated dependency in my Python stack. It quietly handles the long tail of broken JSON cases without me having to think.
Production Numbers: Before and After
Here are the actual numbers from my indie SaaS, which sits at about 4,200 monthly extraction calls:
- Before (prefill only)
- Parse success rate: 99.34% (28 of 4,200 failed)
- Monthly operational alerts: 11
- Mean time to manual recovery: 14 minutes (after a user reported it)
- After (four-layer defense)
- Parse success rate: 99.98% (1 of 4,200 reaches Layer 4, and even that user sees a usable empty shell)
- Monthly operational alerts: under 1
- Mean time to recovery: 0 minutes (automatic retreat)
- Additional cost: roughly 7 to 9 USD per month from Layer 3b Haiku calls
What mattered more than the numbers was that my sleep stabilized. When you put AI in production as an individual developer, operational quality directly shapes how much of your life you get back. Designing this pipeline taught me that protecting that boundary is part of the engineering, not separate from it.
Three Pitfalls Worth Naming
Here are the mistakes I actually made along the way. You can skip them.
- Do not use Sonnet for Layer 3b repair. It is more expensive, and it has a tendency to "improve" the data rather than restore it. Haiku stays surgical, which is what you want.
- Keep Layer 3b's prompt schema in sync with your validator. I had a week where my Zod schema added a field but the repair prompt still referenced the old one. Repair worked — into the wrong shape.
- A safe default is not always an empty default. For search results, an empty array is fine. For pricing or counters, returning
0can show users misleading information. Pick a fallback value that is safe in your domain, not just empty.
Two adjacent reads pair well with this article: Claude API Structured Output: Practical Mastery covers schema design strategy, and Troubleshooting JSON Parse Errors with Claude API is the entry-level companion to this piece. If you also use prompt caching, Prompt Caching for Cost Optimization shows how to bring Layer 3b's repair calls down even further.
Observability: Know Which Layer Saved You
Once the pipeline is in place, the most useful operational metric is not the global success rate but the distribution across layers. Knowing that 0.40% of requests are saved by Layer 3a and 0.05% by Layer 3b tells you whether your prompts are deteriorating, whether max_tokens is too tight, or whether a recent schema change has introduced new failure modes.
// observability.ts — emit a counter per layer outcome
type LayerOutcome = "1" | "3a" | "3b" | "4";
export function recordOutcome(outcome: LayerOutcome) {
// Cloudflare Analytics Engine, Sentry breadcrumbs, or a Postgres counter all work
fetch(process.env.METRICS_ENDPOINT!, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ metric: "json_layer_outcome", outcome, ts: Date.now() }),
}).catch(() => {});
}In my dashboard I keep three trend lines: success-at-Layer-1 should hover above 99%, Layer 3a should stay under 1%, and Layer 3b should be vanishingly small. When Layer 3a starts climbing, it usually means the input distribution has shifted (a new client started sending longer documents) and max_tokens needs to grow. When Layer 3b climbs, it usually means the underlying prompt or schema has drifted and the fix is upstream.
Tying this to alert thresholds is straightforward. I page myself when the seven-day Layer 3b rate exceeds 0.2%, because that consistently correlates with a real upstream regression rather than transient noise. Anything below that is "interesting but not urgent" and gets reviewed at the weekly check-in.
Cost Math You Can Reason About
For solo developers worried about the marginal cost of adding repair calls, here is the model I use. Let r be the failure rate at Layer 1, m3a the share of those failures recovered by 3a, and m3b the share of remaining failures recovered by 3b.
- Cost added per request =
r * (1 - m3a) * cost_haiku_repair - For my numbers,
r ≈ 0.0066,m3a ≈ 0.83,cost_haiku_repair ≈ 0.0008 USD, giving roughly0.0066 * 0.17 * 0.0008 ≈ 0.0000009 USDper request, or under one ten-thousandth of a cent. At 4,200 monthly requests, that is barely measurable.
The reason the cost is negligible is that Layer 3b only fires for the small slice that 3a failed on. The dominant cost remains Layer 1 itself, where you are paying full Sonnet rates for the actual extraction. If anything, this design lets you raise max_tokens on Layer 1 without anxiety, because the repair pipeline catches whatever truncation slips through.
If you want to push costs even lower, two changes work:
- Cache the system prompt and schema description with prompt caching. Layer 1 calls share the same system message, so you can capture roughly 60 to 80% of input-token cost at break-even after a few hundred calls per day.
- Move Layer 3b to a smaller off-hours model when latency is not user-facing. For batch jobs, I run Layer 3b on a queued worker that delays repair by up to thirty seconds without anyone noticing.
Production Edge Cases: Tools and Streaming
Tool Use and Prefill Together (Case B Revisited)
Case B — the rare instance where prefill collides with tool definitions — deserves a more concrete remedy than "be aware of it." If your endpoint genuinely needs both tools and structured output, I have settled on this pattern:
// dual.ts — separating tool routing from JSON extraction
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
export async function extractOrToolCall(input: string, tools: any[]) {
// First call: decide whether tools are needed; no prefill
const decide = await client.messages.create({
model: "claude-sonnet-4-6",
max_tokens: 1024,
system: "Decide if any of the listed tools are required to answer the user. If yes, call the tool. If no, respond with the literal text NEED_EXTRACTION.",
tools,
messages: [{ role: "user", content: input }],
});
if (decide.stop_reason === "tool_use") {
return { kind: "tool", response: decide };
}
// Second call: pure extraction, with prefill — no tools attached
const extract = await client.messages.create({
model: "claude-sonnet-4-6",
max_tokens: 4096,
system: "Output strict JSON matching the schema.",
messages: [
{ role: "user", content: input },
{ role: "assistant", content: '{
"result":' },
],
});
return { kind: "extract", raw: '{
"result":' + (extract.content[0] as any).text };
}The trick is to separate the routing decision from the extraction call. Once routing is decided in a tools-enabled call, you can run a second tools-free call with prefill attached and never see Case B again. This adds one round trip, but for the small fraction of routes that actually need tools, the cost is acceptable. For routes that always extract, you can skip the routing step entirely.
Handling Streaming Responses Inside the Pipeline
Many production deployments stream tokens to the client as they arrive. The four-layer design still applies, but you need to buffer the stream until the response completes and the stop_reason is known, otherwise validation has nothing solid to chew on.
// stream.ts — buffering before validation
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
export async function streamThenValidate(input: string) {
const stream = await client.messages.stream({
model: "claude-sonnet-4-6",
max_tokens: 4096,
messages: [
{ role: "user", content: input },
{ role: "assistant", content: '{
"result":' },
],
});
let buffer = '{
"result":';
for await (const event of stream) {
if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
buffer += event.delta.text;
// You can still ship deltas to the user UI here, just don't validate yet
}
}
return buffer; // Now hand this to validateRaw()
}For products with strict latency SLAs, you can ship the partial buffer to the client for visual feedback and only run validation once the stream completes. If validation fails, you fall through to Layers 3 and 4 server-side without exposing the failure to the front-end.
When the Pipeline Is Overkill
A few situations where the four-layer pipeline is overkill:
- One-shot internal scripts with no SLA and no user. Just catch the parse error and retry once.
- Prompts that emit free-form prose, not JSON. This pipeline is purely for structured outputs.
- Use cases where the cost of a wrong answer is so high that you should not silently fall back at all. In that case, Layer 4 should hard-fail and surface the error to the caller rather than returning a default. The pipeline still applies, but the "graceful retreat" becomes a "loud refusal."
The right test for whether you need this is to look at how often you have manually recovered from a JSON parse failure in the last quarter. If the answer is "more than zero," the layered design pays for itself within a few weeks.
A Year of Living With This Pipeline
It is one thing to build a defense and another to live with it. After running this design for the better part of a year across two of my products, the most useful thing I can share is what changed in the texture of operations.
The first surprise was that the bug reports stopped looking like bugs. Before the pipeline, "the AI broke the page" tickets came in through email, support chat, and once memorably through a tweet at midnight. After the pipeline, the same underlying failures still happened — they just stopped reaching users. What I see now in operations are dashboards with quiet bumps in Layer 3a usage, which I investigate at a calm hour rather than reactively. That difference, more than the raw success rate, is what I would put in front of any indie developer thinking about whether the additional plumbing is worth it.
The second surprise was that the design taught me where my prompts were brittle. Whenever Layer 3b started spiking for one specific endpoint, it almost always pointed back to a system prompt that had grown organic over six months and was now nudging Claude into ambiguous territory. Treating Layer 3b telemetry as a code smell rather than a runtime problem turned out to be the cheapest way to find prompt regressions. I now do a small audit of system prompts whenever the seven-day Layer 3b rate creeps over 0.1%, and roughly half the time I end up trimming or restructuring the prompt rather than touching the pipeline at all.
The third surprise was about how this changed my willingness to ship. Before the pipeline, "ship a feature that talks to Claude" was preceded by two days of paranoia and a reluctance to enable it for paying customers. After the pipeline, I started shipping AI-touching code paths on the same cadence as any other backend feature, because I trusted the safety net underneath. Speed of iteration matters more than I expected; in the months since this pipeline went live, I have launched two new AI-driven features that I would have stalled on indefinitely beforehand.
If there is a single takeaway from a year of using this design, it is that defensive infrastructure is not a tax on velocity, it is a precondition for it. The same principle applies whether you are running a solo SaaS or a team product: invest once in the safety layer, and afterwards you get to think about features instead of failure modes.
What to Try Next
The honest, lowest-risk way to adopt this pattern is to wire only Layers 1 and 2 into a single extraction endpoint and log the failure rate for one week. With a real number in hand, you can decide whether you actually need Layer 3 at all, or only its mechanical half. I almost over-engineered Layer 3b before I had measured anything; the measurement saved me from that. Measure first, then defend — in that order, this design comes in cheap and stays out of your way.
Thank you for reading. I hope this saves your team a few late nights.