When Thinking Is Always On, Prefill Quietly Stops Working — Fixing Streaming and Token Budgets for Fable 5

Broken assumption 2: the first streamed block isn't text

In the stage that renders incrementally to a UI, I treated the first streamed content_block as text. With thinking on, the first block is a thinking block. Render the head as body and you either show fragments of reasoning or stall on a type mismatch.

Route by the block type instead. Keep thinking_delta out of the body and show only text_delta. A signed signature_delta also streams through; it is not for display.

def stream_answer(prompt: str):
    answer_parts = []
    current_type = None
 
    with client.messages.stream(
        model=MODEL,
        max_tokens=12000,
        messages=[{"role": "user", "content": prompt}],
    ) as stream:
        for event in stream:
            if event.type == "content_block_start":
                # only here do you learn the block type (thinking or text)
                current_type = event.content_block.type
            elif event.type == "content_block_delta":
                delta = event.delta
                if delta.type == "text_delta":
                    answer_parts.append(delta.text)
                    yield delta.text          # the only thing readers see
                elif delta.type == "thinking_delta":
                    pass                       # record only; never display
                elif delta.type == "signature_delta":
                    pass                       # internal use
            elif event.type == "content_block_stop":
                current_type = None
 
    return "".join(answer_parts)

The expected behavior is that only text_delta reaches the reader and no reasoning fragments are ever shown. I once appended thinking deltas straight into the buffer and leaked the model's internal monologue into an article preview. Routing by type is the only reliable guard.

There is a second catch for agents that also use tools. When you return a tool result and continue the conversation, you must put the thinking block from the previous assistant turn back into history, signature and all. Drop the thinking block and return only the tool_result and you break reasoning continuity, sometimes with an error. I had been trimming history down to "text only," which tripped me here too. Alongside a guide to handling Claude's stop_reason, review your history-reconstruction logic before you ship.

Broken assumption 3: max_tokens is a tug-of-war between thinking and body

Set max_tokens small, as if it were the body limit, and on a turn where thinking runs long the body has no room left. stop_reason comes back as max_tokens and the body is short or empty. Because adaptive thinking scales with how hard the problem is, the same prompt produces a thin body only sometimes — a low-reproducibility symptom.

Two countermeasures. First, explicitly budget headroom for thinking in max_tokens. Second, detect stop_reason == "max_tokens", raise the budget, and retry exactly once.

def answer_with_budget(prompt: str, want_answer_tokens: int = 2000) -> str:
    # Stack several times the expected body for thinking (harder = longer)
    budget = want_answer_tokens * 4
 
    for attempt in range(2):
        msg = client.messages.create(
            model=MODEL,
            max_tokens=budget,
            messages=[{"role": "user", "content": prompt}],
        )
        text = "".join(b.text for b in msg.content if b.type == "text")
 
        # cut off mid-thinking, before any body
        if msg.stop_reason == "max_tokens" and not text.strip():
            budget *= 2          # raise the budget and retry once
            continue
        return text
 
    raise RuntimeError("No body even after raising the budget")

Look at usage and you'll see thinking tokens count toward output billing. Measuring my own pipeline for a week, output tokens per request noticeably rose while body length stayed the same. Thinking is not free preprocessing; it is a cost to account for. For designing around always-on thinking from the cost side, a production cost analysis of Claude extended thinking goes deeper, and for the trade-off with long single-pass generation, notes on generating long content in one pass with Fable 5's 128k output is useful context.

Why I dropped prefill and leaned on tools

Dropping prefill felt like a loss. It was short, reliable, and a habit of many years. I leaned on tools anyway because on a thinking-on model another block always precedes the output, so "continue from a seed" simply no longer holds.

If you want a guaranteed shape, constrain the structure, not the next characters. That looked like the long way around, but it was more robust precisely because it doesn't depend on whether the model thinks. The biggest lesson from the migration wasn't a technique — it was raising the level of what I guarantee.

Three things worth checking before you migrate

If you have a stage moving to a thinking-on model, verifying these three things on real calls before deploy will spare you the quiet failures I hit.

First, find any stage that depends on prefill or a custom temperature; replace it with a forced tool call and a validation loop. Second, confirm your streaming code routes by the content_block type — if it assumes the head is text, fix it. Third, check that max_tokens leaves headroom for thinking and that you retry on stop_reason == "max_tokens".

Swap a single stage to the thinking-on model first, confirm it produces none of the three symptoms (empty, thin, truncated), and only then widen it across your pipeline. If this spares one person the same quiet failure, I'll be glad.