"Extended Thinking improves quality" — true enough. But knowing which tasks justify the cost? That only becomes clear once you measure.
I run several AI-focused tech blogs on an automated pipeline using Cowork's scheduled tasks. To improve content quality, I tested Extended Thinking in production and spent three months measuring cost, quality, and latency across task types. Here's what I found — with the actual numbers.
Experiment Design
Measurement period: February–April 2026. I embedded an A/B structure in my live article pipeline, alternating Extended Thinking ON and OFF for identical prompts and logging results.
Models used: claude-sonnet-4-6 (thinking budget 8,000–16,000 tokens) and claude-opus-4-6 (10,000–20,000 tokens). Tasks were organized into five categories.
import anthropic
import time
client = anthropic.Anthropic()
def generate_with_thinking(prompt: str, budget: int = 10000):
start = time.time()
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=budget + 4000,
thinking={"type": "enabled", "budget_tokens": budget},
messages=[{"role": "user", "content": prompt}]
)
elapsed = time.time() - start
return {
"output": next(b.text for b in response.content if b.type == "text"),
"input_tokens": response.usage.input_tokens,
"output_tokens": response.usage.output_tokens,
"elapsed": elapsed
}Costs were calculated using Anthropic's official rates: Sonnet 4.6 at $3/MTok input, $15/MTok output, and $15/MTok thinking tokens.
Results by Task Type
I measured approximately 2,800 tasks across five categories. Quality scores were hand-evaluated on a 5-point scale by sampling 20 items per condition.
Task A: Technical Article Generation (3,000–5,000 characters)
The highest-volume task. Extended Thinking OFF produced quality scores of 3.8/5 at $0.018 per task (12 seconds). With 8,000 thinking tokens: 4.0/5 at $0.041 (34 seconds). With 16,000 tokens: 4.1/5 at $0.068 (58 seconds).
Conclusion: Extended Thinking is not worth it here. Cost triples while quality improves by only 0.2–0.3 points (~5%).
Task B: Code Design Review and Refactoring Suggestions
Extended Thinking OFF: 3.5/5 at $0.031 (18 seconds). With 10,000 tokens: 4.4/5 at $0.082 (48 seconds). With 20,000 tokens: 4.6/5 at $0.134 (89 seconds).
Conclusion: Extended Thinking clearly delivers here. Quality improved by ~0.9–1.1 points (~25%). Without it, reviews were surface-level ("rename this variable"). With it, they became substantive: "This design will cause N+1 queries. Specifically, in scenario X, O(n) queries will fire."
system_prompt = """
You are a senior software engineer.
Review the code for:
1. Performance bottlenecks
2. Design issues affecting long-term maintainability
3. Unhandled edge cases
Explain each with a concrete scenario and estimated impact.
"""
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=22000,
thinking={"type": "enabled", "budget_tokens": 10000},
system=system_prompt,
messages=[{"role": "user", "content": f"Review this code:\n\n{code}"}]
)Task C: Multi-step Comparative Analysis
"Compare A and B, recommend the best option given constraint C." Extended Thinking OFF: 3.6/5 at $0.024. With 12,000 tokens: 4.5/5 at $0.094.
Conclusion: Extended Thinking helps significantly here. Without it, the model tends to present both options in parallel. With it, the model recognizes that "the best answer depends on the user's preconditions" and structures the response accordingly.
Task D: Format Conversion and Extraction
MDX frontmatter generation, tag extraction, JSON formatting — tasks with fixed input/output schemas. Extended Thinking OFF: 4.7/5 at $0.007. With 5,000 tokens: 4.8/5 at $0.024.
Conclusion: Do not use Extended Thinking here. Quality is already high and the improvement is negligible. You're paying 3x for nothing.
Task E: Long Document Summarization
Summarizing 10,000+ character documents into ~500-character summaries. OFF: 4.1/5 at $0.029. With 8,000 tokens: 4.3/5 at $0.071.
Conclusion: Borderline. Quality improves by 0.2 points (~5%), but the improvement in interpretive accuracy feels more meaningful than the number suggests. Use it only when cost isn't a primary concern.
Decision Framework and Implementation
Here's the thinking policy I implemented:
from enum import Enum
from dataclasses import dataclass
from typing import Optional
class TaskType(Enum):
FORMATTING = "formatting"
SIMPLE_GENERATION = "generation"
ANALYSIS = "analysis"
CODE_REVIEW = "code_review"
COMPLEX_REASONING = "reasoning"
@dataclass
class ThinkingConfig:
enabled: bool
budget_tokens: Optional[int] = None
THINKING_POLICY: dict[TaskType, ThinkingConfig] = {
TaskType.FORMATTING: ThinkingConfig(enabled=False),
TaskType.SIMPLE_GENERATION: ThinkingConfig(enabled=False),
TaskType.ANALYSIS: ThinkingConfig(enabled=True, budget_tokens=10000),
TaskType.CODE_REVIEW: ThinkingConfig(enabled=True, budget_tokens=12000),
TaskType.COMPLEX_REASONING: ThinkingConfig(enabled=True, budget_tokens=20000),
}
def build_params(prompt: str, task_type: TaskType, model: str = "claude-sonnet-4-6") -> dict:
config = THINKING_POLICY[task_type]
params = {
"model": model,
"messages": [{"role": "user", "content": prompt}],
}
if config.enabled:
params["thinking"] = {"type": "enabled", "budget_tokens": config.budget_tokens}
params["max_tokens"] = config.budget_tokens + 4000
else:
params["max_tokens"] = 4000
return paramsBy assigning a TaskType to each step in your pipeline, thinking budget is automatically determined without touching prompts.
Real Cost Comparison Over 3 Months
Extended Thinking always ON (hypothetical): ~$340/month. Policy-based operation (actual): ~$97/month. Cost reduction: 71%.
No measurable quality regression. If anything, quality improved slightly — by concentrating Extended Thinking budget on tasks where it matters, the overall system performed better than spreading it uniformly.
Model Selection Matrix
My current production rules for model × thinking combinations:
Batch/format tasks use Haiku 4.5 with thinking OFF. Article generation and summarization use Sonnet 4.6 with thinking OFF. Code reviews use Sonnet 4.6 with thinking ON at 12,000 tokens. Architecture consultation uses Opus 4.6 with thinking ON.
Design consultations happen a few times a month, so the Opus + thinking cost is acceptable. Code reviews happen frequently, so Sonnet + thinking is the right balance.
Qualitative Observations
A few things I noticed that aren't captured in numbers:
Extended Thinking clarifies reasoning. Outputs don't just improve in quality — they show why the answer was reached. Technical writing naturally develops a structure of "establish premise, consider counterexamples, reach conclusion."
More budget isn't always better. For simpler tasks, 8,000 tokens sometimes produced cleaner output than 16,000 tokens. Overthinking is a real phenomenon. Match the budget to the task's actual complexity.
Pair with low temperature. When using Extended Thinking, I set temperature to 0.1–0.3. This noticeably improves reasoning consistency.
Takeaway: Extended Thinking Is a Precision Tool
Extended Thinking isn't a general quality booster. For formatting and simple generation, the cost-benefit ratio is poor. For code reviews, design analysis, and complex comparisons, it genuinely earns its cost.
The key insight is that the question isn't "use it or not" — it's "which tasks get which budget." Implement a policy, measure against your actual workload, and let the data guide your configuration. In my case, that meant 71% lower costs with no quality tradeoff.
Start by sampling 10–20 tasks from your pipeline and measuring both conditions. The numbers will tell you where to invest.