Claude Code + Ollama: Cutting API Costs with Local LLMs

If you've been running Claude Code seriously for a while, you've probably had at least one moment of surprise when checking your monthly bill. I ran into this myself while managing automated workflows across multiple sites — the costs crept up faster than I expected.

The solution I landed on was a hybrid approach: routing repetitive, straightforward tasks to a local LLM running via Ollama, while reserving Claude Sonnet and Opus for work that genuinely needs their reasoning depth. This split cut my API costs by roughly 50–60% without meaningfully impacting the quality of the output that matters.

Here's how to set it up, and more importantly, how to decide what goes where.

Why Local LLM + Claude Code?

Claude Code's per-token pricing is substantial. Sonnet runs at $3 per million input tokens; Opus at $15. If you're using Claude Code heavily — writing comments, renaming things, generating boilerplate — those small tasks accumulate quickly.

Ollama is a local LLM runtime that runs models directly on your machine. Once you pull a model, inference is free. M-series Macs handle mid-sized models surprisingly well with Apple Silicon; a 14B parameter model runs at a usable speed without a discrete GPU.

The key insight: don't try to replace Claude — augment it. Local models like Qwen2.5-Coder or Gemma 3 fall short of Claude Sonnet on complex multi-step reasoning and long context tasks. But for the category of tasks where "good enough, quickly" beats "perfect, slowly," they're perfectly capable.

Setup: Ollama + litellm Proxy

Claude Code respects the ANTHROPIC_BASE_URL environment variable. Point that at a litellm proxy, and litellm will translate Claude Code's Anthropic-format requests into whatever your local model needs.

1. Install Ollama

# macOS via curl
curl -fsSL https://ollama.com/install.sh | sh
 
# Or via Homebrew
brew install ollama
 
# Start the Ollama service
ollama serve &
 
# Pull a coding-focused model
ollama pull qwen2.5-coder:32b
# Short on GPU memory? Use a smaller variant:
# ollama pull qwen2.5-coder:14b

Verify it works: ollama run qwen2.5-coder:32b and ask it something simple.

2. Install litellm

litellm acts as a proxy that accepts Anthropic-format requests from Claude Code and converts them to the format your local model expects.

pip install 'litellm[proxy]'

3. Create the litellm Config

Save this to ~/.claude-local/litellm_config.yaml:

# Route Claude Code requests → litellm → Ollama
model_list:
  # When Claude Code requests claude-sonnet-4-6, serve qwen2.5-coder:32b locally
  - model_name: claude-sonnet-4-6
    litellm_params:
      model: ollama_chat/qwen2.5-coder:32b
      api_base: http://localhost:11434
 
  # Optional: lighter model for haiku-level tasks
  - model_name: claude-haiku-4-5-20251001
    litellm_params:
      model: ollama_chat/qwen2.5-coder:14b
      api_base: http://localhost:11434
 
litellm_settings:
  port: 4000
  set_verbose: false

Heads up: Ollama doesn't fully implement the Anthropic Messages API spec, so litellm handles compatibility shims for features like tool_use. Most basic operations work fine, but test complex tool interactions before relying on them in production workflows.

4. Start the Proxy

litellm --config ~/.claude-local/litellm_config.yaml
 
# Or run it in the background
nohup litellm --config ~/.claude-local/litellm_config.yaml > /tmp/litellm.log 2>&1 &

The proxy listens on http://localhost:4000.

Connecting Claude Code to the Proxy

Just set two environment variables:

# Add to your .zshrc or .bashrc
export ANTHROPIC_BASE_URL=http://localhost:4000
export ANTHROPIC_API_KEY=local-dev-key  # litellm requires a key field, but doesn't validate it

For quick switching between modes, I use shell aliases:

# Switch to cloud Claude (real API)
alias cc-cloud='unset ANTHROPIC_BASE_URL && export ANTHROPIC_API_KEY=your_real_api_key'
 
# Switch to local LLM mode
alias cc-local='export ANTHROPIC_BASE_URL=http://localhost:4000 && export ANTHROPIC_API_KEY=local-dev-key'

Add a visual indicator to your prompt so you always know which mode you're in:

# In .zshrc — shows [LOCAL] in yellow when ANTHROPIC_BASE_URL is set
RPROMPT='%F{yellow}${ANTHROPIC_BASE_URL:+[LOCAL]}%f'

Which Local Model Should You Use?

Not all local models are equally suited for coding tasks. Here's a quick breakdown of what I've tested:

Qwen2.5-Coder (32B or 14B) — My current go-to for coding work. Alibaba's Qwen Coder series is specifically trained on code and handles most Claude Code-compatible tasks well. The 32B version produces noticeably better output than the 14B, though it requires more memory (roughly 20GB VRAM for 4-bit quantized on a GPU, or 32GB+ unified memory on Apple Silicon).

Gemma 3 (27B) — Google's Gemma 3 has strong multilingual capability and solid code understanding. If you're working in Japanese or need good prose alongside code, Gemma 3 is worth trying alongside Qwen.

Llama 3.3 (70B) — If you have the hardware, Meta's Llama 3.3 at 70B is the closest to Claude Sonnet you'll get locally. It's overkill for comment generation but handles light refactoring tasks impressively well.

For most developers, I'd suggest starting with qwen2.5-coder:14b — it downloads in a reasonable time, runs on modest hardware, and produces output good enough for the "easy win" task categories. If you're on an M3 Max or better, upgrade to the 32B once you've confirmed the workflow fits your needs.

The Task Routing Framework

Not everything belongs on a local model. Here's the split I use:

Send to local LLM (cost savings zone):

• Renaming variables, functions, files
• Adding or updating JSDoc comments
• Generating boilerplate (basic CRUD, config scaffolding)
• Adding logging statements
• Sorting and organizing imports
• Fixing straightforward bugs with clear stack traces
• Code formatting corrections

Keep on Claude Sonnet/Opus (quality-critical zone):

• Architecture discussions and design decisions
• Complex refactors spanning multiple files
• Security-sensitive implementation
• Debugging that requires understanding long, interleaved context
• Turning vague requirements into concrete specs
• Novel feature design through to implementation

My rule of thumb: "If the AI makes a mistake here, will I catch it immediately?" If yes, local is fine. If a wrong answer could propagate silently through your codebase, use Claude.

Common Pitfalls

Pitfall 1: Ollama isn't running when you start Claude Code

Write a startup script so both services come up together:

#!/bin/bash
# ~/bin/start-local-claude.sh
 
if ! pgrep -x "ollama" > /dev/null; then
  echo "Starting Ollama..."
  ollama serve &
  sleep 3
fi
 
if ! curl -s http://localhost:4000/health > /dev/null 2>&1; then
  echo "Starting litellm proxy..."
  nohup litellm --config ~/.claude-local/litellm_config.yaml > /tmp/litellm.log 2>&1 &
  sleep 2
fi
 
echo "✅ Local Claude Code mode ready"
cc-local

Make it executable: chmod +x ~/bin/start-local-claude.sh

Pitfall 2: tool_use failures on complex operations

Claude Code uses tool_use internally for file edits, searches, and shell commands. Local models sometimes stumble on the tool-calling protocol, causing silent failures or unexpected behavior mid-task. If a local session gets stuck or produces garbled results, switch to cc-cloud and don't fight it.

Pitfall 3: Forgetting which mode you're in

The RPROMPT visual indicator helps, but adding a quick echo $ANTHROPIC_BASE_URL to your workflow before starting any production work is a good safety habit.

Real-World Cost Impact

My usage: Claude Code running 4–6 hours daily across automation and development work. Before the hybrid setup, I was spending $40–60/month. After routing repetitive tasks locally, that dropped to $20–30. The biggest wins came from comment generation and iterative file editing — tasks where I'd otherwise be burning tokens on work any capable model can handle.

For tracking and visualizing your Claude Code spend, Claude Code Cost Visualization & Budget Management Guide walks through the tooling. If you're trying to optimize which Claude model tier to use for cloud tasks, Claude Code Model Selection and Opus+Plan Mode Strategy covers that decision framework. For more advanced multi-provider litellm setups beyond the local LLM use case, see Claude API × litellm Multi-Provider AI Gateway Guide.

Next Step: Route One Task Type Locally

The setup takes maybe 20 minutes once Ollama is downloaded. The bigger shift is building the habit of asking "does this task actually need Claude Sonnet?" before reaching for the cloud.

Start small: pull qwen2.5-coder:14b (lighter and faster), configure the proxy, and route just comment-writing to it for a week. See if the output quality holds up for that use case. Then expand from there.

Full replacement of Claude Code isn't the goal — and wouldn't be wise given the quality gap on complex tasks. The hybrid model is about spending Claude's premium capability where it earns its keep.