Claude Code + Ollama: Cutting API Costs with Local LLMs

If you've been running Claude Code seriously for a while, you've probably had at least one moment of surprise when checking your monthly bill. I ran into this myself as an indie developer maintaining automated workflows across several sites single-handedly — the costs crept up faster than I expected.

The solution I landed on was a hybrid approach: routing repetitive, straightforward tasks to a local LLM running via Ollama, while reserving Claude Sonnet and Opus for work that genuinely needs their reasoning depth. This split cut my API costs by roughly 50–60% without meaningfully impacting the quality of the output that matters.

Here's how to set it up, and more importantly, how to decide what goes where.

Why Local LLM + Claude Code?

Claude Code's per-token pricing is substantial. Sonnet runs at $3 per million input tokens; Opus at $15. If you're using Claude Code heavily — writing comments, renaming things, generating boilerplate — those small tasks accumulate quickly.

Ollama is a local LLM runtime that runs models directly on your machine. Once you pull a model, inference is free. M-series Macs handle mid-sized models surprisingly well with Apple Silicon; a 14B parameter model runs at a usable speed without a discrete GPU.

The key insight: don't try to replace Claude — augment it. Local models like Qwen2.5-Coder or Gemma 3 fall short of Claude Sonnet on complex multi-step reasoning and long context tasks. But for the category of tasks where "good enough, quickly" beats "perfect, slowly," they're perfectly capable.

Setup: Ollama + litellm Proxy

Claude Code respects the ANTHROPIC_BASE_URL environment variable. Point that at a litellm proxy, and litellm will accept Claude Code's Anthropic Messages API requests, translate them into whatever your local model expects, and translate the response back on the way out.

1. Install Ollama

# macOS via curl
curl -fsSL https://ollama.com/install.sh | sh
 
# Or via Homebrew
brew install ollama
 
# Start the Ollama service
ollama serve &
 
# Pull a coding-focused model
ollama pull qwen2.5-coder:32b
# Short on GPU memory? Use a smaller variant:
# ollama pull qwen2.5-coder:14b

Verify it works: ollama run qwen2.5-coder:32b and ask it something simple.

2. Install litellm

# Recommended — keeps the proxy isolated from your project environments
uv tool install 'litellm[proxy]'
 
# pip works too
pip install 'litellm[proxy]'

3. Create the litellm Config

Save this to ~/.claude-local/litellm_config.yaml.

One detour worth skipping: my first config hijacked Claude Code's own model IDs — I wrote claude-sonnet-4-5 as the model_name and pointed it at Ollama. That works on the day you write it. It quietly stops working the day Claude Code changes its default model, and because nothing in your config changed, that failure is genuinely hard to trace.

Give your routes names you chose yourself, then ask for them explicitly at launch:

# Route Claude Code requests → litellm → Ollama
model_list:
  # Primary: local work that needs real code comprehension
  - model_name: local-coder
    litellm_params:
      model: ollama_chat/qwen2.5-coder:32b
      api_base: http://localhost:11434
 
  # Light: renames, comments, reflexive edits
  - model_name: local-fast
    litellm_params:
      model: ollama_chat/qwen2.5-coder:14b
      api_base: http://localhost:11434
 
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

The master_key isn't there to protect a public endpoint — it's there so that "only I can reach this proxy" is enforced by the software rather than by assumption. Note that the port is a launch flag, not a config key; the proxy defaults to 4000.

Heads up: Ollama doesn't fully implement the Anthropic Messages API spec, so litellm handles compatibility shims for features like tool_use. Most basic operations work fine, but test complex tool interactions before relying on them in production workflows.

4. Start the Proxy and Confirm It Answers

export LITELLM_MASTER_KEY="sk-local-$(openssl rand -hex 8)"
litellm --config ~/.claude-local/litellm_config.yaml
 
# Or run it in the background
nohup litellm --config ~/.claude-local/litellm_config.yaml > /tmp/litellm.log 2>&1 &

The proxy listens on http://0.0.0.0:4000. Before you connect Claude Code, hit the Anthropic-format endpoint directly. This one step separates "Ollama is down" from "litellm is misconfigured" from "Claude Code isn't picking up my environment variables":

curl -X POST http://0.0.0.0:4000/v1/messages \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-coder",
    "max_tokens": 128,
    "messages": [{"role": "user", "content": "What is 2 + 2?"}]
  }'

If that returns a completion, every layer below Claude Code is confirmed working.

Connecting Claude Code to the Proxy

Set two environment variables, then name the model at launch. Put the credential in ANTHROPIC_AUTH_TOKEN rather than ANTHROPIC_API_KEY — the latter is easy to confuse with your real Anthropic key, which is exactly the mix-up that leads to accidentally billing a "local" session.

export ANTHROPIC_BASE_URL="http://0.0.0.0:4000"
export ANTHROPIC_AUTH_TOKEN="$LITELLM_MASTER_KEY"
 
# Use the names you defined in the config
claude --model local-coder
 
# Or the lighter route
claude --model local-fast

For quick switching between modes, I use shell aliases:

# Switch to cloud Claude (real API)
alias cc-cloud='unset ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN'
 
# Switch to local LLM mode
alias cc-local='export ANTHROPIC_BASE_URL=http://0.0.0.0:4000 && export ANTHROPIC_AUTH_TOKEN=$LITELLM_MASTER_KEY'

Add a visual indicator to your prompt so you always know which mode you're in:

# In .zshrc — shows [LOCAL] in yellow when ANTHROPIC_BASE_URL is set
RPROMPT='%F{yellow}${ANTHROPIC_BASE_URL:+[LOCAL]}%f'

Switching Mid-Session with /model

The weakness of the alias approach is that every switch means quitting Claude Code and starting over — losing the conversation context each time. That friction was what kept me from actually using the hybrid setup for the first few weeks.

Claude Code v2.1.129 and later can pull the model list from your gateway, which means you can switch without restarting:

export CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1
claude

With that set, Claude Code calls GET /v1/models against your ANTHROPIC_BASE_URL on startup and adds each result to the /model picker, labeled From gateway. The local-coder and local-fast entries from your config show up there directly.

The workflow this unlocks is the one I actually wanted: start on local-fast for the mechanical groundwork, and when a real design decision surfaces, /model up to a cloud tier without losing the thread. Cheap model for the rough pass, expensive model for the judgment calls — as a single continuous session.

Discovery is opt-in. Without that environment variable, Claude Code never queries your proxy's /v1/models, and the picker stays limited to its built-in list.

Which Local Model Should You Use?

Not all local models are equally suited for coding tasks. Here's a quick breakdown of what I've tested:

Qwen2.5-Coder (32B or 14B) — My current go-to for coding work. Alibaba's Qwen Coder series is specifically trained on code and handles most Claude Code-compatible tasks well. The 32B version produces noticeably better output than the 14B, though it requires more memory (roughly 20GB VRAM for 4-bit quantized on a GPU, or 32GB+ unified memory on Apple Silicon).

Gemma 3 (27B) — Google's Gemma 3 has strong multilingual capability and solid code understanding. If you're working in Japanese or need good prose alongside code, Gemma 3 is worth trying alongside Qwen.

Llama 3.3 (70B) — If you have the hardware, Meta's Llama 3.3 at 70B is the closest to Claude Sonnet you'll get locally. It's overkill for comment generation but handles light refactoring tasks impressively well.

For most developers, I'd suggest starting with qwen2.5-coder:14b — it downloads in a reasonable time, runs on modest hardware, and produces output good enough for the "easy win" task categories. If you're on an M3 Max or better, upgrade to the 32B once you've confirmed the workflow fits your needs.

The Task Routing Framework

Not everything belongs on a local model. Here's the split I use, with the reasoning behind each call:

Laying it out as a table clarified something I'd been doing without naming: the line isn't drawn by difficulty. My actual rule of thumb is one question — "If the AI makes a mistake here, will I catch it immediately?" If yes, local is fine. If a wrong answer could propagate silently, use Claude. Verifiability, not complexity.

Common Pitfalls

Pitfall 1: Ollama isn't running when you start Claude Code

Write a startup script so both services come up together:

#!/bin/bash
# ~/bin/start-local-claude.sh
 
if ! pgrep -x "ollama" > /dev/null; then
  echo "Starting Ollama..."
  ollama serve &
  sleep 3
fi
 
if ! curl -s http://0.0.0.0:4000/health/liveliness > /dev/null 2>&1; then
  echo "Starting litellm proxy..."
  nohup litellm --config ~/.claude-local/litellm_config.yaml > /tmp/litellm.log 2>&1 &
  sleep 2
fi
 
echo "Local Claude Code mode ready"
cc-local

Make it executable: chmod +x ~/bin/start-local-claude.sh

Pitfall 2: tool_use failures on complex operations

Claude Code uses tool_use internally for file edits, searches, and shell commands. Local models sometimes stumble on the tool-calling protocol, causing silent failures or unexpected behavior mid-task. If a local session gets stuck or produces garbled results, switch to cc-cloud and don't fight it.

Pitfall 3: Forgetting which mode you're in

The RPROMPT visual indicator helps, but adding a quick echo $ANTHROPIC_BASE_URL to your workflow before starting any production work is a good safety habit.

Real-World Cost Impact

My usage: Claude Code running 4–6 hours daily across automation and development work. Before the hybrid setup, I was spending $40–60/month. After routing repetitive tasks locally, that dropped to $20–30. The biggest wins came from comment generation and iterative file editing — tasks where I'd otherwise be burning tokens on work any capable model can handle.

To be honest about it, that difference isn't free money. Routing work locally moved some verification burden onto me. What keeps me on this setup isn't the dollar figure — it's that I now ask "does this actually deserve the expensive model?" before every task, which turned out to be the more valuable habit.

If you're deciding which Claude tier to use for the cloud half of the split, Claude Code Model Selection and Opus+Plan Mode Strategy covers that framework. For multi-provider litellm setups beyond the local LLM case, see Building a Multi-Provider AI Gateway with Claude API and litellm.

Three Calls I'm Glad I Made

After running this setup for about six months, the things that mattered weren't config details. They were these three, and none of them were obvious at the start:

1. Naming my own routes instead of hijacking model IDs

Mapping Ollama onto a real model ID like claude-sonnet-4-5 feels great the moment you write it. It breaks when Claude Code updates its default model — and because your config didn't change, the debugging path is long. Names from your own vocabulary (local-coder) decouple your setup from Claude Code's release cycle. A configuration that keeps working isn't the correct one; it's the one that fails legibly.

2. Verifying with curl before ever launching Claude Code

Early on I debugged from inside Claude Code, guessing whether Ollama, litellm, or my shell environment was at fault. Adding one curl against /v1/messages cut that triage time to roughly a third. In any layered setup, confirm the lower layer is alive before you interrogate the upper one — a lesson I've had to relearn in every automation pipeline I've built.

3. Routing by verifiability rather than difficulty

I initially split tasks by how hard they looked. That failed for a specific reason: the tasks that look easy are the ones I stop checking. A local model writing a subtly wrong comment sailed straight past me. Switching the criterion to "will I notice if this is wrong?" fixed the routing. The only work safe to hand to the cheaper model is work whose failures come back to you.

Next Step: Route One Task Type Locally

The setup takes maybe 20 minutes once Ollama is downloaded. The bigger shift is building the habit of asking "does this task actually need Claude Sonnet?" before reaching for the cloud.

Start small: pull qwen2.5-coder:14b, register it as local-fast in your config, and route just comment-writing to it for a week. See if the output quality holds up for that use case. Then expand from there.

Full replacement of Claude Code isn't the goal — and wouldn't be wise given the quality gap on complex tasks. The hybrid model is about spending Claude's premium capability where it earns its keep.