It was a week when I had Cowork tidying up app store release notes, taking my Japanese text into English, French and Spanish. Reading the results that evening, I noticed the same feature name translated three different ways across paragraphs. That hadn't happened in the morning.
My first thought was that Cowork had gotten worse. Then I looked at what I'd actually done: I'd stayed in one conversation since breakfast, and I'd swapped the glossary file halfway through. Nothing I was comparing had been held constant.
What I'd like to pass on first is this: a feeling alone can't tell you whether the change is in the model or in how you're using it. As an indie developer I can't stop work for a formal evaluation, so I put together a recording routine small enough to keep going. A spreadsheet and about twenty lines of Python are enough.
Some of the reasons it "feels worse" live outside the model
In November, someone opened an issue on the claude-code repository (#109277) saying results had slipped over a few weeks on the same plan and the same settings. The reporter also noted that every correction adds more turns, and because each message re-reads the whole conversation, usage climbs with it.
That observation is useful for narrowing down causes. When results seem to change, I check three things before blaming the model:
- Conversation length. In a long chat, early instructions and files get buried. How fast you hit limits also depends on length (the day my limit ran out before noon).
- Different inputs. A file was replaced mid-session, or the connected folder grew.
- Different wording. You retype the request each time, and the phrasing drifts.
Only after holding those three steady and still seeing a change do I think it's fair to say something changed.
Build a small test bench: five requests, pass conditions written first
Pick five requests you make over and over in real work. The important part is writing down what counts as a pass before you look at any result. Decide afterwards, and you'll grade generously on good days and harshly on bad ones.
Here's the shape I use. Each request gets two or three yes/no conditions.
| ID | Request (paste this exact text every time) | Pass conditions |
|---|---|---|
| R1 | Translate notes_ja.txt into English using the glossary in terms.csv | c1: glossary terms used everywhere / c2: same paragraph count as the source |
| R2 | Total sales.csv by month and write the table to summary.xlsx | c1: totals match the source / c2: original not overwritten |
| R3 | Sort the files in inbox into folders by type | c1: zero files left behind / c2: original names kept |
| R4 | Read meeting.docx and list only the decisions, in five lines or fewer | c1: five lines or fewer / c2: no decisions that aren't in the file |
| R5 | Resize the images in photos to 1200px wide and save them to out | c1: every image is 1200px / c2: originals still there |
Keep the request text in requests.txt and copy from it every time. Put the input files in a dedicated bench folder and don't touch it while the test runs. For checking the R2 totals, the steps in my spreadsheet cross-check script carry over as they are.
Run each request three times in fresh chats, one row per result
Once a week, run each of the five requests three times, each time in a new conversation. That's 15 runs a week, about 30 minutes once you're used to it. Running three times in the same chat doesn't work as a comparison, because the second and third attempts lean on the first answer.
After each run, add one row per condition to runs.csv: 1 if it was met, 0 if not.
date,request_id,run,condition,passed
2026-11-09,R1,1,c1,1
2026-11-09,R1,1,c2,1
2026-11-09,R1,2,c1,1
2026-11-16,R1,1,c1,0
2026-11-16,R1,1,c2,1When I'm unsure, I record 0 and write the reason in a separate notes column. Once "close enough" starts counting as 1, the record drifts back into impressions.
Let a short script do the counting
Counting the rows by eye gets old by week three. The script below reads runs.csv, lists "conditions met / total conditions" per week and per request, and flags only the requests whose rate fell by a third or more between the last two weeks. It uses only Python's standard library, so there's nothing to install.
import csv
import sys
from collections import defaultdict
from datetime import date
# Usage: python3 tally.py runs.csv
path = sys.argv[1] if len(sys.argv) > 1 else "runs.csv"
hits = defaultdict(int) # (week, request) -> conditions met
total = defaultdict(int) # (week, request) -> conditions checked
with open(path, newline="", encoding="utf-8") as f:
for row in csv.DictReader(f):
y, m, d = map(int, row["date"].split("-"))
week = "%d-W%02d" % date(y, m, d).isocalendar()[:2]
key = (week, row["request_id"])
total[key] += 1
hits[key] += 1 if row["passed"].strip() == "1" else 0
weeks = sorted({w for w, _ in total})
requests = sorted({r for _, r in total})
print("request " + " ".join(weeks))
for r in requests:
cells = []
for w in weeks:
k = (w, r)
cells.append("%7s" % ("%d/%d" % (hits[k], total[k]) if total[k] else "-"))
print("%-8s " % r + " ".join(cells))
# Compare the last two weeks; flag drops of one third or more
if len(weeks) >= 2:
prev, last = weeks[-2], weeks[-1]
print("\n%s -> %s" % (prev, last))
for r in requests:
a, b = (prev, r), (last, r)
if not total[a] or not total[b]:
continue
drop = hits[a] / total[a] - hits[b] / total[b]
if drop >= 1 / 3:
print(" check: %s %.0f%% -> %.0f%%" % (r, 100 * hits[a] / total[a], 100 * hits[b] / total[b]))Run it on the five sample rows and R1 shows 3/3 1/2, falling from 100% to 50%, so it gets flagged. A real week has six rows per request (three runs × two conditions), so a week where only one of the six fails is a drop of about 17% and stays quiet. I treat one miss in three runs as noise and only look closer when two or more slip. That's the line the one-third threshold draws.
I use ISO week numbers (2026-W47) so the order holds across month boundaries. You could hand the script to Cowork and ask it to tally runs.csv, but having the thing under test do its own grading weakens the point, so I run it in my own terminal. If the terminal is new to you, the piece on choosing where to start may help.
When the numbers drop, and when they don't
Since I started keeping this record, I've had several weeks where Cowork felt worse, yet fresh runs met nearly every condition. In those cases the cause was almost always the long conversation I'd been living in. The fix was to start a new chat at each natural break and hand over files like the glossary again at the start.
If the same request keeps failing in fresh chats for two weeks running, that's a change your habits can't easily explain. Then I send the record, the request text and the input files along with feedback through the rating buttons or support. "R1's c1 went from 3 of 3 runs last week to 1 of 3 this week" is far easier for someone to reproduce than "the quality got worse."
Let the feeling open the record; let the record make the call. I don't dismiss the feeling, but I try not to switch tools or rethink a subscription on the feeling alone.
This week, pick just the request you make most often, write two pass conditions, and run it three times in fresh chats. That fills the first row of the table.