I Ran 1,000 App Store Reviews Through Claude API — Here's What My Data Was Hiding

When my wallpaper app crossed 3,000 reviews, I hit a wall I hadn't anticipated.

Every weekend, I'd open App Store Connect, read through reviews one by one, and paste them into Notion with my own categorization notes. It was a ritual I'd kept since 2014 — but as my app portfolio grew, it became unsustainable. Four apps, over 1,000 new reviews per month, half a day gone and I still couldn't get through all of them.

So I built a batch analysis pipeline with the Claude API. What surprised me wasn't the analysis accuracy — it was watching my own cognitive biases laid out in a spreadsheet.

The Real Problem With Reading Reviews Manually

When you read reviews by hand, your eyes go straight to the extremes. Five-star praise and one-star criticism both pull emotional weight. Three-star reviews sit in the middle and get skimmed.

After running a thousand reviews through Claude API — structured by sentiment, category, and improvement signal — I found that three-star reviews contained far more actionable improvement hints than any other rating tier. "Notification timing is slightly off." "You have to close and reopen the app after download for the wallpaper to apply." "The upgrade prompt is confusing."

Each one appeared once or twice in the raw data. Across 1,000 reviews, the same issues surfaced in roughly 1 of every 30 entries. That pattern was completely invisible to me when I was reading manually.

Hirokawa Masaki here — I've shipped apps since 2014 across wallpaper, wellness, and lifestyle categories. 50M+ cumulative downloads later, I still get surprised by what users are actually experiencing versus what I assume they're experiencing. Automated review analysis has become part of how I close that gap.

The Implementation

The pipeline has three steps:

1. Fetch reviews via iTunes RSS API or App Store Connect API
2. Send them to Claude API in batches of 20 for structured analysis
3. Aggregate results by priority score to surface the top issues

Here's the core Python code (API keys and app IDs come from environment variables):

import anthropic
import json
import os
import time
import requests
 
client = anthropic.Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))
 
def analyze_reviews_batch(reviews: list[dict]) -> list[dict]:
    """
    Takes a list of App Store reviews and returns structured
    sentiment/category/improvement data for each.
    """
    reviews_text = "\n---\n".join([
        f"Rating: {r['rating']} stars\nTitle: {r['title']}\nBody: {r['body']}"
        for r in reviews[:20]
    ])
 
    message = client.messages.create(
        model="claude-haiku-4-5-20251001",  # Haiku is cost-effective for classification
        max_tokens=2048,  # Always set this — without it, Claude adds prose around the JSON
        messages=[
            {
                "role": "user",
                "content": f"""Analyze the following App Store reviews and return a JSON array.
For each review include:
- sentiment: "positive" / "neutral" / "negative"
- category: "ui" / "bug" / "feature_request" / "performance" / "billing" / "other"
- improvement_hint: a specific improvement suggestion if present, otherwise null
- priority_score: 1–5 (5 = highest; concrete, reproducible problems score higher)
 
Reviews:
{reviews_text}
 
Return only the JSON array. No explanation or preamble."""
            }
        ]
    )
 
    try:
        return json.loads(message.content[0].text)
    except json.JSONDecodeError:
        return []
 
def process_all_reviews(reviews: list[dict]) -> list[dict]:
    """Process all reviews in batches of 20."""
    all_results = []
    batch_size = 20
 
    for i in range(0, len(reviews), batch_size):
        batch = reviews[i:i + batch_size]
        results = analyze_reviews_batch(batch)
        all_results.extend(results)
 
        if i + batch_size < len(reviews):
            time.sleep(1)  # Conservative rate limit buffer for Tier 1–2
 
        print(f"Processed: {min(i + batch_size, len(reviews))}/{len(reviews)}")
 
    return all_results
 
def fetch_reviews_rss(app_id: str, page: int = 1) -> list[dict]:
    """
    Fetch App Store reviews via iTunes RSS API.
    Works up to ~10 pages (500 reviews). For more, use App Store Connect API.
    """
    url = f"https://itunes.apple.com/us/rss/customerreviews/page={page}/id={app_id}/sortby=mostrecent/json"
    response = requests.get(url)
 
    if response.status_code != 200:
        return []
 
    data = response.json()
    entries = data.get("feed", {}).get("entry", [])
 
    # Index 0 is app metadata, not a review
    return [
        {
            "rating": int(e["im:rating"]["label"]),
            "title": e["title"]["label"],
            "body": e["content"]["label"]
        }
        for e in entries[1:]
        if "im:rating" in e
    ]

The two-model approach — Haiku for classification, Sonnet for summarization — keeps costs around $0.50–$1.50 for 1,000 reviews.

Summarizing High-Priority Issues

Once you have the structured data, Sonnet 4.6 can synthesize the top improvement areas:

def summarize_top_issues(analyzed_reviews: list[dict]) -> str:
    """
    Extract high-priority improvement hints and return a concise action list.
    """
    high_priority = [
        r["improvement_hint"]
        for r in analyzed_reviews
        if r.get("priority_score", 0) >= 4 and r.get("improvement_hint")
    ]
 
    if not high_priority:
        return "No high-priority improvement signals found."
 
    issues_text = "\n".join(f"- {h}" for h in high_priority)
 
    message = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=512,
        messages=[
            {
                "role": "user",
                "content": f"""The following improvement hints were extracted from App Store reviews.
Remove duplicates, merge similar issues, and return the top 5 as a prioritized bullet list.
Format for a developer sprint planning session.
 
Hints:
{issues_text}"""
            }
        ]
    )
 
    return message.content[0].text

What the Analysis Revealed About My Blind Spots

The highest-priority issue surfaced by the analysis was widget instability on the latest iOS version. I had been investing heavily in the widget feature and thought it was solid. When I scanned reviews manually, I assumed the scattered complaints were edge cases. The frequency data said otherwise — this category appeared more often than any other single issue.

This is the pattern I've seen repeat across multiple apps: the features I care about most are the ones I'm most likely to give the benefit of the doubt when reading critical feedback. Automated frequency analysis corrects for that bias in a way that's hard to replicate manually.

Pitfalls I Hit (So You Don't Have To)

JSON parse failures. Even with max_tokens=2048 set, Claude occasionally adds a line like "Here is the JSON:" before the array. Strip the response and slice from the first [ to the last ] before parsing.

Feature requests misclassified as negative. "I wish this app had a dark mode" reads as a complaint in terms of sentiment markers, but it's a feature_request, not a bug. Adding an explicit instruction in the prompt — "requests for new features should be categorized as feature_request, not negative" — improves this significantly.

Emoji-heavy reviews. Haiku handles Japanese and English review text extremely well. Reviews with a high density of emoji characters (especially mixed with short text) are the most error-prone for category classification.

The easiest way to start is to grab the first 100 reviews from your existing app using the RSS endpoint, run the analysis, and see what comes back. Even at that scale, you'll likely find at least one recurring issue you hadn't noticed.

Related: Claude API Python SDK Chatbot Tutorial