Automated Play Store Staged Rollout Monitoring with Claude Code — Lessons from 50+ Crashes in v2.0.0

Step 1: Fetching Android Vitals Data

Play Console's Android Vitals tracks three metrics we care about most:

• Crash-free Users Rate: percentage of users who experienced zero crashes
• ANR-free Users Rate: percentage of users who experienced zero Application Not Responding events
• Crash count: total crashes during the period

For teams using the Google Play Developer API, authentication goes through a service account linked to Play Console via the androidpublisher.v3 scope:

# google_play_vitals.py
# Fetch crash metrics via Google Play Developer API
 
from google.oauth2 import service_account
from googleapiclient.discovery import build
from datetime import datetime, timedelta
 
PACKAGE_NAME = "net.dolice.beautifulwallpapers"
SERVICE_ACCOUNT_FILE = "service_account.json"
SCOPES = ["https://www.googleapis.com/auth/androidpublisher"]
 
def get_crash_metrics(days_back: int = 7) -> dict:
    """Retrieve crash metrics for the specified period."""
    credentials = service_account.Credentials.from_service_account_file(
        SERVICE_ACCOUNT_FILE, scopes=SCOPES
    )
    service = build("androidpublisher", "v3", credentials=credentials)
 
    request = service.vitals().crashrate().get(packageName=PACKAGE_NAME)
    return request.execute()
 
def parse_crash_free_rate(vitals_data: dict) -> float:
    """Compute Crash-free Users Rate from the API response."""
    # The API returns crash_rate (fraction of users who crashed)
    # Crash-free rate = (1 - crash_rate) * 100
    crash_rate = vitals_data.get("crashRate", {}).get("rate", 0)
    return round((1 - crash_rate) * 100, 3)

For simpler setups, exporting a CSV directly from Play Console works just as well:

# csv_reader.py
# Read a Play Console Android Vitals CSV export
 
import csv
 
def read_vitals_csv(csv_path: str) -> list[dict]:
    """Parse a Play Console Vitals CSV into structured dicts."""
    rows = []
    with open(csv_path, encoding="utf-8-sig") as f:
        reader = csv.DictReader(f)
        for row in reader:
            rows.append({
                "date": row.get("Date", ""),
                "crash_free_rate": float(row.get("Crash-free users (%)", 0)),
                "anr_free_rate": float(row.get("ANR-free users (%)", 0)),
                "crash_count": int(row.get("Crashes", 0)),
                "version": row.get("App version", "")
            })
    return rows

Step 2: Go/No-Go Logic with Wilson Confidence Intervals

This is where the first significant challenge appeared. When I described the problem to Claude Code and asked it to build a threshold-based verdict system, the initial script worked — but had a critical flaw.

At 5% rollout, sample sizes are small. A single crash can drag the Crash-free Rate down dramatically when you have only a few hundred active users in the rollout cohort. Reacting to that as if it were a 50% rollout signal would generate constant false alarms.

I raised this issue with Claude Code, and it proposed using Wilson score confidence intervals — specifically the lower bound — as the basis for judgment rather than raw observed values. This means instead of asking "what is the crash-free rate right now?", we ask "what is the worst plausible crash-free rate, given the sample size?"

# go_no_go_analyzer.py
# Staged rollout Go/No-Go decision logic
 
import math
from dataclasses import dataclass
from enum import Enum
 
class Decision(Enum):
    PROCEED = "PROCEED"
    WATCH = "WATCH"
    PAUSE = "PAUSE"
    ROLLBACK = "ROLLBACK"
 
@dataclass
class RolloutMetrics:
    crash_free_rate: float    # e.g. 99.8 (percent)
    anr_free_rate: float      # e.g. 99.9 (percent)
    crash_count: int
    active_users: int
    rollout_percentage: int   # 5, 25, 50, or 100
    days_elapsed: int
 
def wilson_lower_bound(successes: int, total: int, z: float = 1.96) -> float:
    """
    Wilson score confidence interval lower bound.
    Treat crash-free users as 'successes'.
    z=1.96 corresponds to 95% confidence.
    """
    if total == 0:
        return 0.0
    p_hat = successes / total
    denom = 1 + z**2 / total
    center = p_hat + z**2 / (2 * total)
    margin = z * math.sqrt(p_hat * (1 - p_hat) / total + z**2 / (4 * total**2))
    return (center - margin) / denom * 100
 
def analyze_rollout(metrics: RolloutMetrics) -> tuple[Decision, str]:
    """
    Determine whether to proceed, watch, pause, or roll back.
    Returns: (decision, human-readable reason)
    """
    crash_free_users = int(metrics.active_users * metrics.crash_free_rate / 100)
    wilson_lower = wilson_lower_bound(crash_free_users, metrics.active_users)
 
    # Minimum observation period per rollout stage
    min_days = {5: 1, 25: 2, 50: 3, 100: 5}.get(metrics.rollout_percentage, 3)
 
    reasons = []
 
    # Absolute thresholds — these override everything
    if metrics.crash_free_rate < 99.5:
        return Decision.ROLLBACK, (
            f"Crash-free Users at {metrics.crash_free_rate:.3f}% — "
            f"immediate rollback recommended (threshold: < 99.5%)"
        )
 
    if metrics.anr_free_rate < 99.8:
        return Decision.ROLLBACK, (
            f"ANR-free Users at {metrics.anr_free_rate:.3f}% — "
            f"rollback recommended (threshold: < 99.8%)"
        )
 
    # Wilson lower bound check
    if wilson_lower < 99.5:
        reasons.append(
            f"Wilson lower bound {wilson_lower:.3f}% in caution zone "
            f"(sample: {metrics.active_users:,} users)"
        )
 
    # Insufficient observation time
    if metrics.days_elapsed < min_days:
        reasons.append(
            f"Only {metrics.days_elapsed}d elapsed (recommended: ≥{min_days}d at {metrics.rollout_percentage}%)"
        )
 
    # Warning zone
    if metrics.crash_free_rate < 99.7:
        reasons.append(
            f"Crash-free Users {metrics.crash_free_rate:.3f}% — warning zone (target: ≥99.7%)"
        )
 
    if reasons:
        return Decision.WATCH, "; ".join(reasons)
 
    return Decision.PROCEED, (
        f"All metrics within targets — "
        f"Crash-free: {metrics.crash_free_rate:.3f}%, ANR-free: {metrics.anr_free_rate:.3f}%"
    )

The key design decision here is the separation between absolute hard stops (crash-free < 99.5% → rollback immediately, regardless of sample size) and probabilistic caution (Wilson lower bound in the warning zone → watch and wait). This prevents the system from either ignoring genuine early signals or over-reacting to noise.

Step 3: Claude API for Natural-Language Reports

Numbers alone don't survive across release cycles. What helps in the next release is knowing why a decision was made and what actually happened — in plain language.

# report_generator.py
# Generate rollout summaries via Claude API
 
import anthropic
from datetime import datetime
 
client = anthropic.Anthropic()  # ANTHROPIC_API_KEY from environment
 
def generate_rollout_report(
    app_version: str,
    metrics_history: list[dict],
    decisions: list[tuple],
    final_outcome: str
) -> str:
    """Generate a structured post-rollout report."""
 
    history_text = "\n".join([
        f"- {d['rollout_pct']}% ({d['days_elapsed']}d): "
        f"Crash-free {d['crash_free_rate']:.3f}%, "
        f"ANR-free {d['anr_free_rate']:.3f}%, "
        f"crashes: {d['crash_count']} → {decisions[i][0].value}"
        for i, d in enumerate(metrics_history)
    ])
 
    prompt = f"""Here is the staged rollout monitoring log for {app_version}:
 
## Stage-by-Stage Data
{history_text}
 
## Final Outcome
{final_outcome}
 
Write a concise post-rollout report (English) covering:
1. Any notable metric movements during this rollout
2. Root cause of any WATCH or PAUSE decisions (if any)
3. One or two specific, actionable improvements for the next release
 
Skip preamble — start directly with the content."""
 
    message = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )
    return message.content[0].text

The prompt design matters more than it might seem. My initial version produced verbose reports that I never re-read. I asked Claude Code to "rewrite this prompt to produce a concise report focused on actionable next steps." The revised prompt — the one above — generates reports compact enough to actually reference before the next release.

Step 4: Production-Grade Integration with Claude Code

Once the individual components existed, I used Claude Code to integrate them into a long-running monitor — the kind of script that needs to survive API rate limits, token expiration, and transient Play Console data gaps over several weeks.

# monitor.py — Main monitoring loop with retry logic
 
import time
import logging
from dataclasses import dataclass
from typing import Optional
 
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[logging.FileHandler("monitor.log"), logging.StreamHandler()]
)
logger = logging.getLogger(__name__)
 
@dataclass
class MonitorConfig:
    package_name: str
    check_interval_hours: int = 6
    rollout_percentage: int = 5
    crash_free_warning: float = 99.7
    crash_free_stop: float = 99.5
    anr_free_stop: float = 99.8
    notification_webhook: Optional[str] = None
 
def run_monitor(config: MonitorConfig) -> None:
    """Main monitoring loop with exponential backoff on errors."""
    retry_count = 0
    max_retries = 3
 
    while True:
        try:
            logger.info(
                f"[{config.package_name}] Check at {config.rollout_percentage}% rollout"
            )
 
            metrics = fetch_current_metrics(config)
 
            if metrics is None:
                logger.warning("No metrics available — possible data lag. Waiting.")
                time.sleep(config.check_interval_hours * 3600)
                continue
 
            decision, reason = analyze_rollout(metrics)
            logger.info(f"Decision: {decision.value} — {reason}")
 
            if decision in (Decision.PAUSE, Decision.ROLLBACK):
                send_alert(config, decision, metrics, reason)
                if decision == Decision.ROLLBACK:
                    logger.critical("ROLLBACK recommended — manual intervention required")
                    break
 
            retry_count = 0
 
        except Exception as e:
            retry_count += 1
            wait = min(300 * (2 ** retry_count), 3600)  # cap at 1 hour
            logger.error(f"Error (attempt {retry_count}/{max_retries}): {e}")
 
            if retry_count >= max_retries:
                logger.critical("Max retries reached — sending alert and stopping")
                send_alert(config, Decision.PAUSE, None, f"Monitor error: {e}")
                break
 
            logger.info(f"Retrying in {wait}s")
            time.sleep(wait)
            continue
 
        time.sleep(config.check_interval_hours * 3600)

The exponential backoff — capped at one hour — was Claude Code's suggestion when I described the long-running nature of the monitor. My initial version had a flat 30-second retry, which would have hammered the API during an extended outage.

v2.1.0 Rollout: Actual Results

I ran this system for the Beautiful HD Wallpapers v2.1.0 release. The two main changes in this version were:

1. The RecyclerView defensive copy fix (root cause of the v2.0.0 crash surge)
2. coreLibraryDesugaringEnabled true in build.gradle — which resolved a NoClassDefFoundError on Android 6.0.1 devices caused by Glide 5.0.5's use of Java 8's Supplier interface under AGP 9.x

The rollout progression:

5% rollout (Day 1–2): Crash-free 99.94%, ANR-free 100%. Wilson lower bound 99.71% (small sample). Decision: WATCH (observation period < 2 days required). No action taken.

25% rollout (Day 3–5): Crash-free 99.89%, ANR-free 99.97%. Three crashes total. Wilson lower 99.82%. Decision: PROCEED.

50% rollout (Day 6–9): Crash-free 99.91%, ANR-free 99.98%. Seven crashes total. Decision: PROCEED. Claude API report at this stage automatically noted: "No crashes detected in the Android 6.0.1 segment — consistent with the desugaring fix in build.gradle."

100% rollout (Day 10+): 28-day Crash-free average: 99.88%. During the equivalent window for v2.0.0, the rate had been 99.61%.

The 50+ users who had crashed consistently on v2.0.0 were no longer reporting issues after the update completed.

Three Areas Where Claude Code Made the Difference

Wilson confidence intervals: I knew this approach was appropriate for small samples, but implementing it correctly in Python — including the edge case where total == 0 — would have taken time to verify. Describing the problem to Claude Code and asking for a "95% confidence lower bound for a proportion" produced correct code on the first attempt.

Exponential backoff design: A monitor that runs for weeks will encounter transient failures. My first version retried immediately with a fixed delay. Claude Code's suggestion to cap the backoff at one hour and alert after max retries is the pattern used in production distributed systems — not something I would have reached for instinctively for a personal project.

Prompt iteration for the report generator: Asking Claude Code to "rewrite this prompt so the output is concise and focused on next-release improvements" — using AI to improve AI prompts — cut the report length by half while making it significantly more useful.

Implementation Notes

Play Console data has lag. Android Vitals data arrives with a delay of several hours to 24 hours. The days_elapsed minimum per stage accounts for this: don't make decisions on Day 0 data at any rollout stage.

Understand what Crash-free Rate measures. Google defines it as the percentage of users who experienced zero crashes during the period. One user crashing ten times still counts as one affected user. Thresholds should be set with this definition in mind.

Calibrate thresholds to your app's baseline. The 99.7% warning threshold that works for Beautiful HD Wallpapers might not fit your app. Pull three or four previous stable release windows from your Play Console and use those as your baseline before you set any numbers.

Designing Your Rollout Percentages

The 5% → 25% → 50% → 100% progression I use isn't Google's default — their documentation suggests 10% → 20% → 50% → 100% — but I've settled on it through trial and error across a dozen releases.

For an app with tens of millions of lifetime downloads, even a 10% rollout reaches tens of thousands of users on day one. Keeping the first stage at 5% limits the blast radius while still collecting statistically meaningful data faster than a completely manual rollout would allow.

The minimum-days constraint per stage exists for two reasons. First, Play Console data arrives with a lag of several hours to a full day, so early readings can be misleading. Second, usage patterns are not uniform across the week. A wallpaper app tends to see higher engagement on weekends — users browsing and setting new wallpapers during leisure time. A rollout that only captures weekday behavior may miss crash patterns that appear primarily on weekends when heavier browsing sessions occur.

Waiting at least one full day at 5% and two days at 25% ensures that the data spans a representative slice of weekly usage patterns before a proceed decision is made.

For smaller apps — those with fewer than 50,000 DAU — a 5% first stage may produce too few samples for Wilson confidence intervals to be meaningful. In those cases, starting at 10–15% and relying more heavily on the absolute threshold checks is a reasonable adjustment.

What the v2.0.0 Failure Actually Looked Like

Looking back at the v2.0.0 incident with some distance, three compounding factors explain why a significant crash went undetected through a full staged rollout.

Baseline normalization. When an app has high daily active users, some level of crash activity is always present in Android Vitals. Over time, the noise becomes part of the background. I was comparing each day's crash graph against a mental model of "normal" rather than against a specific numerical baseline from the previous stable version. The crash rate was rising, but slowly enough that no single day looked alarming. A Δ comparison — "crash-free rate this version vs. last three versions at equivalent days post-release" — would have caught it.

Reproduction difficulty. The RecyclerView IndexOutOfBoundsException was a race condition tied to specific scroll timing. It didn't reproduce on any device I tested. In retrospect, asking Claude Code "what patterns of RecyclerView crash are typical for this kind of concurrent adapter modification?" before the release might have surfaced the defensive copy pattern as a precaution. Instead, the code shipped with a known theoretical risk that I had rationalized as unlikely to manifest in practice.

Overconfidence in staged rollout as a safety net. Staged rollout limits exposure. It doesn't automatically surface problems. Without a monitoring system that converts raw Vitals numbers into a verdict, you can reach 100% rollout and still be waiting for the data to tell you something has gone wrong. That's precisely what happened.

Setting Thresholds for Your App

The 99.7% warning and 99.5% stop thresholds I use are not universal values. They were derived from 12 years of historical data for Beautiful HD Wallpapers specifically.

Across 10 previous stable releases, the 28-day Crash-free Rate had a median of 99.84% and a standard deviation of roughly 0.07 percentage points. The 99.7% warning threshold sits approximately two standard deviations below the median — statistically, that's the point at which "normal variation" stops being a plausible explanation for what you're seeing. The 99.5% stop threshold sits below v2.0.0's worst measured value (99.61%), providing a hard floor that would have triggered an alert well before the crash count reached 50.

To calibrate thresholds for your own app, pull 28-day Crash-free Rate data for three to five previous stable releases from Play Console. Calculate the mean and standard deviation. Set your warning threshold at mean minus two standard deviations, and your stop threshold at mean minus three standard deviations. These aren't magic numbers, but they have a statistical basis that "99.7% feels right" does not.

How Claude Code Shaped the Design

Working iteratively with Claude Code on this system revealed some patterns worth noting for anyone building monitoring tools with AI assistance.

Problem framing determines answer quality. When I asked "write a staged rollout monitoring script with crash-rate thresholds," I got a script that worked in the happy path but would have generated false alarms on small samples. When I reframed the problem as "what goes wrong with simple threshold checks when sample sizes are small at 5% rollout?", the Wilson confidence interval solution emerged naturally from the conversation.

Claude Code responds well to problems stated as constraints and edge cases, not just as feature requests. "Make it work" yields a different result than "make it work correctly when the sample is too small to trust the raw rate."

Verify the math yourself. I checked the Wilson score formula against the reference implementation before using it in production. For code that drives release decisions, that verification step isn't optional. Claude Code's implementation was correct, but I wanted to understand why before deploying it — especially the edge case handling when total == 0. Trusting AI-generated code and understanding AI-generated code are separate habits.

Use AI to improve AI prompts. The report generation prompt went through three iterations. After the second draft produced reports that were technically accurate but too long to re-read before a release, I asked Claude Code directly: "Rewrite this prompt so the output is more concise and weighted toward next-release improvements." The resulting prompt is the one in Step 3 above. It produces reports roughly half the length of the original, focused on the two things I actually want to know: what was notable, and what to do differently next time.

Start by pulling three to five versions of historical Crash-free Rate data from your Play Console. Compute the mean and standard deviation. That baseline is your threshold calibration starting point. Once you have real numbers, the analyze_rollout() function is straightforward to configure and deploy.

Releasing an app alone — no QA team, no dedicated ops — doesn't have to mean accepting uncertainty as a constant companion. The right monitoring system turns the anxiety of a staged rollout into something closer to a measured process, which is where I would rather be after 12 years of shipping apps by myself.