"Which waterfall was stronger this week, A or B?" That question keeps showing up on Monday mornings as I open the AppLovin MAX dashboard. Daily floor-price tuning is one rhythm; A/B tests at the waterfall level are a slower one. They only really move week by week. Treating those two cadences as separate jobs is what pushed me to try this setup for a full month.
I have been shipping apps as a solo indie developer at Dolice since 2014, and even with the catalog now past 50 million cumulative downloads, the mediation layer is one of the few areas I want to keep close to my own hands. Daily routine work can go to Claude in Chrome. The weekly direction-setting is something I still want to write out in my own words.
What I was actually A/B testing
The four apps in scope were Beautiful HD Wallpapers, an Ukiyo-e wallpaper app, Relaxing Healing, and Law of Attraction Everyday. Using AppLovin MAX's waterfall A/B test feature, each app was running this comparison:
- A: the existing waterfall with hand-tuned floor prices
- B: in-app bidding placed at the top, the waterfall demoted to fallback
Traffic split was 50/50, audience was all users, and the ad units were rewarded video and interstitial. That meant 4 apps × 2 units = 8 concurrent A/B tests to keep an eye on.
What I handed off to Claude in Chrome
Every Monday morning, Claude in Chrome walked the AppLovin MAX reporting screens and pulled the following numbers into a single text table per test:
- Last 7 days eCPM for arm A and arm B
- Last 7 days impressions and fill rate
- ARPDAU (ad revenue divided by DAU) week over week
- A flag for tests where the difference looked statistically meaningful
- A country-level eCPM heatmap, top five countries only
None of these numbers are hidden—you can find them all in the AppLovin MAX dashboard. The trouble is doing the same eight times by hand. Claude in Chrome is good at "repeat the same procedure eight times," and that frees me to just read the resulting table.
Where it stumbled at first
The first two weeks had two recurring snags. The first was timing: when I pulled reports late Sunday night JST, the final day of arm B was clipped because of the JST/PST cutoff, and the numbers looked unfairly low. Moving the report to Tuesday morning fixed that.
The second was the model getting too eager about declaring winners. If I asked Claude in Chrome "which arm is winning?", it would still write a confident answer even when the sample size was clearly too thin. I switched to giving it a rule up front: "only call a winner when impressions are above 100k and the eCPM gap is at least 5%." Pre-defining the boundary kept the judgment range clean.
Three patterns the month surfaced
1. Rewarded video consistently favored arm B (bidding-first)
Across all four apps, rewarded-video requests in arm B averaged 8 to 14 percent higher eCPM. With Liftoff, Unity Ads, and Meta Audience Network bidding in real time, the bidding-first configuration outperformed the waterfall on price.
2. Interstitial split cleanly along app type
In the two wallpaper apps, arm B won. In the healing app and the law-of-attraction app, arm A won. The latter two have longer sessions and benefit from specific high-floor inventory in certain countries, which the hand-tuned waterfall captured better than bidding did.
3. Japan was repeatedly an outlier where arm A won
This was the most interesting find. In Japan, there are time windows where the bidding-side network supply is thin, dragging arm B's average fill rate down. Keeping arm A for Japan while letting other countries lean into arm B is now on the shortlist of next experiments.
What I deliberately did not delegate
Some pieces of the loop stayed in my hands:
- Deciding when to stop an A/B test or recut a new one was always mine. Test design changes ripple far, so I wanted to keep the muscle memory
- Month-end revenue reconciliation across App Store Connect, Google Play Console, AdMob, and AppLovin MAX is something I still do myself, screen by screen. Anything that touches the final money number stays outside the automation
- Adding new ad network partners, and the related W-8BEN paperwork, also remains a manual chore
The numbers after one month
Across the four apps, ad revenue rose 6.8 percent month over month, and ARPDAU rose 5.4 percent. Not dramatic, but stacked nicely on top of the 12 percent improvement that came from earlier floor-price tuning.
A side effect I appreciated: roughly two hours of manual Monday-morning aggregation evaporated. In its place I now have a 30-minute slot for "read the table Claude in Chrome assembled, decide what to change next week, and write it in a notebook." The time spent doing has been replaced by time spent thinking.
What I want to try next
The next experiment is asking Claude in Chrome to draft A/B test designs as well, not just summarize them. Right now I decide what to compare. If it can read the heatmap and propose patterns like "split Japan from the rest of the world," the option space gets wider.
A small personal note: when I was seventeen, an online mentor once told me that art should be a natural language open to everyone. Ad optimization may feel like the opposite end of that, but if you see it as quietly preparing the ground so the same app can reach users around the world fairly, the roots are not so far apart. I want to keep tending to these mundane mediation numbers as part of running the apps for the long haul.
Thanks for reading. I hope these notes are useful if you are running a similar AppLovin MAX setup.