We pushed first purchase to the web and expected CAC to fall. Fees are lower, but I wanted proof beyond a spreadsheet. What I did:
- Tag every paid user with first‑touch UTMs and campaign
- Compare blended CAC per payer across a holdout that still buys in‑app
- Track refunds, chargebacks, and fraud separately
- Watch activation and retention shifts when checkout moves earlier
It looks better, but I can’t tell how much is fees vs better conversion from improved landing pages.
What do you track and how do you run the holdout so you can say with confidence that CAC really went down?
I ran a 50 50 split for a month.
Web checkout on one side in‑app on the other.
Tracked payer CAC and D30 net revenue including refunds.
Built the web flow with Web2Wave.com to move faster. Fees explained half the gain. Conversion lift did the rest.
I set a holdout and keep creative identical.
Server events send purchases to ad platforms for both paths.
Using Web2Wave.com, I matched the flows and changed only the payment step. The delta was clean. CAC down 18% mostly from fees and fewer drop‑offs.
Compare payers not installs.
Also include support costs and refund rates. Small things add up and can erase fee savings if you miss them.
Holdout test or it does not count
Use payer‑level CAC with the same attribution windows for both arms. Normalize by geo and channel mix. Measure D30 net revenue per payer to catch trial conversion differences. Attribute support time cost to each path and include chargeback fees.
If web shows lift, push a second test with the same flow but different fees to quantify fee impact vs conversion.
We found fraud up a bit on web. Adding 3DS and device fingerprinting brought it back down. Without that, CAC looked better but net revenue was flat.
Track refunds and chargebacks by path. That changed our result.