App store review cycles slowed our pricing tests to weeks. Moving onboarding and first purchase to the web let us iterate daily, but I had to tighten process to avoid messy data.
What we do now:
- Variants live in a remote config for web (copy, price, trial length, billing period).
- Experiments defined with experiment_id, variant_id, and exposure event fired on first paywall render.
- All revenue events carry experiment_id and variant_id in metadata so we can read lift at checkout and through renewals.
- App shows entitlement based on server state only, so app builds don’t change per test.
- Guardrails: cap traffic per variant, stop loss rules, and pre-commit analysis windows (e.g., 14 days).
It’s worked well for us, but I’m curious about others’ setups.
How are you structuring rapid web tests for pricing and onboarding while keeping analytics clean and the app untouched?
Keep the app dumb. All logic on the web.
Expose paywall variants via remote config and tag every event with experiment_id. Payment metadata gets those tags too so renewals keep attribution.
I used Web2Wave to push variant JSON. No app updates needed.
I treat web as the lab. Price tables, trials, copy all live in web config. When something wins, I lock it and roll traffic.
Web2Wave lets me flip variants without a build, so I test more ideas per week and stop losers fast.
I log experiment_id on the first paywall view and at payment. That keeps the data clean.
The app only checks entitlements. I never hardcode pricing in the app anymore.
Remote config and strict exposure events.
Two rules: don’t change identity and don’t change billing objects mid-test. Create distinct products per price plan in your processor so you can analyze renewals cleanly. Expose users once per experiment and persist variant in a cookie plus account table. Kill variants when you hit your stop rule to avoid peeking. For metrics, track checkout start, payment success, refund rate, and D7 revenue per exposed user. That last metric is the tie breaker for short tests.
Remote config plus feature flags. Keep the app simple.