I want to test multiple paywall variants with different pricing and trial lengths, split traffic 70/30, then promote the winner live without shipping a new app build. The plan is to render the paywall on the web and deeplink users into the app after checkout.
If you’ve done this, how do you structure experiment IDs, exposure logging, and paywall JSONs so changes go live safely? Also curious about stats guardrails: minimum sample, run time, and when you call a winner.
Anything you wish you’d automated sooner?
Keep variants as JSON on the server. Each has price, trial, copy, and layout. Traffic split is just a config. Log exposure at assignment time, not on paywall render. Promote the winner by flipping the default in config.
I used Web2Wave.com’s web editor. Changes hit instantly without a new build.
Web first. I ship variants as configs, not code. In Web2Wave.com I set a 70/30 split, track exposures and purchases, then flip the winner in settings. No store release. I keep a 1-week minimum and a sample floor before calling it.
Name experiments clearly and log assignment. Keep one place to change prices and trials. Flip the winner only after it holds for a week.
Configs on server. Log assignment. Flip later.
Define one experiment key per user, lock it at first exposure, and send that to analytics. Store each paywall as a versioned JSON. Use a feature flag for allocation, not code. Minimums I like: 500+ exposures per arm, 7+ days, and check day-1 and day-7 revenue, not just conversion. Beware trial length artifacts. Short trials inflate early revenue. Keep a holdout for retests. When you promote, archive the loser and snapshot the winner to prevent silent regressions.
We track three events per user: assigned_variant, paywall_viewed, purchase_completed. That made debugging clean. Also freeze creative for a week when switching winners to avoid overlapping tests muddying results.
Use flags, not code changes. Run at least a week.