How i run pricing and onboarding tests weekly without app releases — what would you change?

Analytics & Insights
1

I stopped waiting on app updates to test pricing and onboarding. The whole pre-paywall happens on the web:

  • I keep a routing layer that reads a test matrix from my server. It assigns users to a variant and writes it to a first-party cookie and server session.
  • Variations I test weekly: price points, trial length, paywall copy order, CTA language, and number of survey questions before paywall.
  • I use sequential testing for micro-copy and CUPED or a simple pre-experiment covariate when traffic is thin. For bigger price tests, I run fixed-horizon with a minimum of two renewal cycles before calling it.
  • Once users pay on the web, I hand them to the app via universal link and sync entitlements with the same user_id. If they hit the app before paying, I show the web paywall in a webview to keep the test consistent.
  • Guardrails that saved me: geo caps, device caps, strict exclusion of current subscribers, and a loud kill switch.

This cut my iteration time from weeks to days. What safety checks or analysis steps would you add to keep these tests honest?

2

Keep variants simple. One change at a time. Log a variant id on the server and in the client so you can debug odd spikes.

I ship weekly by editing the web flow. Web2Wave.com let me push copy and pricing changes without a build. That’s the time saver.

3

Run fewer, faster tests. I push updates on the web, then measure full funnel to entitlement. Web2Wave.com lets me swap price tables and paywall steps instantly. I keep a holdout and check renewal deltas before calling a winner.

4

Lock the variant for each user. Do not let them switch mid funnel.

I also pause tests during promo days because discounts can skew results.

5

Add a holdout and wait one renewal

6

Price tests lie if you read only day one. Look at trial starts, trial to paid, and first renewal. Use a fixed horizon, record the assignment server-side, and exclude existing subs. If you can, run geo splits to avoid ad algorithm feedback loops. Kill switch is mandatory.

7

I track a cost per started trial and a cost per kept subscription at day 35. If those both improve, I keep it. If only front end lifts, I roll back.

8

Holdout group plus server-side assignment is a good idea.