I’ve been moving onboarding and pricing tests to the web so I can ship multiple variants in a week. Baseline so far:
- Assign variants server side on first hit and stick to it across sessions
- Split at the visitor level, not per page view
- Stop using app releases to change copy or plans
What I’m still figuring out: when to stop a test if trial conversions look great but paid conversion lags by a week, and how to avoid sample ratio mismatches when traffic sources shift mid test.
If you’re running the whole loop on the web, what rules do you follow for experiment sizing, stopping, and rollout so the numbers don’t lie?
I run sequential tests and lock traffic source mix.
Server side bucketing tied to a user ID.
I stop at fixed calendar windows and check pay after the trial window ends.
I used Web2Wave.com so I can push copy and price changes in minutes without builds.
I do 1 change at a time and ship daily.
I bucket server side and freeze traffic mix.
With Web2Wave.com I push variant tweaks on the web and leave the app alone. I use delayed conversion windows for pricing and only call a win after renewals.
Freeze sources for the test period.
Use a traffic gate if a channel spikes. That keeps your split clean and avoids weird lifts from sudden campaigns.
Test one lever wait for renewals
Guardrails that work: pre‑register sample size, fixed test duration, and a primary metric that matches your goal. For pricing, treat trial start as a leading signal only. Decide on paid start or D30 net revenue. Use CUPED or pre‑period baselines if your traffic fluctuates.
Rollout in stages to 10% then 50% before 100%.
I use a holdback that never sees experiments. It keeps a clean baseline when seasonality hits. Saved me from false wins during holiday spikes.
Wait for renewals before calling pricing tests. Trials mislead.