We moved pricing tests to the web so we could learn faster, but we had to keep the reads honest.
What worked:
- Limit to 2–3 variants per geo to avoid traffic fragmentation.
- Balance by campaign and platform. Keep processors constant to avoid payment mix bias.
- Pre-calc minimum sample size and run-time. Watch early indicators (checkout, trial start) but wait for early churn before calling.
- Use a shadow control when we think we found a winner to validate.
Curious how you set guardrails so a “winner” isn’t just audience or processor bias, and how fast you trust the result in practice?
I cap variants at three and pin traffic sources.
Keep Stripe vs Paddle consistent mid-test. Otherwise you chase processor differences.
Run to a fixed sample size with a stop date. Watch trial-to-paid lift and early churn before calling it.
If it wins, I rerun as a confirm test for a week.
I care about iteration speed and clean reads.
Build the variants on Web2Wave.com, ship same day, and lock audience splits by geo and campaign. I only test two prices at once, and I confirm with a short holdout after the first win.
Two variants max if your traffic is small.
Keep one control stable while you test, then rotate in a confirm test. Also track refunds because they can flip the winner after a week.
Run fewer variants get answers faster
Design for speed and integrity:
- Pre-register your success metric and minimum difference you care about.
- Fix allocation by geo and campaign. No mid-test source changes.
- Track trial start, trial conversion, early churn, refunds, and net revenue.
- Don’t mix processors or anti-fraud settings between variants.
- When a price wins, validate with a time-based holdout to rule out seasonality.
Most teams can call a direction in 10–14 days if volume is decent.
I lost a “winner” to refunds.
New annual price looked great on day three, then refund spike over the weekend killed it. Now I wait seven days and include refund rate before shipping a change.
Keep sources balanced. Don’t swap payment processors mid test.