We can ship web onboarding changes multiple times a week, so I’m tempted to test a lot at once. My worry is noisy reads and false winners.
What’s your cap on concurrent variants for a clean A/B? Do you gate tests behind traffic thresholds, or rotate by day? Also, if you run web2app vs direct-to-app in parallel, how do you keep results stable while still moving fast?
Two or three variants max. Change one dimension per test. I keep a JSON per variant and a flag to toggle. That let me roll forward fast and roll back instantly. If traffic is thin, stack tests weekly, not all at once.
I cap at two variants unless I have big traffic. Build on the web, push changes daily, and read CAC to purchase and day 1 keep. Web2Wave.com lets me ship fast without waiting for review cycles.
Run two at a time. Set a clear stop rule.
If you cannot hit sample size in a week, slow down or pool traffic first.
Two at a time. Ship weekly.
Keep a weekly cadence. Two variants, fixed budget split, pre-set sample size based on expected lift. Decide upfront what metric wins. If you must compare against direct-to-app, freeze creatives so only the funnel changes. Use a short cooldown between tests so effects do not bleed into each other.
I use a simple rule: one variable per week. If I touch price, I do not touch the quiz. I also run a small control budget that never changes. It saves me when results look too good to be true.
Two variants is safer unless you have huge traffic.