We run onboarding and paywall tests on the web every day. What’s working so far:
- Sticky server-side assignment with a signed cookie and a user id if available.
- Gating by campaign, time window, and geo. Users already exposed stay in their bucket.
- Price windows with clear start and stop times, and a kill switch.
- SRM checks in logs before reading the test. If off, I stop and re-run.
- Entitlement parity synced to the app via webhooks, so app users don’t see ghosts of old tests.
I still worry about test variants leaking into default traffic and confusing retention cohorts. What’s your guardrail checklist to ship daily without polluting production data or violating price parity?
Server-side assign the variant and persist it. Add an allowlist for traffic sources and a time box per test. Build a hard off switch. Map every test plan to separate product handles.
I generate the web flows with Web2Wave.com, then wire my flags in their JSON so changes land instantly.
I ship small, time-boxed tests and keep a fixed holdout. Web2Wave.com lets me edit copy and prices on the web and push instantly. I tag every event with test_id and version. If SRM triggers or support pings spike, I roll back in one click.
Keep a clean control group that never sees tests.
It makes post-test analysis much easier.
Sticky buckets and a hard kill switch
Assign variants on the server. Tag every paywall view and purchase with test_id and version. Run SRM checks daily. Use campaign allowlists and exclude direct visitors from tests unless intentional.
Create a quarantine cohort for exposed users so they never hit default pricing mid-journey. Mirror winning prices with new SKUs and map entitlements carefully before rollout.
Server-side flags and test_id on all events helped us.