Running user surveys for a while now, but I’m questioning if I’m only getting responses that align with my preconceptions.
Could it be that I’m unintentionally leading the conversation? What’s the best way to design surveys for genuine feedback?
Selection bias is a major issue. Most users who respond are your engaged ones. The quiet majority who left or barely used the app won’t give feedback. I stopped using surveys alone. Now I check who responds versus who doesn’t. Then, I ask the non-responders directly why they stopped using the app. Timing is also key. Asking right after conversion gives you different insights than surveying them weeks later.
I split test my survey questions now because people respond differently to how you word things.
Send version A to half your users and version B to the other half, then compare what you get back. If the answers are totally different, your questions are probably pushing people toward certain responses.
Also helps to survey people who just canceled or uninstalled instead of only active users.
Ask what they did not what they think about it.
Open ended questions help but people still give short answers. We add incentives to get longer responses.
Been down this rabbit hole myself. Your survey questions are probably doing exactly what you suspect.
I used to ask stuff like “How satisfied are you with our onboarding?” instead of “What happened when you first used the app?” The first one already assumes they think about satisfaction. The second one gets you actual behavior.
Also noticed people lie on surveys without meaning to. They tell you what sounds reasonable, not what they actually did. Started mixing in analytics data to cross-check survey responses - found tons of gaps.
One trick that helped: I’d show the survey to someone who knows nothing about the product first. If they can guess what answer I want, the question needs work.
This video breaks down how to structure questions that actually get honest responses:
Most important thing I learned - ask about specific moments instead of general feelings. “Tell me about the last time you used feature X” beats “How often do you use feature X?” every time.