Here you are, running another Customer Validation (CV) survey. Your survey is packed full of questions that will yield meaningful product insights, like “What issues are you encountering?” and “What do you like most about the product?” You’ve scanned for signs of psychological bias. You’ve recruited targeted survey-takers, participation is high, and feedback is rolling in. You’re feeling pretty good.
Flash forward to a few days later when the survey ends. Now, you’re slogging through mountains of data and reading response after response. Delivering definitive results is demanding hours of follow-up with dozens of testers. You did everything to manage your survey efficiently, so why does analyzing the data take so much time?
In traveling across the country to coach dozens of teams on making their CV processes more effective, Centercode Product Manager Chris Rader sees a lot of surveys. Through it all, the challenge he spots most often isn’t poor survey management.
“It’s not that these teams are managing their surveys inefficiently,” says Chris. “It’s that it’s inefficient to use surveys as your primary mode of feedback.”
That’s right. The biggest reason for survey management time sinks is…using surveys in the first place.
How Surveys Cause Friction During CV
Chris elaborates: “A lot of the friction CV professionals encounter when they’re collecting tester feedback comes from using surveys as a feedback catch-all. Technically, you can use a survey to collect issues, ideas, and praise from testers — the same way you could save a phone number in a word processor. But when you actually go to call someone, you need to transfer that information from one tool to another.”
He explained that he sees the same thing happening with surveys. “You can ask open-ended questions like, ‘What issues did you encounter?’ but you’ll spend a lot of time reviewing feedback that your testers could have entered directly into your feedback management system,” says Chris. “It doesn’t matter how well-crafted your survey is at that point; feedback collection just won’t be as efficient as it could be.”
Surveys vs. Feedback Forms
Relying predominantly on surveys can also damage the accuracy of your data because your testers aren’t reporting issues in real-time. Say your testers respond to their weekly survey, then run into a bug two days later. They may not report that issue until the next survey — five days after the initial encounter.
“Surveys offer a snapshot of a single point in time,” says Chris. “If your testers are only reporting once a week, you risk losing the details, impressions, and sequence of events that shape what happened. It’s the details that come out in real-time reporting that give you an accurate portrait of the user experience — and make CV so valuable.”
Instead, Chris recommends training your testers to use feedback forms. Ongoing collection methods like these forms let you capture and respond to feedback as it flows in, so your team can get the jump on product fixes. Efficiency doubles if you’re using a feature like smart issue prioritization to surface your most impactful feedback. Either way, the immediacy of ongoing feedback channels turns tester follow-up into a dynamic conversation.
Using dedicated issues, ideas, and praise forms also have the time-saving effect of keeping your feedback flowing through a single channel. This keeps your CV project manager from spending time scrubbing duplicates, triaging mountains of data, and manually sorting all of your data.
So, When Should You Use a Survey?
“Surveys work best when you want to collect a large sample of data measuring your customers’ attitudes,” says Sabrina Solis, one of Centercode’s researchers. “They’re a perfect complement to other CV feedback channels, like issues, ideas, and praise, because the data provided by surveys puts that feedback in context.”
On the flip side, Sabrina warns against using surveys to collect usability information since they don’t produce reliable behavioral data. “People are typically unable to self-report their own behavior with accuracy,” she says. “Your product analytics tell a more conclusive story here.”
Similarly, survey responses don’t provide enough context to answer probing (“why”) questions. But they can push you in the right direction. “Surveys reveal opportunities to gather in-depth, qualitative insights. But you don’t necessarily need or want those insights from every single one of your testers — they take time to process. One of the many cool things about survey data is that the trends in customer attitudes it reveals can be a jumping-off point for deeper, more targeted product exploration.”
Before Building Your Survey, Ask Yourself…
Both Chris and Sabrina said the same thing: “First, ask yourself, ‘What am I trying to learn?’ Then identify the most effective way to learn that information.”
It’s common for your stakeholders to see surveys as the most effective way to answer all of the product questions your company has. But getting the depth of information you need to fully achieve your CV objectives — and not spending all your time triaging your feedback — means refining your processes so that they work for you, and not the other way around.
Learn the most effective strategies for collecting Customer Validation feedback with the best practices in The Feedback Playbook.
