In a perfect world, all the data you collect from your surveys would be 100% unbiased. But I don’t have to tell you that’s not the world we live in. In reality, bias is common in research. These psychological biases — the tendency to make decisions or take action in an illogical way — are introduced both in the way questions are asked and the way they’re answered.
Fortunately, addressing these biases does wonders to help you notice when they creep into your survey questions — and inadvertently skew your results. This blog will dive deeper into the most common psychological biases found in survey research and how to combat them to get more reliable data.
Social Desirability Bias (Conformity)
This bias occurs when people report attitudes or behaviors that they feel are socially desirable (or underreport what they feel is undesirable). Deep down, people want other people to think the best of them. This is especially true when you’re asking something personal or sensitive in nature (such as details about personality, knowledge, or performance).
Example:
How easy/difficult was it for you to install your router?
Very difficult
Difficult
Moderate
Easy
Very easy
Asking someone to admit to a potential shortcoming (i.e., it was difficult for them personally to install the router) might discourage a truthful response. In the suggestion below, the emphasis falls on the installation process, not on the person performing it. It’s a subtle but powerful difference.
Try this instead:
How easy/difficult was the router installation process?
Very difficult
Difficult
Moderate
Easy
Very easy
Other ways to discourage social desirability bias include: ensuring anonymity, randomizing options (and possibly questions), and cross-referencing data to identify outliers.
Acquiescence Bias (Yea-Saying)
It might surprise you, but people generally want to be agreeable. This causes another type of bias: acquiescence bias, or the tendency to report attitudes and experiences in a more positive light. Acquiescence bias goes hand in hand with social desirability bias because people want you to think they’re agreeable and polite. As you’re writing your survey, make sure you avoid framing your questions in a way will that unfairly bias the results in your favor.
Example:
Did our services meet all of your needs?
Yes
No
Because this question is a yes/no, people are more likely to respond “yes” — even if their needs weren’t completely met. By offering a scale, survey-takers are able to respond in a way that reflects their experience more accurately.
Try this instead:
Please indicate the extent to which you agree/disagree with the following statement: Our services met your needs.
Strongly disagree
Disagree
Neutral
Agree
Strongly Agree
Order Effects Bias (Question Order)
In psychology, “priming” is an effect that occurs when one stimulus influences the response to another stimulus. For example, people who see or hear the word yellow will recognize the word banana faster than those who don’t!
The order of your survey questions can change the results in a way that is very similar to priming. By asking particular questions first, you are potentially influencing the response to later questions. If you’re not careful, this will bias your overall responses and affect the accuracy of your data.
Example:
1) Describe any problems you faced while using this product. [insert answer choice]
2) How dissatisfied/satisfied are you with this product?
Very dissatisfied
Dissatisfied
Neither satisfied nor dissatisfied
Satisfied
Very satisfied
Priming survey-takers to think of the negatives before they assess your product overall could falsely weigh down your satisfaction scores. By asking respondents to rate their satisfaction then explain why they felt that way, they’re able to report their overall perception of the product experience with less bias.
Try this instead:
1) How dissatisfied/satisfied are you with this product?
Very dissatisfied
Dissatisfied
Neither satisfied nor dissatisfied
Satisfied
Very satisfied
2) Please explain why you were [insert answer choice].
Think about the mindset you are putting your respondents in. You can also combat question order bias by randomizing question categories. And, of course, by testing your survey.
Confirmation Bias
Biased data doesn’t always come from the people taking the survey. When you start building your project objectives, having a hypothesis in mind for how that research data will look is a given. Confirmation bias occurs when a researcher is so focused on proving their hypothesis that they ignore any data that could disprove it.
It’s important to recognize when you’re projecting your thoughts and attitudes onto your data, rather than staying true to your audience. Here are three steps to combating confirmation bias in your research.
- Acknowledge confirmation bias
If you start to see results that confirm your hypothesis, it’s important to take a step back and acknowledge any personal biases. - Ask yourself, “How does this data back up my hypothesis?”
Take a moment to check in and see what’s supporting your hypothesis. Are there many indicators pointing to your hypothesis or just one? - Have someone else review your results
It’s always good practice to have someone check your work. A second pair of eyes (especially when those eyes belong to a trained researcher) exposes biases that you are too close to your research to see.
Ready to put these techniques into practice? Learn how to pull in squeaky clean data from your CV surveys with the structure and question writing tactics in our free ebook, Building Surveys for Customer Validation.
