You’ve designed your survey and you’re ready to launch—congrats! But one nagging worry: I’ve put all this work writing my survey, but what if people skip some of my questions? Ah-hah! I’ll just require that respondents answer all my questions—problem solved.
Well, not so fast. According to researchers there are some definite downsides to requiring questions.
Although it seems counterintuitive, if you require all questions in a survey you might end up with fewer responses overall. How does that work?
Say your survey included a question like "what color is your dog?", and you made the question required.
How can you answer this question if you don’t have a dog, yet the question requires an answer in order to continue through the survey? This is a frustrating experience for a survey taker, and they may get so frustrated that they drop out of the survey entirely. Not good.
Similarly, if you ask personal or intrusive questions, people may choose to end the survey rather than give an answer. A recent study by Jean Philippe Décieux et al showed that 35% of respondents dropped out of a survey when they were required to answer personal questions compared to 9% when they were allowed to skip questions that felt too personal.
This particular survey asked about relationship satisfaction and sexual history, but other types of questions that may be considered personal are those about health and medical history, criminal behavior, income and debt, or questions that just don’t seem relevant to the topic of the rest of the survey.
In addition to getting fewer overall responses, an even bigger potential problem with requiring questions is getting wrong responses. Let’s go back to the two scenarios above: the question that isn’t relevant (what color is your dog?), and the personal question that people don’t feel comfortable answering. Faced with either of these types of questions the survey taker could choose to drop out or they could choose to make up an answer or pick an answer at random in order to continue.
This situation is more than just hypothetical. In the same research study by Décieux, at the end of the survey, respondents were asked how honest they had been on the previous questions, and those in the condition where they were not able to skip reported higher levels of dishonesty than those who were able to skip.
In reality, dishonesty like this is even more problematic than missing data because enough of these untruthful answers could lead you as the survey designer to draw incorrect conclusions. So what are you to do if you want to get as much (accurate) data as possible?
The best advice? Only require questions when absolutely necessary. For example, it’s a great idea to make a question required in order to create weighted responses or to cut your data.
Some of this comes from the survey taker’s level of engagement. The more someone cares about a survey, the more time and effort they will put into completing the survey. Since Audience members are contributing to charity with each survey, they care a lot about the surveys they take.
An alternative is to require all questions but include a Don’t know or Prefer not to answer as answer chocies. Décieux’s study showed that required questions with a Don’t know option resulted in a dropout rate significantly lower than without a Don’t know (9% vs 35%) and a lower percent of respondents who reported answering less-than-honestly (13% vs 25%).
So now you know the ins and outs of requiring questions in surveys—congrats again! Always keep in mind that the more make the survey-taking experience a smooth one, you’ll have happier survey takers, and even better data.
