Here in Silicon Valley, it seems like everybody’s got a startup. And one of the most important things about making your startup a success is getting the word out.
Let’s say you’re gearing up to launch your personal shopper startup. You’ve got everything set to go and now all you need is a killer ad campaign. Your design team put their heads together and come up with two different ads they like:
Buuut, no one can agree on which ad is best and tempers are flaring. Which ad communicates your image the clearest? Which will people like the most? Which will increase your sales? Well, tell your staff to simmer down because there are two kinds of survey designs that can help you find out.
DESIGN 1: WITHIN-SUBJECTS
In a within-subjects design, every person who takes the survey sees both ads, and then answers questions about each ad including how likely they’d be to shop at your store.
Design strengths:
- You will get an exact comparison. The people who see the first ad are the same people that see the second ad. That means you can directly compare each person’s rating of the first ad to their rating of the second.
- You need fewer respondents to get meaningful results–to learn more about calculating sample size, just click here–this can make your survey process cheaper and quicker.
Design weaknesses:
- The dropout rate can increase. Let’s say there are 20 questions about each ad. In a within-subjects design, each person has to answer 40 questions total (20 for each ad). Making a survey longer means that people are less likely to finish it.
- You only get that “blank slate” reaction once. A survey respondent’s reaction to that second ad will be biased by the ad they saw first. For example, if they saw the ad with the woman standing in the clothing store first (Ad Version 1), that might communicate that your brand is about in-store customer service. When they see the ad of the woman in a mall with shopping bags (Ad Version 2), they might dislike it just because it doesn’t match what they now think your brand is about.
How do you fix it? Randomization of which ad you see first will make sure that both your ads take turns getting that first spot, but it won’t take away the bias or fix the dropout problem. If you think that these might be a problem for you, you might want to use another kind of design.
DESIGN 2: BETWEEN-SUBJECTS
In a between-subjects design, each person who takes the survey sees one ad OR the other—but not both. In this design, your sample would be split into two groups of respondents, one group that sees the clothing store ad and one that sees the ad with the shopping bags.
Design strengths:
- With between-subjects designs, respondents only have to answer questions about one ad and not both. This cuts the survey time in half (compared to a within-subjects design) and helps to keep the dropout rate down.
- Respondents see only one ad, so all of their responses are guaranteed to be unbiased. That lack of bias makes this design better at capturing subtle opinion shifts. Survey respondents may not know that an ad is influencing their opinions, but it may be happening nonetheless. For example, let’s say the group who sees the shopping bag ad thinks that your service seems very fun on average, but the group who sees the clothing store ad thinks that your service seems moderately fun. You might not have seen this difference if you were asking the same person to rate both ads.
Design weaknesses:
- You will need twice the amount of respondents that you’d need for a within-subjects design, this can make a between subjects design slower and more expensive. (However, remember that with dropout rate down, this may not be the case.)
- You also lose the direct comparison that you had with a within-subjects design. That is, there is a chance that people who see the clothing store ad are different from the people who see the shopping bag ad. Maybe they’re just grumpier and would hate anything you showed them!
How do you fix it? The problem of losing direct comparisons can be addressed with random assignment. Random assignment of respondents means that every person in your sample has an equal chance of getting assigned to see either ad. So, each grumpy person has an equal chance of seeing the clothing store ad or the shopping bag ad. Consequently there should be no mean differences in grumpiness between the groups, because respondents are sent to one group or the other completely randomly.
Now that you know the strengths and weaknesses, pick your survey design and watch the cash come rolling in!
Have a question about what design to use for your survey? Ask us below!
