Asking the wrong questions gives you the wrong answers!
A very large part of market research is based on asking people questions, for example in surveys, focus groups, depth interviews, and online discussions. In general, people are very willing to answer our questions, but the problem is that they will do it even when they can’t give us the right answer.
At IIeX last week Jan Hofmeyr shared the results of some research where respondents had been asked about which brand they buy most often and he compared it to their last 3 and last 6 purchases from audit data. He found that in the last 3 purchases 68% of people had not bought the product they claimed to buy ‘most often’, and in the last 6 purchases 58% of people had not bought their ‘most often’ brand.
The video below is designed for entertainment, but it illustrates the bogus answer problem really well:
There are two key reasons why asking questions can produce bogus answers:
- Social desirability bias. People are inclined to try to show themselves in the best possible light. Ask them how often they clean their teeth and they are going to want to give an answer that makes them look good, or at least does not imply they are lazy or dirty. In the video, many of the people know that music fans are supposed to know about music, so they don’t want to appear dumb.
- Being a poor witness to our own motivations and actions. Writers like Daniel Kahneman, Dan Ariely, and Mark Earls, have written about how people tend to be unaware of how they make decisions. Some of the people in the video are being primed in the question to assume that they know about the brand may possibly be deceived by their own thought processes, with what they do know being used a s pattern generator to produce plausible thoughts.
Of course, in addition to these two reasons, some people simply lie – but in my experience that is a tiny proportion (when seeking the views of customers and the general public) compared with the two reasons listed above. However, the problem of conscious lies increases if incentives are offered.
One way to reduce the number of false answers is to make it much easier for people to not answer a question, ideally by not having to say “I don’t know”, and letting people guide you to the strength of their answer. Look at the video again and you will see that many of the people being interviewed are trying to signal they don’t really know about the bands, for example “I don’t know any of their music but I’ve heard from my friends that ….”. For the sake of the interview and the comedy situation the interviewer presses them into appearing to know more. In an information gathering process we should take that as a cue to back off and make it safe or even ‘wise’ to avoid going any further.
Another important step is to avoid asking questions that most people won’t ‘know’ the answer to, such as “What is the most important factor to you when selecting a grocery store?”, “How many cups of coffee will you drink next week?”, “How many units of alcohol do you drink in an average week?”.
If you’d like to know more about asking questions, check out this presentation from Pete Cape.The problems with direct questions are one of the major reasons that market researchers are looking towards techniques that use one or more of the following:
- • Implicit or ‘neuro ‘techniques, such as facial coding, implicit association, and voice analytics.
- • Passive observations, i.e. recording what people actually do.
- • In the moment research, where people give their feedback at the time of an event, not at a later date via recall.