When and why does the wrong sample give you the right answer?

Most samples used by market research are in some sense the ‘wrong’ sample. They are the wrong sample because of one or more of the following: They miss people who don’t have access to the internet. They miss people who don’t have a smartphone. Not representing the 80%, 90%, or 99% who decline to take part. They miss busy people. Samples that suffer these problems include: Central location miss the people who don’t come into central locations. Face-to-face, door-to-door struggles with people who tend not to be home or who do not open the door to unknown visitors. RDD/telephone misses people who decline to be involved. Online access panels miss the 95%+ who are not members of panels. RIWI and Google Consumer Surveys – misses the people who decline to be involved, and under-represents people who use the internet less. Mobile research – typically misses people who do not have a modern phone and who do not have a reliable internet package/connection. But, it usually works! If we look at what AAPOR call non-probability samples with an academic eye we might expect the research to usually be ‘wrong’. In this case ‘wrong’ means gives misleading or harmful advice. Similarly, ‘right’ […]

Analysis, the difference between qual and quant

Earlier this month, NewMR held its first Explode-A-Myth session and my contribution was a discussion why there is no method that is a melange of qual and quant, because the underlying paradigms are different. Through the Q&A session at that event, and in particular a question from Betsy Leichliter, I gained a clearer understanding of the core difference between qual and quant. Betsy asked “So should the ‘qual’ or ‘quant’ labels be driven by the method of analysis, not necessarily the method of “data collection”?”. I think this question from Betsy is the best answer to the question about what is the difference between qual and quant I have seen. Within reason, any data can be assessed quantitatively or qualitatively. Of course, there are some limits to both approaches. A very small amount of data is likely to produce findings that are hard to generalise. We can count the sales of brand X, in one store, on one day, but it is hard to draw any inferences about the world from that. Similarly, ten-thousand open-ended responses could only be assessed qualitatively with a large team, or a large amount of time. The quantitative approach is based on an assumption that there […]