Are market research surveys qual or quant?
Posted by Ray Poynter, 21 June 2019
To market research and insight professionals this may sound like a silly question, the received wisdom is that surveys produce quantitative information. However, over the last couple of years I have, on several occasions, heard data scientists refer to the results from 1000 surveys (or even in one case, 10,000 surveys) as ‘qualitative’. My initial assumption was that these data scientists simply did not understand the principle of sampling, i.e. that a sample of 1000 people, drawn randomly from a normal distribution, will produce estimates of the population mean that, 95% of the time, are within plus or minus 3% of the true value. Because of this lack of appreciation of the science of sampling, the data scientists were failing to see that samples of 1000 surveys were more like their data set of 10 million than they were like 16 depth interviews.
However, more recently, I have started to wonder whether there is another way of addressing this issue. Previously, I used the definition of qual and quant common to social and market researchers. Broadly this relates to qual being the creation of descriptions of the world by applying a subjective assessment of observed phenomena, and quant being the application of algorithms to discover/measure things.
However, there are other ways of looking at qualitative. For example, qual is about describing, it is about assessing the nature of things, it is about judging what matters. Let’s think about a U&A survey for a supermarket and compare it with the big data / data science approach. The big data approach measures things that require little interpretation, for example what was purchased, how was it paid for, what things were purchased in combination. This data can be processed to find a wide range of things, such as what happens when prices and choices change, or whether somebody’s home circumstances have changed (e.g. somebody has moved in, or their income has gone down, or they have become vegetarian).
If and when we market researchers use a U&A, we tend not to focus on what people do (although that is part of what we do), the focus is how they feel about what they do, why do they do what they do. If we take a question like “How likely are you recommend this supermarket to friends and family on a scale 0-10?” are we really measuring the sort of thing that a data scientist would measure? People do not actually have a number in their mind in the range 0-10. The shopper has a set of emotional relationships to the supermarket that are in flux, they are engaging in a qualitative exercise when they give us a number – they are constructing the number to represent how they feel. Not only is this number not ‘real’ in the sense that ‘how many tins of beans they bought’ is real, it is not stable. If we ask people the same question again it can and does changes – in the sense of the data scientists it is not a characteristic of the person – in the way that their bank record is a characteristic.
It could therefore be argued that a large part of social and market research is focused on understanding qualitative topics, for example what do people think, why do they think it, how strongly do they believe it etc. Although the tools used, from averages to structural equation modelling are quantitative tools, the essence of what is being researched is qualitative. Perhaps this is part of the language gap between Data Science and Market Research? Market Researchers might be using the terms qual and quant to talk about how the measurements are conducted. But data scientists might be focusing more on the nature of what is being measured?
As a simplification of the argument:
- In market research qual the participant and the researcher make a qualitative assessment about something.
- In market research quant the participant makes a qualitative assessment about something.
- In data science quant nobody makes a qualitative assessment about the action being measured.
Thoughts?
5 thoughts on “Are market research surveys qual or quant?”
Comments are closed.
This is an interesting point of view, one that I can appreciate. Quantitative shouldn’t simply be defined by whether numbers are used but rather by what those numbers reflect and how confident we are about those numbers.
I do think there are some qualitative assessments coming from data science though. Any sort of transactional data analysis will bucket certain products together which causes researchers to hypothesize about reasons and emotions.
There is room for researchers to be less strict about what qualitative and quantitative really mean.
Hi, Ray. I like the definitional distinction, but I think we may be missing something important. This has to do with the reliability of the information for decision-making. Traditionally, this has been one of the differences between quant survey data and qual. You rightly touch on this with your comments on sample size and statistical confidence levels.
By way of example I was involved with a client that is a large retailer in the U.S. with thousands of stores and millions of customers. The client has a few competitors of similar size. They have traditionally relied on their internal sales data, and to some degree on loyalty data. They are, surprisingly, relatively new to primary research.
In this project they conducted a survey with over two thousand respondents. This should be a sound foundation on which to reach conclusions and make decisions, right? It was fascinating, however, to see how quickly this “stable” foundation became “unstable”. As one partitioned for competitor, geography, districts, stores, and demographic subgroups, sample sizes reduced to the point where conclusions were untenable. Indeed, in many cases client questions could not be answered at all because there was no sample. The report actually emphasized this “unreliability”; page after page had MR’s standard caveat – sample size is too small, and conclusions are directional at best.
This is not to say that client was unhappy with the project. It provided a perspective that was missing from their secondary data sources. However, the majority of findings in this project are best described as “qualitative” because the foundational sample size is inadequate. They make for interesting ideas and hypotheses – which is traditionally the role of qualitative research. It is easy see why data scientists – and client management – would regard quantitative survey research as qualitative. From their perspective, a survey of 30 or 300, or 3000 cannot have the same reliability as millions of customers or millions of occasions or thousands of stores.
I would be interested in your thoughts.
Thanks for the comment, however, I think it runs the risk of falling into an all too common problem, that of assuming anything that not A is B. In this case that research that is not Quant is therefore Qual. In order for something to be qual it has to tell us something about the essence of something. A survey of 3000 people would, if it had been well designed, facilitate quantitative analysis and produce quantitative inferences. The same survey might also be amenable to qualitative analysis, if it had been designed in a suitable way (e.g. collecting video, open-ends, biometrics on a relevant topic), and in that case, would facilitate qualitative inferences. However, if we take a survey of 3000 people and we find we have 8 interviews for a particular store or demographic, it might be neither Quant or Qual. It is not quant because it is too small to draw statistical inference, but it might not be qual either. There are several reasons why it might not be qual, for example: because the questions asked did not lend themselves to qual analysis, or because the composition of the 8 responses did not reflect a relevant sub-group of customers (qual also requires that the right people to have been researched. In this case ‘not quant’ might meant not anything. I think we need to avoid thinking that ‘not quant’ means it ‘is qual’.
In your case, millions of records about what people do will be more reliable than 3000 survey responses about what people do. However, if we are trying to work out more qualitative questions, such as why people buy more X at the weekend, the survey may be much more reliable – compared with a data scientist looking at the transactional data and making a guess about the why.
Hey Ray,
I think my comments are being misconstrued as an argument for or against something; e.g., for or against survey research. This is not my intent.
The original question was why might data scientists perceive survey data to be qualitative instead of quantitative. In my evidently poor example, there was a client with big data and data scientists with little experience with primary research. They conducted a fairly large survey. Sample size was more than adequate to answer some questions with confidence, and some questions could not be answered with confidence – not unlike most surveys. In this case, and as the information was shared across the company, there were more questions that could not be answered than could be.
As I noted, the client was actually quite happy with the study; it provided a lot of information on the “why’s”, and comparisons to competitors that were unavailable from their secondary data sources. However, the general conclusion was that the study was more useful in the hypotheses it generated than in the conclusions. This is not a bad thing, but in thinking about how client talked about findings, it was very clear they perceived these survey findings to be more “qualitative” than “quantitative”. It was their perception, based on their definitions, and surprisingly I can see their point of view.
With regard to your definitions, I did think it might be useful to have more discussion on the question of managerial confidence. Most managers I have known go through three phases for good research; (1) that’s interesting and has potential, (2) I think I know what to do but it will cost money, and (3) do I have adequate faith in the information to make the investment? In the old days answering that last question was mostly about sample size and statistical confidence. Now it might be about big data and data science.
Clearly we don’t have to make “confidence in the information” part of the discussion, but it is equally clear it is affecting our world and maybe it should be part of the discussion.
Jeff
Yes, managerial significance is very important. With non-survey sources, such as transactions, the sample sizes are enormous, so almost any difference is statistically significant, but most of them will not be managerially significant.