Thoughts on Validity

Recently Kevin Gray asked, in the LinkedIn NewMR group, “When you hear a claim that a marketing research methodology is “valid”, what does that mean to you?” The question elicited a range of views and here is a tidied up version of my thoughts.

My feeling is that we need to start not with science or the theory of research methodology (both passions of mine BTW), but with what the users of research want and need the research to deliver. I think that what a research user means by valid is that what the research tells her/him is true*, that the results do not exclude important information, and that the information useful.

In my experience, much of the criticism of market research focuses on the second two criteria, i.e. that research does not include everything that the user needs to know and that the information is not sufficiently useful. Much of the drive towards new research (e.g. mass mobile ethnographics) is driven by a desire to produce findings that are more complete and more useful.

The tests for qual and quant in terms of complete and useful are very similar.

*The BIG problem with my definition is the reference to ‘true’. The traditional quant epistemology (positivism) held that absolute truths can be established, and that information that was not reliable (i.e. replicable) and not consistent with a theory was not valid. Quant research has established a family of validities, ranging from face validity at the weakest end of the spectrum i.e. does it look right) through to construct validity (is it consistent with theory and can it be tested).

Move the clock forwards, past Karl Popper and Thomas Kuhn, and into post-positivism and we see that research has had to soften its definitions. Most quant researchers accept that when measuring and researching human feedback and society there can be no absolute truth. The reflexivity of the researcher, the biases of observation, and the feedback loop in human systems mean that there are no absolute truths, just associations that for a period of time are useful. In modern market research the term valid tends to be used in a sense closest to Convergent Validity – i.e. to what extent does a method produce results that are similar to ones that it is supposed to be similar to (e.g. predicting sales).

Qual research occupies a much wider spectrum, in terms of epistemology, than quant research. There are post-positivists who aim for something similar to quant researchers, i.e. replicable research that predict real world events. There are naturalists who liken themselves to 19th Century botanists, as opposed to 20th Century physicists, who seek to accurately describe what they see, even though they can’t explain the underlying mechanisms. However, there are various modern schools who reject the notion of objective truth completely, who typically seek to create a narrative that is credible and useful.

Whenever methodology and theory is discussed, somebody (or several somebodies) chime in with comments that say clients do not want to know about methodology or even that clients do not care about methodology. Whilst I am sure that only a minority of clients want to delve into the minutiae (although I have dealt with three clients who have PhDs in the area and who were very interested), I am sure that most clients want to know that the research they are buying/using is in some sense ‘fit for purpose’, i.e. that it is likely to ‘work’. This means somebody has to be interested in methodology, somebody has to put the hard yards in, so clients don’t have to.

If you want to join the debate on validity, why not visit Kevin Gray’s discussion on LinkedIn? []