Market research is being deluged with new sources of data, from social media, from electronic communications, and from research communities. Whilst some of this information is suitable for quantitative analysis, large parts of it are unstructured, for example tweets, posts, comments, and uploads. Whilst this data presents an interesting opportunity for market research, it presents a sequence of inter-connected problems and challenges:
- Many of the market researchers who are most proficient with unstructured data, the qualitative researchers, are not instinctively drawn towards online data, preferring to deal with people in a face-to-face environment.
- Many of the researchers most attracted to large amounts of online data, the hard core quantitative researchers, have little appreciation of the different epistemologies of quantitative and qualitative research.
- Many of the software vendors, perhaps in a rush to market, have released products before they were really ready and with massive over-claims.
In order for market research to fully leverage the potential benefits of the discourses being generated, market research needs to address the qualitative deficit. The qualitative deficit is the shortage of talent, software, and approaches designed to utilise massive amounts of qualitative data.
Some of the new approaches and skillsets that are needed relate to the process of quantifying messages within discourses – Google’s Flu Trends being a straightforward example (where Google use phrases that indicate people are searching for flu remedies to quantify the incidence of flu). This use of unstructured data will be of value in areas such as brand and ad tracking.
However, the bigger deficit relates to taking qualitative information and extracting qualitative findings. In online discussions the meaning is often unrelated to the frequency of words and phrases, the meaning rests in the structure of the conversation and the outcomes of the conversation. Feedback for product design, the identification of opportunities, core reasons for product dissatisfaction are likely to be found in the meaning of discourses, as opposed to counts of terms and phrases.
Whilst part of the answer will be new software, my feeling is that research urgently needs to expand the number of researchers with a good understanding of qualitative methods and epistemology.