2012 – the year manual coding made a comeback!

In 2011, at events and conferences around the world the world seemed to be on the edge of a new world, a world where automated coding, and in particular automated sentiment analysis, would allow researchers to tackle megabytes of open-ended text. A great example of that confidence was the ESOMAR 3D Conference in Miami.

What a difference a year makes. Last week in Amsterdam the news was all about researchers manually coding vast amounts of open-ended comments, because the machines would not deliver what the researchers had hoped they would deliver. The prize, undoubtedly, went to Porsche and SKOPOS who reported on a social media study where they captured 36,000 comments, mostly from forums, and ended up coding the comments manually.

I remain convinced that automated techniques will continue to develop and will soon open the door to large data sets. But for the time being, much of the material that market researchers handle will need to be, at least partly, coded by hand.

My suspicion is that Twitter will prove to be less useful than blogs, open-ended comments in surveys, and conversations in MROCs. When I work with Twitter, my feeling is that the grammar is to unstructured, the prevalence of irony too high, and the error by people tweeting too high to render even manual coding useful.
I think the swing back in 2012 was probably a response to the over-claims by many of the providers in 2011. I suspect that 2013 will be characterised by very specific examples where text analytics will have been applied successfully to a market research problem.

2 thoughts on “2012 – the year manual coding made a comeback!

  1. I agree with this Ray. I’ve used several different social media listening tools to gather data to enhance market research. In my experience, the automated categories are useful to get a general sense of what people saying about the topic (if they are saying anything!). But then one must carefully clean and hand code a sample of the comments to truly draw meaning.

    I used to try to code the entire data set like your example above, but that isn’t feasible or even necessary. Coding anywhere between 200-300 comments is usually adequate. You can provide the entire data set in Excel if someone wants to wade through all the comments. Coding is usually reserved for junior staff in market research, but given the unstructured nature of social media research, I found it helpful to code the data myself. Then it’s relatively easy to write a qualitative research report and use the coded category to summarize the findings.

    As for your comment about Twitter being less useful than more topic specific data sources, I would agree. The richest social media data tends to be in forums rather than on Twitter or Facebook. For many research objectives (especially for B2B), it is better to bypass the listening tools and just go directly to forums on the topic and read all of the posts.

    I am looking forward to the advancement of automated sentiment analysis and specific examples of their successful application to market research. It will save much time and aggravation.

Comments are closed.