Six challenges facing Market Research and Insights

Posted by Ray Poynter, 5 August 2019

One of the questions I am often asked relates to the challenges facing market research and insights. The first thing to highlight is that the world of market research and insights is a very diverse group, including leaders of client-side insight teams, leaders and entrepreneurs from research agencies, leaders of suppliers to the research industry (e.g. panel and software companies), and employees of all of these (ranging from neophytes to veterans). Some of the key points are the same for all of these groups, but for some situations the emphasis is different.

Opportunity

Here are six key challenges.

  1. The need for speed
  2. Curation
  3. The decline of science
  4. The human dimension
  5. What AI? When? How?
  6. New business models

The Need for Speed

Most of the projects that organisations undertake are not research projects. They are projects where research can provide help, but they are not research projects. A new product is not a research project, the management of 160 stores is not a research project, and optimising the media mix from a new advertising campaign is not a research project. The speed of business is getting faster; agile product development, real-time, big data integrating with the management of stores, and programmatic advertising is taking the human out of the process. If research is going to expand its role, to promote the use of evidence-based decision making, it needs to operate at the speed of business.

Things that will help deliver increased speed are standardisation, automation, AI, and new business models.

Curation

There is too much data to handle in the ways we used to handle data. The data streams that are available today are not being adequately blended to maximise their usefulness. There are two key aspects to curation:

  1. Reducing the size of information presented to users. When the size of the data doubles, the size of the delivered answer should normally not increase. When the data becomes 1000 times bigger, the size of the answer should remain short and succinct.
  2. Drawing information from different sources to provide a clear and succinct answer. Note, blending data should not result in a bigger answer, just a better answer.

Curation is one of the key skills that needs to be taught to insight professionals. Curation is the art and the science of extracting answers that help businesses make better decisions. There is also a need for a major change from the suppliers of analytic and visualisation software. At present most software providers highlight how much they can produce, but they need to find a way of highlighting how little they need to produce to provide the relevant answer. One of the interesting developments in AI is the growth in summarising techniques (including topic modelling).

The Decline of Science

There seems to be a decline in the confidence in science and an increase in the acceptance of opinion. The rise of interest in the Dunning-Kruger effect is perhaps a symptom of this change. It is clear that there are many leaders, followers, and others who feel their views on climate change, the impact of trade tariffs, and the economic consequences of Brexit are equally valid to those of the experts.

The challenge for insight professionals is that evidence from research that contradicts the views of stakeholders is likely to be ignored. This change means that the rise of tools such as video and storytelling even more important. Insight professionals may choose to use science to find their answers, but they need to utilise emotional techniques to convey their results.

The Human Dimension

In the rush to big data, automation, and AI, the human dimension is becoming increasingly important. The more we know what people do, the more we need to know why they do it and what interventions might cause people to change what they do. Market research has always focused on understanding people, and since the 1940s has sought to do that by utilising both qualitative and quantitative approaches. Key elements of this human-centric focus will include:

  • Establishing what the business question really is.
  • Defining a research question, something that research can find out that will help answer the business question.
  • Understanding the research results. With Big Data, AI and automation the calculation of the results will increasingly be a task for data scientists and bots, but the interpretation requires human insight.
  • Finding the story in the results.
  • Communicating the story in ways that lead to action.

What AI? When? How?

Automation is now a mainstream component of market research and insights, but there are a large number of questions about how and when AI is going to be deployed more widely in market research. We are seeing some early steps in areas like panel management, the coding of open-ended comments, speech-to-text translation for video, sentiment analysis, and chatbots. However, none of these have made major changes, yet. For example, companies not using AI are still able to compete.

Coming back to the issue of speed, I think the key developments will be those that facilitate faster research. I foresee three levels of AI

  • Things which optimise processes (data cleaning, routing, management etc)
  • Things which improve the scope of DIY tools – so non-experts can do more, and do it quicker and safer.
  • Things that will enable power users to do more. One of the key shortages in the future will be in the area of experts (including data scientists, business analysts, and ethnographers) – tools that make experts more productive (i.e. faster) will be valuable.

New Business Models

The traditional research model was that a client would speak to a small number of agencies, and issue RFQs. The agencies would bid for the project, centring their timeline and costing on the data collection process, and one agency would be appointed. The project would then be conducted and debriefed, with the focus of the presentation being the research that has just been conducted.

Clients are increasingly finding this business model, based on commissioning data collection, does not work for them. Data is cheaper, data is plentiful, data sources need blending, and the value is in the answers, not in the process. Models with low marginal costs (for example a simple, automated project from a self-serve portal), or a DIY project, or a project that is part of a subscription (e.g. an online community) are all more attractive than the traditional model. Perhaps other new models will appear, for example, perhaps somebody will get the market place model to work. Or perhaps a gig economy system where an app or company sits between end clients and large numbers of skilled freelancers will evolve. But I think it is clear the old business model will decline.

Advice to Young Researchers

Whilst the picture is different for different people, the group I am most often asked about are the people new to the world of market research and insights. Here are my thoughts specifically for them.

  • Understand business – for example, how does your client make its money?
  • Understand people – even data scientists need to understand motivations and psychology
  • Be T-shaped – have a broad understanding of the core skills for market research and insights and have a deep speciality (it could be coding, it could be semiotics, it could be visualisation – the topic almost doesn’t matter, as long as you are good at it and passionate about it).
  • Learn how to be fast – keep looking for approaches, tools, methods, apps etc that will allow you to work faster than other people around you.

7 thoughts on “Six challenges facing Market Research and Insights”

  1. You are so on point – as always. Great summary. Thanks for providing such a well articulate synopsis! I will be quoting you – if that’s ok!?

  3. As insightful as ever Ray. It’s interesting to me that you point out the “decline of science” yet numbers are still king in so many places. There is a misconception that Qualitative = Opinion when, in fact, it’s simply additional data to curate.

    My advise to more experienced researchers is to challenge the status quo. Just because you’ve always done it this way doesn’t mean you need to continue to do it the same way. Occasionally go back to the start: “What is it that the business needs to make great decisions?”

  4. Ray. I found your analysis of the main challenges facing market research spot on. Thank you for these observations and reflections.
    Looking on the positive side – if as a industry we become the go-to experts in being the wide angle lens of organisations, able to make sense of (curate) multiple sources of (often imperfect) data, then we will always be in demand.
    If we add to this the ability to create powerful business solutions by applying agile methodologies – mastering the build the plane as you fly it philosophy – then this will further strengthen our position.
    And, if we can boost our capabilities as storytellers, able to tell the insight story in an influential and persuasive way, then we will be offering organisations a powerful skills package – particularly if we cultivate the ability to win both the evidence based (rational) and the emotional argument.
    Mastering these skill-sets is a challenge, but as an industry we start from a sound platform. A lot of our success will be about thinking big, believing in ourselves and enjoying the journey to the next level of what it takes to be a high performing customer insight professional.
    David Smith

  5. Hi Ray – many thanks for an excellent and very thought-provoking article.

    The sentence that stands out for me is when you speak about new business models, and clients finding self-serve portals, DIY/ subscription projects being more attractive than the traditional model for data collection.

    I do agree with you, but would be really interested to hear your thoughts on whether you feel the profile (i.e. skill-set/ job role) of the clients responsible for the data collection and analysis is changing/ has changed too?

    We know the numbers of those working in a pure insight function in client companies have been on the decline for a while, so do you think there is a still a future role for an insight professional within this new model, or do you think that role will be squeezed by marketers/others at the front end and the data scientists at the back?

  6. One of the questions any industry has to pose from time-to-time is what are our core competencies. Like you I think the ability to understand business problems and to employ critical thinking to answer them is a core competence we need to have to thrive over the next 20 years.

  7. Any article mentioning Dunning-Kruger makes me clap in glee.

    The advice for younger researchers to understand their client’s business, so important. And appreciated by the client. That should drive the study design more than anything else imo.

    Using video as an example of ‘unambiguous’ data point sure is a good recommendation. Also, that does liven up a research report.

    Thanks, Ray for sharing this.

Comments are closed.