NewMR – The Big Picture
Sometimes when I run a workshop or training session people want detail, they want practical information about how to do stuff. However, there are times when what people want is a big picture, a method of orientating themselves in the context of the changing landscape around them.
Tomorrow I am running a workshop for #JMRX in Tokyo and we are looking at emerging techniques, communities, and social media research – so a big picture is going to be really useful to help give an overview of the detail, and to help people see where things like gamification, big data, and communities all fit.
So, here is my Big Picture of NewMR (click on it to see it full size), and I’d love to hear your thought and suggestions.
The Big Picture has five elements
The heart of the message is that we have reached an understanding that surveys won’t/can’t give us the answers to many of the things we are interested in. People’s memories are not good enough, many decision are automatic and opposed to thought through, and most decision are more emotion that fact. Change is needed, and the case for this has been growing over the last few years.
The four shapes around the centre are different strands that seek to address the survey problem.
In the top left we have big data and social media data, moving away from working with respondents, collecting observations of what people say and do, and using that to build analyses and predictive models.
In the top right we have a battery of new ways of working with respondents to find out why they do things, going beyond asking them survey questions.
In the bottom left we have communities, which I take as a metaphor for working with customers, co-creating, crowdsourcing, treating customers and insiders, not just users.
The bottom right combines elements from the other three. ‘In the moment’ is perhaps, currently, the hottest thing in market research. Combining the ability to watch and record what people do, with interacting with them to explore why and what they would do the options changed.
Thoughts?
So, that is my big picture. Does it work for you? What would you add, change, delete, or tweak?
7 thoughts on “NewMR – The Big Picture”
Comments are closed.
Great diagram, a really useful summary. I’ve tended to see the problem as a bit bigger though – it’s not just that surveys don’t work, but also interviews & focus groups – these three being the classic tools in the market researcher’s kit.
The problem may in fact lie with the “direct question” at all – certainly behavioural economics would say so, ditto neuro, and we at FACE argue that the value in social is substantially from its unprompted nature. The more we learn about decision-making & behaviour, the less good direct questions seem to be at getting us good information.
Would you agree?
Hi Jay, I would agree about direct questions, but my experience qual is that it very rarely uses direct questions. A good qual researcher does not base their report on what people say, but on what they mean and why they say what they say. Good qual, and I see a lot of good qual, is firmly rooted in good psychological theory and uses a variety of indirect approaches to uncover why lies below.
I do hear about a lot of factory qual where people work through a detailed discussion guide and report what people say, but I don’t personally come across that nearly as much as the good stuff.
I agree and it’s just a matter of time. In the old days, opinions were hidden, technology was not advanced and having a survey was the appropriate approach. Nowadays everyone can have their says at any time, any where, and the technology over achieves in accommodating our needs on information. In fact, overloaded.
We need a reliable tool to filter what is good or relevant and what is ‘noise’. Until we have that, I would think we still rely on surveys and a good design survey will get us a reliable information, directly or indirectly. We live in the transition and I would make use of both past and future approach. Oh, I still believe the technology can’t ever replace the human interaction during focus groups or qualitative approach. We have a sense of human that only human can get it. That’s my two cents…
Hi Ray — This is a nice picture of the NewMR world view, but it is more religion than science. Surveys have been under the microscope for over half a century with legions of very smart people–mostly academics–studying the error properties of surveys and their sources. None of the other methods on your chart have been subjected to anything close to this kind of ongoing and rigorous evaluation. Surely you don’t mean to say that they are error free. We need something like the Total Survey Error Model for each of those new methods, with a record of empirical research to back it up. As we work our way through that exercise we may find that, to quote the Grateful Dead, “The grass ain’t greener, the wine ain’t sweeter, either side of the hill.”
Hi Reg, I certainly agree that we need much more research into the extent to which new research techniques can be relied on, i.e. to what extent do they report what they claim to report and what confidence can we have in the findings. However, there is some good work going on, for example I am coming across more academics who are researching gamification and I think the peer reviews work the IPA has done on what works in advertising and more recently what works in social is great.
However, I am not sure that I would be as sanguine as you about commercial surveys. IMHO, very little of the findings from the academic work from the last 20 years has made it into commercial practice. Even the most useful of Krosnick and Couper’s findings do not seem to be part of everyday practice in the surveys that I see going through the big online access panels.
I will come back to my misgivings about how Total Survey Error is applied in another post (I don’t have a problem with the concept), but I will highlight one key area. Questions about intentions and motivations are very poor predictors of behaviour. Jan Hofmeyr has some great examples in his current presentation deck which show that in many cases what people say does not predict what they do any better than chance. In these cases, any effort spent on sample error, non-response error, frame error and measurement error is likely to delay a solution. The problem, in my very humble opinion, is specification error. The researcher thinks they are measuring intentions, but they are in fact measuring something else (perhaps beliefs about intentions, perhaps forms of response, perhaps discourse practices). In those situations where the researcher is choosing to measure the wrong thing, time spent improving the other elements of TSE is time wasted as it is generally an excuse to keep asking the wrong questions. Quite often when I attend an academic event (most of which I find both useful and enjoyable) I am struck by how often the speaker is describing the problem as measurement error (i.e. we did not have the right scale, the scale was the wrong way round, the respondent was not trying etc) as oppose to specification error (the questions you are asking are not directly related to the phenomenon you say you want to research).
My other concern with academics are the silos. When I attend a conference of methodologists, their knowledge of fields like discourse analysis seems to be very poor. This can lead them to either addressing a problem that has been solved in another way already, or it can lead them to miss-specify the problem.
Very useful diagram of the different ways different ways to address the survey problem. From my point of view, the parts that are missing are arrows between the four alternatives you described.
The integration of (and gap analysis between) results from various research approaches is in itself an important means of identifying potential distortions from what is really impacting purchase decisions versus considering traditional quantitative surveys. Your “in-the-moment” component goes in this direction by integrating both a live behavioral and interactive attitudinal component, but I believe that the power of multi-modal research goes beyond just live “in-the-moment” approaches.
In short, I don’t believe a single superior, always accurate alternative to quantitative surveys is likely to come from any of the four categories in the diagram. I reminds me of the old story about blind men feeling different parts of an elephant. It’s only by talking to each other (integration of multi-modal data collection) that the true nature of the elephant can be discerned.