In this post Ray Poynter sets out a set of rules that help ensure that a questionnaire collects valid information.
Are you asking survey questions the way you were taught 20 years ago, or are you learning from recent ‘Research on Research’?
Back in 2010 ,I caused a minor stir in the research world by predicting (at the MRS Conference in London) that surveys would have disappeared in 20 years (i.e. by 2030). This prediction was put into wider circulation when I clarified my prediction in a blog. The key point being that I was predicting the end of the commercial, long survey, and it being replaced with social media listening, online communities, new ways of researching, the use of open-ended questions, and the use of stored information to remove the need to keep asking questions. In 2014 I updated my prediction and showed some numbers from the ESOMAR Global Market Research Report. The table below shows the figures from ESOMAR for 2007, 2010 and 2013, and my projections for 2016 and 2019. Note the figures show the spend on research, not the volume. (Click on the tables to enlarge them.) So, how did my predictions stand up? The table below shows the ESOMAR figures for 2016, below my estimates. Note, I have added a new column which combines Other Quant (e.g. traffic and audience data) with Other (e.g. big analytics). In the future I will focus on Surveys, Qual, and a single […]
In this post I am sharing the summary and two key charts. The eight-page version of the results can be downloaded. Summary The top four things that I want to share about the use of statistics and statistical tools are: Most statistical tests/approaches are not widely used. Only Correlation, Regression, z- or t-tests, and Cluster Analysis have been used by more than 50% of the participants in this research, during the first half of 2017 – and this sample probably over-represents people using statistics, and under-represents those using statistics less often. SPSS is the dominant software package amongst people using statistical packages. Given SPSS is approaching 50 years old, that may not be the sign of a dynamic industry? But, there are many people using tools such as Q, Sawtooth Software, SAS – and beyond them programs such as Latent Gold, Tableau, and XLSTAT. One of the growth areas is the use of tools is the use of integrated data collection / analysis solutions, for example Confirmit, Askia, Vision Critical, Qualtrics. The use of these tools requires the researcher to make fewer decisions. For example, survey monitoring flows into the analysis without any extra steps, the packages have a default […]
At the moment (August 16 to August 31, 2017) NewMR is running a survey to collect data about the stats commonly used in market research. [Note, since August has finished, the survey is now closed. We will be posting the results soon. However, you can still see the raw data report below.] If you have not already taken the survey, please do so [by clicking this link] before reading the data below. The background to this survey is that we are writing some materials for two university courses and for workshops that we are running. We would like a clear idea of which stats are commonly being used, and which are more specialist. Stats that are commonly used need to be taught in a way they convey how to use them as well as when to use them and how to interpret them. Our feeling is that stats that are more specialist should (in the context of the courses and workshops we are involved in) be focused on when to use them and how to find out how to use and interpret them. Below is an automated report of what the data looks like at the moment (you might want […]
Last week I had was invited to have a play with the 1Q.com system, an innovative and new alternative for market researchers and marketers. 1Q.com is a panel, currently with a North American focus, that operates via consumer’s phones. Like a growing number of new survey and micro-survey options the 1Q.com platform is very DIY. You can log in, specify the sample, the questions you want to ask, pay with a credit card and launch a survey, all within a few minutes. The system also allows geo targeting. One feature of the platform is that it pushes clients, strongly, to use very short surveys in two ways: 1. The pricing model is per question, per participant. So, 1 question to 1000 people costs the same 2 questions to 500 people. This is likely to make people think hard about how many questions they need. Further questions can be asked in the future to people who take part in surveys, via the DIY platform – this can be done in a way that creates a virtual panel of your own. Market Research And Marketers? For a few years now I have been predicting that marketing and market research will become more […]
Conjoint Analysis is used when a brand wants to know how important different elements of a decision are. We know from neuroscience that people (people like you and me) cannot put numeric values to how important is, say, flying direct versus flying on your preferred airline. But brands need numeric values when they seek to maximise revenue, profit, customer choice and satisfaction. Consider an organisation producing tablets, perhaps a competitor to the Apple iPad and Samsung Galaxy. The organisation needs to understand how different customers value Attributes such as Size, Brand, Price, and Battery Length. Armed with this information they can create their product range and offering. Conjoint Analysis seeks to assign values to these product Attributes and Levels by creating realistic choices and asking people to evaluate them. Maths are then used to calculate what the underlying values are. Example In the case of Choice-Based Conjoint Analysis (currently the most popular form of Conjoint Analysis) participants are shown a series of options, like the one below, and asked to select the one they would be most likely to buy. Choice-Based Conjoint asks people to pick the option they would be most likely to buy, other forms of Conjoint Analysis […]
Here are six tips for using mobile market research. All six are drawn from the workshops I will be holding across the USA in October with Roddy Knowles and Maria Domoslawska from Research Now. To find out more about the workshops click here. You do not get to choose what device your research participants use. About one-third of them are already taking your online surveys via mobile and the number is increasing. Your role is to make sure the experience is good and the data you collect is right. That is why device agnostic is so important. In many cases mobile devices deliver the same results as PC-based interviews, but in some cases they don’t – for example multi-select grids deliver different results on PC and smartphone. You need to know what works and what doesn’t, and avoid the things that don’t provide comparable information. There is not a hardware or automated solution to making your surveys device agnostic. Good software helps, but intuitive design is critical. You need to design surveys with shorter questions, shorter answer lists, and in many cases shorter surveys. Location-based research is about more than GPS. Locating people via iBeacon, cell tower, or Wi-Fi are […]
Guest post by Jeffrey Henning. Jeffrey Henning, PRC, is president of Researchscape International. He is a Director at Large on the Marketing Research Institute International’s Board of Directors. You can follow him on Twitter @jhenning. Ray Poynter argues that the long survey will be replaced by ongoing discussions, in-the-moment research, and observational data. I think it is far more likely that the long survey will be replaced by another type of survey. This is much less of a change for corporate researchers and research agencies than shifting to radically different methodologies. Researchers today are very comfortable with developing long and complex surveys. And of course an entire industry exists providing access to panelists who will take surveys. Long surveys, however, produce lower quality results than shorter surveys, as Ray points out. Respondents tire, and begin satisficing, providing less accurate answers as survey length increases. Respondents who stick with a long survey often differ in key ways from the target demographic; this bias is exaggerated even further when researchers throw out incomplete responses. Respondents are increasingly taking surveys on mobile devices, where long surveys get even longer as compound questions such as matrix or grid questions get rendered as a series […]
For those of you who do not follow UK news, there was an election last week in the UK and the Conservative Party managed to squeak a small majority of the seats with 37% of the votes. This has caused a big fuss and makes the market research industry look very bad since the prediction (based on many, many polls) was for the votes to be split 34% to Labour and 34% to Conservatives – which would have left Labour as the largest party, and they probably would have formed a coalition or minority government. Some countries already ban polls in the run up to the election and there are broadly two arguments that people who want to ban polls put forward: Polls can encourage people to vote for a party or candidate who is not their first choice – especially in countries where the voting system is not proportional. There is also concern that polls encourage copying behaviour, as opposed to considered decision making. If the polls are wrong, people may vote for a party that is not their first choice on the basis of bad information, distorting the election result. The first argument about polls is a philosophical […]