Photo of Ray Poynter trail running

Does Running Damage Your Heart? Another example of the problems of using observational data to infer cause and effect.

Back in 2012, there were several research papers that suggested that running for more than about 20 miles a week was either not giving any benefit, or, worse still, it could be damaging hearts and making death more likely. The key sources were a piece of research by Dr Duck-chul Lee with 50,000 patients (presented at the American College of Sports Medicine (ACSM) 59th Annual Meeting 2012, see The Not-So-Long Run: Mortality Benefit of Running Less Than 20 Miles per Week) and heart findings from Dr James O’Keefe that looked at issues such as fibrosis, calcified arteries, and arrhythmias. This was picked up by a joyful media, with stories about how running was bad for you, and that anything other than a small amount of exercise was either useless or damaging. However, as is often the case with observational data (as opposed to control and test experiments), there were several problems with the conclusions. The key problem with Lee’s study was identified in 2013 by Dr Thomas Weber. In the sample of 50,000 people there were some long distance runners, for example, marathon runners. Lee wanted to assess these people against the non-runners and the occasional runners. However, he needed […]

Image of digits

How many significant digits should you display in your presentation?

Posted by Ray Poynter, 27 August 2018 Too many digits can obscure the story being communicated by numbers. Let’s consider a simple example from a trip to your gym and its hi-tech weighing machine. Perhaps the machine says that your weight is 101.7865 kilograms and that it should be 82 kilograms. The story is that you are about 20 kilograms too heavy. To see the story you need to focus on comparing 102 kilograms with 82, not 101.7865 with 82. If your presentation or report displays too many digits you will obscure the story you are trying to communicate. The choice about the right number of digits to display is the choice about how many significant digits to display – the topic of this post. Digits Obscure – Example 1 Consider the table below, which is extracted from the ITU (International Telecommunication Union) and shows how many mobile phones there were in each of the countries displayed, from 2010 to 2017, per 100 people. If you click on the data tables they get bigger. The data shows four decimal places and is not very easy for most humans to quickly review. This data is not friendly for the analyst looking […]

Hypothesis

Is it time for market research to ditch the ritual of the Null Hypothesis?

From time to time, I am asked to write some notes (or teach a section) on hypothesis testing. Each time I do this, I am reminded how little the theory of hypothesis testing has to do with modern, commercial market research. Perhaps we should stop focusing on a theory that does not really apply, and talk about what we actually do? At its simplest, the hypothesis process is as follows: Decide we want to show X is correct Design a situation ‘Not X’ and collect data to investigate ‘Not X’ Show that ‘Not X’ is very unlikely Assume X is right. This is highly unnatural for most people. People want to focus on X, not show it as a by-product of something completely different. This method is loosely what is done in academia, but almost never in the commercial world. Consider an example from concept testing Assume we are testing three new concepts and the forecast market share values are 5%, 6%, and 12%. What do we really want to know? On most occasions, I think we would like to know whether we should choose the concept with the 12% score. For example, is it genuinely better than the 5% […]

Stormy sky

When will surveys disappear? (2017 update)

Back in 2010 ,I caused a minor stir in the research world by predicting (at the MRS Conference in London) that surveys would have disappeared in 20 years (i.e. by 2030). This prediction was put into wider circulation when I clarified my prediction in a blog. The key point being that I was predicting the end of the commercial, long survey, and it being replaced with social media listening, online communities, new ways of researching, the use of open-ended questions, and the use of stored information to remove the need to keep asking questions. In 2014 I updated my prediction and showed some numbers from the ESOMAR Global Market Research Report. The table below shows the figures from ESOMAR for 2007, 2010 and 2013, and my projections for 2016 and 2019. Note the figures show the spend on research, not the volume. (Click on the tables to enlarge them.) So, how did my predictions stand up? The table below shows the ESOMAR figures for 2016, below my estimates. Note, I have added a new column which combines Other Quant (e.g. traffic and audience data) with Other (e.g. big analytics). In the future I will focus on Surveys, Qual, and a single […]

Statistics

The Statistics and Statistical Tools Used in Market Research in 2017

In this post I am sharing the summary and two key charts. The eight-page version of the results can be downloaded. Summary The top four things that I want to share about the use of statistics and statistical tools are: Most statistical tests/approaches are not widely used. Only Correlation, Regression, z- or t-tests, and Cluster Analysis have been used by more than 50% of the participants in this research, during the first half of 2017 – and this sample probably over-represents people using statistics, and under-represents those using statistics less often. SPSS is the dominant software package amongst people using statistical packages. Given SPSS is approaching 50 years old, that may not be the sign of a dynamic industry? But, there are many people using tools such as Q, Sawtooth Software, SAS – and beyond them programs such as Latent Gold, Tableau, and XLSTAT. One of the growth areas is the use of tools is the use of integrated data collection / analysis solutions, for example Confirmit, Askia, Vision Critical, Qualtrics. The use of these tools requires the researcher to make fewer decisions. For example, survey monitoring flows into the analysis without any extra steps, the packages have a default […]

Somebody listening

Survey about the stats we use in market research

At the moment (August 16 to August 31, 2017) NewMR is running a survey to collect data about the stats commonly used in market research. [Note, since August has finished, the survey is now closed. We will be posting the results soon. However, you can still see the raw data report below.]  If you have not already taken the survey, please do so [by clicking this link] before reading the data below. The background to this survey is that we are writing some materials for two university courses and for workshops that we are running. We would like a clear idea of which stats are commonly being used, and which are more specialist. Stats that are commonly used need to be taught in a way they convey how to use them as well as when to use them and how to interpret them. Our feeling is that stats that are more specialist should (in the context of the courses and workshops we are involved in) be focused on when to use them and how to find out how to use and interpret them. Below is an automated report of what the data looks like at the moment (you might want […]

Image of Survey

Should Market Research Still Be Using Significance Testing?

Over the last few years there have been many calls for market researchers to stop using significance testing based on assumptions of random probability testing to measure the potential impact of sampling error. For example, Annie Pettit writing in The Huffington Post asked “Stop Asking for Margin of Error in Polling Research”. But, despite the concerns about the correctness of using this technique, it seems to still be in common use. In this post, I briefly explain what significance testing is (experts can jump this bit), why it doesn’t do what people seem to think it should do, and the way I think we should be using it in the future. What Is Significance Testing? The type of testing I am talking about in this post relates to sampling error. In quantitative research, a sample is taken from a population and one or more statistics are calculated. These statistics are then used to estimate the values for the total population. For example, assume 1000 people are selected at random from a population of 20 million. Assume that 50% of the sample are female. The inference from this study is that it would be expected that 50% of the total population […]