The problem of survivor bias in marketing and market research

Imagine that we’re looking at the sales data for 1000 new products launched in the last two years in the snack food market. Let’s assume that 100 of them have been market successes, and 900 have to some extent failed. When we look at the 100 ‘winners’, we see that 80 of them are low fat and/or low calorie, and just 20 were in the ‘normal’ calorie/fat range. We might look at the results and jump to the assumption that low fat/calorie products have a better chance to win than standard products. However we might be wrong, and if we are wrong the reason would be that we have succumbed to survivor bias.

For example, if we looked more closely, we might see that of the 1000 new products, 900 were low fat and/or low calorie, and just 100 were in the normal range. This would show that a low calorie/fat product had a 1-in-9 chance of being a winner, but a regular product had a 1-in-5 chance of succeeding.

Looking at just the winners in order to predict what factors determine the results is prone to survivor bias (also known as survivorship bias). We are looking at the survivors, not the whole picture.

In marketing and market research there is often a temptation is to look at features that winners have in common and assume that they are the reason why things are winners. A famous example of this was Tom Peters book In Search of Excellence. Tom Peters reviewed several successful companies and organisations and published a book claiming that their strengths were those things that these companies had in common. However, within just a few years most of these organisations were in trouble. It turned out that the factors they had in common were not the drivers of success.

What does this mean for marketing and market research? It means that success and the drivers of success cannot be estimated by just looking at the winners. To understand drivers of success it is necessary to investigate the starting position as well as the ending position.

For example, we might notice that many of the successful YouTube videos feature cats. Does this mean that featuring cats makes it more likely that your video will be successful? Not necessarily, perhaps there are disproportionately more videos featuring cats uploaded? Perhaps the success of cat videos is simply in proportion to the number that were uploaded in the first place.

In marketing this means that we can’t just look at the successful messages, campaigns, and brands. We have to look at them in the context of all brands and all campaigns.

In market research this means we can’t just look at the reasons why people like the brand they buy, they may also have similar views about brands they don’t buy.

For more information on survivor bias see this Scientific American article by Michael Shermer.