The problem of survivor bias in marketing and market research

Imagine that we’re looking at the sales data for 1000 new products launched in the last two years in the snack food market. Let’s assume that 100 of them have been market successes, and 900 have to some extent failed. When we look at the 100 ‘winners’, we see that 80 of them are low fat and/or low calorie, and just 20 were in the ‘normal’ calorie/fat range. We might look at the results and jump to the assumption that low fat/calorie products have a better chance to win than standard products. However we might be wrong, and if we are wrong the reason would be that we have succumbed to survivor bias. For example, if we looked more closely, we might see that of the 1000 new products, 900 were low fat and/or low calorie, and just 100 were in the normal range. This would show that a low calorie/fat product had a 1-in-9 chance of being a winner, but a regular product had a 1-in-5 chance of succeeding. Looking at just the winners in order to predict what factors determine the results is prone to survivor bias (also known as survivorship bias). We are looking at the survivors, […]