How many new products should fail?

trash can / rubbish binPosted by Ray Poynter
5 May 2021

Ten years, ago I posted on this topic and I thought it would be good to re-post the original article, with a couple of tweaks (you can read the original post here).

Back in May 2011, in the Market Research Bulletin LinkedIn group, there was a great discussion going on about why 80% of new products fail. The conversation was started by Wasim Shaikh, the topic had attracted a large number of comments, including the suggestion that perhaps 85% failure was a more accurate figure in 2011, rather than 80%.

In Wasim’s discussion I posted a short version of my view that the main reason for the high failure rate is that there is a limit to the absolute number of products that can be successful in any given year, and the number launched greatly exceeds this natural limit – so failure is mandated by the system. However, LinkedIn does not allow space to fully explore the argument, so I wrote this post as my response to Wasim’s question about why so many products fail and ‘What is going wrong’.

Compared with the query from Wasim, I’d like to address the question from the other end, so to speak, namely “How many new products should fail?” Further, I want to visit this revised question in two ways, by considering both probabilities and numbers of winners.

If we were to aim for 100% of new product launches to be successful, we would need to throw away products that we thought had a 99% chance of being successful. In order to get rid of Type I errors (false positives, i.e. launching products that fail) we would have a massive number of Type II errors (false negatives, i.e. not launching products that would have succeeded). If brands only launch products where it was felt there was a 100% chance of being successful, they might have to wait several years between launches and would find their market position eroded as other, less fussy, competitors launched their new products (i.e. each with less than 100% chance of being successful). For example, a competitor might launch 8 products, each of which had an independent chance of being successful of 25%. The company would know that by launching 8 products, each of which had a 25% chance of being successful, they would have a 90% chance of one of them being successful.

The probability of success of a new launch relates to many things, including any errors in the testing, any weaknesses in the marketing, any negative post-purchase experiences, and a wide variety of ‘events’ (such as weather, news, fashion, inflation, wars etc).

However, I think a much bigger issue, in the question of how many products should be successful, is a simple numbers game. All the neuroscience and behavioural economics stuff is showing that we humans tend not to think too much, we utilise heuristics, and we find shortcuts. Becoming a regular buyer of a new product carries a cognitive burden, we have to remove the old habit and install a new habit. For some products, perhaps a subtle line extension in a chocolate bar, the load is pretty small. For other new products (for example a novel way of applying dental floss) the cognitive load is higher.

If we stick to FMCG/CPG (fast-moving consumer goods/consumer packaged goods). Each consumer is, IMHO, only going to become a regular buyer of a few new products each year, perhaps ranging from single figures for creatures of habit, and up to 30-40 for people who embrace the latest coffee, sandwich spread, or hard surface cleaner, etc. Think about how many new products you have become a regular user of in the last 12 months. A brand has to jump through the following hoops to be successful:

  • It has to be one of the ones that the major retailers agree to take and agree to place it in a good position, and usually it needs to agree to it being promoted in stores – stores will only carry a certain number of fixed products each year.
  • A sufficiently large number of consumers need to put it into their repertoire. This will usually mean removing something else from their repertoire, either because the new product is a direct replacement, or simply to keep their overall budget constant (new products that result in people increasing their weekly grocery spend are even rarer).
  • Then the new product needs to embed itself in the pattern of purchase for that household.

As an example of the large numbers of products being launched and failing, consider the 1997 report by Linton, Matysiak & Wilkes which looked at 1935 new products from just the top 20 food companies in the US. The study found a failure rate then of 70-80%, which might imply that 30% of the new products (in that case just under 600) are capable of surviving, but far fewer will be ‘successful’. Since then the number of new products has increased, but the number of winners has remained relatively fixed.

I am still gathering the data, but in the context of a large supermarket, the number of new products that can be launched and succeed in a given year is probably relatively fixed. It may be that the maximum number of successful new products in a large supermarket is perhaps a few hundred, perhaps 1000, but it is relatively fixed, and it is much smaller than the number of products companies would like to launch. New product launches in FMCG/CPG tend to run to large multiples of thousands each year.

If we take the hypothetical view that in a given year, let’s say 2012 (or 2022), that the maximum number of products that can be successful in major supermarkets in a given country is, say, 1000. This probably means that something like 500 products will have to be delisted by the supermarket to make space, allowing for some growth in the size of stores and some narrowing of facings. The percentage of products that will be successful is going to be 1000/N, where N is the number of products launched. As N increases the percentage of products failing increases, almost irrespective of how good they are. In our hypothetical market, if 1000 were to be the limit for success and 5,000 were to be launched, then the failure rate would be 80%. If 1000 is the number of possible successful products and 10,000 new products were launched, then the failure rate would be 90%, regardless of how good the research, products, and marketing were.

There are probably things that change the maximum number of successes from time to time, a recession might make it smaller, more leisure time (but probably not unemployment) might increase it, innovations such as refrigeration in stores and then refrigeration in the home no doubt changed the number, but changes in the maximum are largely beyond the control of individual brands.

As an analogy, consider Formula 1 racing, and think about its first year, 1950, and its most recent year 2010. In 1950 there were 7 races, Nino Farina was the overall winner, but it was possible for 7 drivers to have least been reasonably successful, i.e. winning at least one race. In fact, in 1950 just three drivers won at least one race. In 2010 there were 19 races, with Sebastian Vettel the overall winner, so just like the growth in the size of supermarkets between 1950 and 2010 the limit for how many winners there could have been had increased, to an upper potential of 19. However, just five drivers won at least one race in 2010. [In the 2020 season, there were 17 races, so there could have been 17 winners, but there were just 5 drivers who won at least one race.] Supermarkets, and beyond that the wider marketplace, is bigger than Formula 1, but the same principle applies, the number of winners is not greatly affected by the number of entrants, and the maximum is set by the system.

So, what should researchers and more importantly brands do? Well, they can’t simply say we can’t launch too many products, because we know only some will be successful. The competitors will simply niche their share away. Brands need to use their NPD, their market research, their marketing skills, and their advertising to try and beat the market, i.e. to increase the chance of a new product being successful. If 85% of all new products in your category fail, but only 82% of your products fail, you are doing well, and the more you can improve that ratio the better. Because a) you are winning, and b) competitors are losing. Other factors that come into play are low-cost innovation and systems such as ‘fail and learn’, and agile marketing (where the process of failing is used to help improve the product).

Lots of products SHOULD FAIL
It is not possible to develop a pre-testing system that will allow you to routinely launch products that succeed. If 85% of products fail, then if every company used the same, best pre-testing system, there would still be 85% failure – unless companies massively reduced the number of products launched. A good pre-testing system can enable you to beat the opposition (assuming the opposition don’t also use it and assuming your products and marketing are good enough), but there are systemic reasons why failure rates will exceed success rates.

PS, we should also be careful about describing a launch as a failure without ensuring we know why a product was launched. Some products are only intended to be on the market for a short time, e.g. a new flavour drink might be intended to last just one summer. Its role is to create excitement around the brand and keep the messaging fresh. A new line extension for an ibuprofen product might be launched to increase the brands share of displays and to keep a competitor out of distribution – rather than generate substantial sales. This means that when we categorise a product as a failure, we need to take into account why it was launched and whats its targets were.