Most Predictions Are Wrong, So Let’s Keep Making Them

Future Decisions image
Posted by Ray Poynter, 11 January 2017

The researcher and author Philip Tetlock has been researching predictions since the 1980s and has concluded that most predictions fail, that most experts do worse than a chimp with pin (i.e. worse than chance), and that predictions more than a few months out are particularly flaky. If you are interested in the topic of predicting and forecasting then I strongly recommend his book Superforecasting: The Art and Science of Prediction, which illustrates the problems and shows why and how some people do better than experts.

But despite the dismal record of most forecasters and predictors, we continue to make predictions, especially at this time of the year. Indeed, I recently published my “Five Market Research Trends for 2017” and was pleased to see that over 4000 people had read it.

Why Predictions Can Be Useful
I firmly believe that making predictions, and urging other people to make predictions, can be a really useful exercise, despite the poor track record that predictions have. In order to assess the ways predictions can be useful, we need to separate the different categories and motivations for forecasts and predictions. So, here are seven categories/motivations – and the benefits they can bring.

1 The Planner
Plans, for example business plans, essentially consist of three elements:

  1. A prediction of what is going to happen, for example demand for X will increase, the price of Y will decrease etc.
  2. An aspiration of where we want to be, we want to grow the volume and gross margin of our business by focusing on X.
  3. A set of actions that we think will help us achieve our plan.

A typical plan will include targets, for example targets for sales by month, for new offices opened, and for new staff recruited. These targets are a prediction of what might be possible. However, they are not a passive prediction, they are targets that need to be worked at in order to achieve them. They are as much about creating the future as predicting the future.

During the year the business will measure its performance and compare it with its plan. Three things may happen:

  1. The organisation may be meeting its targets, in which case the plan will (all other things being equal) stay as it is.
  2. The organisation may be exceeding its targets. In this case the organisation needs to assess why this is happening and normally will want to modify its plan, to take advantage of the situation.
  3. Quite often the targets will not be met. In this case the organisation, if it is wise, will investigate why and develop a Plan B. (Sometimes, Plan B is just Plan A harder – but that can often lead to trouble).

In the case of the business plan the predictions and the planning mean that the organisation can work together to achieve shared goals. By comparing outcomes with targets/predictions the organisation has a reference point.

The alternative to having a business plan is to simply react to changes in the market, a strategy that is rarely successful, and very is often very stressful.

2 The Evangelist
The evangelist believes in something and will tend to make forecasts consistent with their beliefs. Examples of these people are the people promoting Internet of Things, Paleo Diets, Neuromarketing, Bitcoin revolutions, the sharing economy and so forth.

According to Tetlock, these forecasters are amongst the least accurate. The more emotionally attached we are to a topic, the less good we are at forecasting it.

For the rest of us these forecasts have two useful elements. Although this group are less accurate than a chimp with a pin, they will sometimes be right. Their descriptions of what will happen can create a set of early-warning indicators that help us see that something is a change not a fad. The early promoters of the Internet, Social Media, and Smartphones were all evangelists – but they turned out to be right.

The evangelists can also serve a role in making change happen. Part of the predicting process can be a conscious or unconscious effort to make a change happen. When Naomi Klein wrote her book No Logo, it was partly her prediction of a less commercial world, but it was also an attempt to move the world in that direction.

3 The Commentariat
The commentariat is a slightly pejorative description of people whose job it is to make comments and forecasts, particularly via the media. This is another group who Tetlock says are particularly poor at making accurate predictions. IMO, the key reason these people are so poor at making correct predictions is that they have to create entertainment value. The two forecasts that are most likely to be true are:

  1. No change
  2. A change that is currently happening and that everybody else is forecasting.

The professional opinion-vendor cannot afford to offer either of these forecasts very often, which means they are fishing in a pool of less probable outcomes.

The value to the rest of us of these predictions and forecasts is that it helps shape the debate about a topic and it indicates areas of thinking that we should understand. In terms of changes, the number of things we don’t know is close to infinite, so it is useful to have people highlighting lines of inquiry that we might find helpful, for example in terms of increasing our understanding of a topic.

4 The Lobbyist
Of all the predictors, Tetlock described these as the least accurate. In some ways the lobbyist is like the evangelist, but the key difference tends to be that the lobbyist is focused on creating an outcome, not on being accurate. When the spokesperson for a political party says that the unpopularity of their party will soon end and that voters will come back to them, they are trying to influence that outcome, whether or not they believe it. The forecasts issued by a company when it launches on the market with its IPO are intended to create success, not simply predict it.

Indeed for a lobbyist, the likelihood of something happening is a restriction not a goal. The claims made cannot be so crazy as to make the message ineffective. For example, a company selling programmatic advertising cannot credibly say that it will replace all other approaches, so the claim may be scaled back to something like ‘massive growth’ or ‘all leading brands will use it’.

5 The Science Fiction Writer
Science fiction writers do not set out (usually) to accurately predict the future. Although in many cases that is exactly what they have done: e.g. Jules Verne predicted the journey to the moon in 1865, Aldous Huxely’s Brave New World predicted antidepressants and many of today’s ills in 1931, and in 1948 George Orwell’s 1984 predicted the surveillance state.

Science fiction often takes one or more changes and then explores how that would shape society. This process can be a useful way to try to innovate new ideas and to help think about how new ideas could be integrated into modern life. This use of predictions is a combination of helping with ideation and helping know how best to communicate a possible future.

6 The Scenarioist
Many futurists do not try to accurately predict a specific future, they are well aware that they are unlikely to be correct. Rather than one specific future the scenario planner creates alternative futures and then looks to:

  1. Create early warning indicators to help determine which scenario seems to be happening.
  2. Develop strategies that are applicable to several scenarios – reducing the risk of future shocks.

Scenario planning is much less sexy than predicting a specific future, because the scenerioist is rarely exactly right, but they are rarely completely wrong. A good introduction to scenario planning is David Schwartz’s The Art of the Long View.

7 The Data Scientist
In recent years there has been a massive growth in the use of predictive analytics. These are statistical processes used to forecast the outcomes of new campaigns, election results, and more complex things such as the weather.

A great introduction to the use of data for making predictions is Nate Silver’s book The Signal and the Noise, where he explains why baseball and the weather are relatively predictable, and why soccer and earthquakes are not.

The key thing to remember about most predictive analytics is that they assume the future is a complex collection of straight-line changes from the present. As Nassim Taleb highlights in his book The Black Swan, this is not always true. My personal prediction is that predictive analytics will be really useful in the future, but not in the next 5 years.

Predicting Project Outcomes
Before closing this post I want to share one of my key uses of predictions. My core business is helping organisations conduct research, either to help them understand current markets and customers, or help predict what would happen if they took a specific course of action (such as increase prices, run a new ad campaign, or launch a new product).

At the scoping stage of the project I usually ask everyone to make predictions about what they think the outcome will be, and I make predictions myself. The reason I do this is to understand where people are coming from, what their expectations are, and to ensure that the research I do will either:

  1. Confirm their beliefs.
  2. Challenge their beliefs.

When the project is complete, comparing the predictions to the outcomes can provide great insight into what we know and what we don’t know. Insight into things that seem to be working and things that need to be challenged and changed.

The Case for Predicting
So, I hope you will agree that even if most predictions are inaccurate, there is merit in making them. The process of predicting the future requires us to examine the present, it enables us to think about what we want the future to be, and can help us be quicker at interpreting change when it happens.

To quote the American general and President Dwight Eisenhower “I have always found the plans to be useless, but planning is indispensable”.