What is the Counterfactual? – Why do we need to assess it?

Posted by Ray Poynter 19 October 2018

If we are told that a before a marketing campaign the sales were at 10,000 units a month, and after the campaign sales are 20,000 units a month, then it is easy to assume that the campaign has increased sales by 10,000 units a month, or by 100%. However, this is an example of the classic fallacy “Post hoc ergo propter hoc” (which is Latin for ‘after this, therefore because of this’.

Consider the chart below, an edited and anonymised version of a presentation I saw at a conference last year.

Chart showing ice cream sales

The chart was presented by the head of social media to the head of insight in a company selling ice cream. The head of social media protested that the spend on social media advertising should be maintained at a high level. He pointed out that when social media advertising was increased sales, increased, and when the advertising budget was cut, the sales dropped. Therefore, the advertising budget should be increased, so that the advertising could be maintained at the higher level.

However, the head of insight took out her pen and scribbled onto the chart three words and images, as in the chart below.
Chart showing seasons
The point she made was that the sales of ice cream were being driven by the seasons, which in turn was also the likely driver of the social media advertising programme. What the social media marketer needed to do was consider the counterfactual, what would have happened without the campaign. The chart below shows the counterfactual in this ice cream case.
Chart showing Counterfactual
The lesson from this post is that whenever we assess the impact of an action, for example a marketing campaign, we need to consider the counterfactual, what would have happened without the campaign. The best way of looking at the counterfactual is to construct an experiment with control and test cells – the control cell is the counterfactual. When a test and control experiment is not available, other options include, post-event matching (where people who were exposed to the campaign are matched with people who were not exposed), data from earlier periods (e.g. the previous year), econometric modelling, and predictions (e.g. sales forecasts, targets, plans.)

3 thoughts on “What is the Counterfactual? – Why do we need to assess it?”

  1. Thank you for the very stimulating point you make. It shows how careful we maybe in making anycase from the data.
    May I ask a follow-up, how to attribute how much was increased in sales of either cause; i.e. how much due to summer and how much more, if any due to ads? Assuming the hypothesis that both contributed.

  3. I assume perhaps the control in this case was historical data. Accepting of course it was a modest contribution by ads; following and in full agreement with your article’s point; maybe also other factors have been in play: e.g. major competitive counter initiatives, or shifts in consumer attributable purchasing behaviour, that led to the only modest increase; Which goes back to your very valid and illuminating illustration about causal factors and how controls need to be in place, to isolate the causation. Thank you again for another thought provoking valuable piece.

Comments are closed.