What is the scientific method, and how does it relate to insights and market research?

Posted by Ray Poynter, 22 April 2017

---

I often hear people grumble that researchers, marketers and insights professionals have forgotten (or have never learned) the ‘scientific method’. However, there is usually very little discussion about what the scientific method is and how it should be applied. In this post, I am going to share a definition of the scientific method and discuss how it can be applied to the process of finding insights in commercial organisations.

A dictionary definition:
Here is a definition of the scientific method from the American Merriam Webster dictionary:

“Principles and procedures for the systematic pursuit of knowledge involving the recognition and formulation of a problem, the collection of data through observation and experiment, and the formulation and testing of hypotheses.”

The definition is a start, but it is not a road map for teaching or using the scientific method, so let’s map it out and then explore how to use it.

Scientific Method Flow Chart
The scientific method uses systematic processes to move from the need to solve a problem, via the creation of a hypothesis (or hypotheses), to testing the usefulness of the hypothesis. The flow chart below spells out the key steps in this journey.

1 Defining the Problem
The scientific method requires that a clear and well-defined problem be established at the start of the process. The nature of the problem depends on what you already know, and what you need to know. For example, you might be asking a question like “What is the best way to test new TV ads?” Or, you might be asking a question like “Are any of these three TV ads good enough to go to market with? And, if so, which should I choose?”

When defining the problem, it is a good idea to state the assumptions. For example, if you are going to test three ads using Millward Brown’s Link test you are making the assumption that the test works. A method is only scientific if you are building your new learning on elements that you have reason to believe are true.

2 Collect Relevant Data
The relevant data can include:

1. Existing data, such as reports and institutional knowledge.
2. Observational data, which could be ‘big data’, but it could also be ethnographic observations.
3. Experimental data, which includes survey responses, passive data collected from experiments, and information gathered via discussions (for example via focus groups)

3 Formulate Hypotheses
Using the relevant data, the researcher looks to see if predictive patterns can be found in the information.

In this context a predictive pattern might be something like:

• There is a link between trial and purchase.
• There is a link between satisfaction and churn.
• There is a link between market share and market penetration.

In market research we are interested in predictive patterns because the aim is to help companies make better business decisions through evidence-based decision making. The objective of (most) market research is NOT to explain what happened last time. The objective of market research is to help improve the next decision.

Starting with the hypothesis
In some cases, the scientific method starts with the hypothesis. For example, somebody may have published a method based on theory and you wish to evaluate it. Or, a method may have been established for consumer durables in North America and you want to test whether it works for financial services in Germany.

4 Test the Hypotheses
The key thing to understand about using the scientific method to work with hypotheses about humans is that nothing can ever be proved. Things can only be proved in abstract, artificial worlds – such as the realms of pure mathematics. In the domain of research about people, the testing of hypotheses is concerned with assessing their reliability and usefulness. The more evidence we gather that supports the hypothesis, the more confidence we will have in its future predictions.

There are some key rules about testing hypotheses that should be followed:

1. The data used to create the hypothesis should not be used to test the hypothesis. A high correlation between a model and the data used to create the model shows consistency, but tells us nothing about whether it is likely to be true in the future or in other situations.
2. The results of the testing are confined to the situation it was tested in. For example, if we test a method of predicting sales of fruit juice in South Africa. We have gathered good evidence for fruit juice purchasing in South Africa, a little bit of evidence about fruit juice, a little bit of evidence about South African purchasing, and not much else.
3. Tests should be designed so they can disprove the hypothesis. If the test is designed in such a way that it can’t disprove the hypothesis it is useless.
4. Although we can never prove a hypothesis, we can become increasingly confident in it if we test it in a variety of ways, with a variety of data sources and if it proves positive in all cases.

The testing of a hypothesis should attempt to assess the reliability of the predictions. For example, if a new product test is shown to be accurate in 70% of cases, that does not mean it has failed. It means the hypothesis should be re-worded to say it will be right 70% of the time.

Similarly, the testing of a hypothesis might show that in 90% of cases, market penetration accounted for 90% of market share. This result would not disprove the hypothesis that market penetration determined market share, it would simply define the limits of its usefulness and reliability.

A good hypothesis should generate predictions that can be tested, and could potentially prove the hypothesis to be untrue. Indeed, many people would consider somebody who generated untestable hypotheses as being little better than a charlatan.

What About Actionable Results and Storytelling?
I can imagine that some people reading this post will be saying, something like “Well that all seems very dry, I thought modern insights was all about actionable results and storytelling?” However, the scientific method is something that precedes the storytelling and the actionable outcomes.

Think of the scientific method as being like the foundations of a new house. The storytelling is like the design of the house; it will determine how happily you live in the house. The scientific method will determine if the house falls down or not.