What is the scientific method, and how does it relate to insights and market research?

Posted by Ray Poynter, 22 April 2017

Scientific MethodI often hear people grumble that researchers, marketers and insights professionals have forgotten (or have never learned) the ‘scientific method’. However, there is usually very little discussion about what the scientific method is and how it should be applied. In this post, I am going to share a definition of the scientific method and discuss how it can be applied to the process of finding insights in commercial organisations.

A dictionary definition:
Here is a definition of the scientific method from the American Merriam Webster dictionary:

Principles and procedures for the systematic pursuit of knowledge involving the recognition and formulation of a problem, the collection of data through observation and experiment, and the formulation and testing of hypotheses.

The definition is a start, but it is not a road map for teaching or using the scientific method, so let’s map it out and then explore how to use it.

Scientific Method Flow Chart
The scientific method uses systematic processes to move from the need to solve a problem, via the creation of a hypothesis (or hypotheses), to testing the usefulness of the hypothesis. The flow chart below spells out the key steps in this journey.

Flow chart of scientific methos

1 Defining the Problem
The scientific method requires that a clear and well-defined problem be established at the start of the process. The nature of the problem depends on what you already know, and what you need to know. For example, you might be asking a question like “What is the best way to test new TV ads?” Or, you might be asking a question like “Are any of these three TV ads good enough to go to market with? And, if so, which should I choose?”

When defining the problem, it is a good idea to state the assumptions. For example, if you are going to test three ads using Millward Brown’s Link test you are making the assumption that the test works. A method is only scientific if you are building your new learning on elements that you have reason to believe are true.

2 Collect Relevant Data
The relevant data can include:

  1. Existing data, such as reports and institutional knowledge.
  2. Observational data, which could be ‘big data’, but it could also be ethnographic observations.
  3. Experimental data, which includes survey responses, passive data collected from experiments, and information gathered via discussions (for example via focus groups)

3 Formulate Hypotheses
Using the relevant data, the researcher looks to see if predictive patterns can be found in the information.

In this context a predictive pattern might be something like:

  • There is a link between trial and purchase.
  • There is a link between satisfaction and churn.
  • There is a link between market share and market penetration.

In market research we are interested in predictive patterns because the aim is to help companies make better business decisions through evidence-based decision making. The objective of (most) market research is NOT to explain what happened last time. The objective of market research is to help improve the next decision.

Starting with the hypothesis
In some cases, the scientific method starts with the hypothesis. For example, somebody may have published a method based on theory and you wish to evaluate it. Or, a method may have been established for consumer durables in North America and you want to test whether it works for financial services in Germany.

4 Test the Hypotheses
The key thing to understand about using the scientific method to work with hypotheses about humans is that nothing can ever be proved. Things can only be proved in abstract, artificial worlds – such as the realms of pure mathematics. In the domain of research about people, the testing of hypotheses is concerned with assessing their reliability and usefulness. The more evidence we gather that supports the hypothesis, the more confidence we will have in its future predictions.

There are some key rules about testing hypotheses that should be followed:

  1. The data used to create the hypothesis should not be used to test the hypothesis. A high correlation between a model and the data used to create the model shows consistency, but tells us nothing about whether it is likely to be true in the future or in other situations.
  2. The results of the testing are confined to the situation it was tested in. For example, if we test a method of predicting sales of fruit juice in South Africa. We have gathered good evidence for fruit juice purchasing in South Africa, a little bit of evidence about fruit juice, a little bit of evidence about South African purchasing, and not much else.
  3. Tests should be designed so they can disprove the hypothesis. If the test is designed in such a way that it can’t disprove the hypothesis it is useless.
  4. Although we can never prove a hypothesis, we can become increasingly confident in it if we test it in a variety of ways, with a variety of data sources and if it proves positive in all cases.

The testing of a hypothesis should attempt to assess the reliability of the predictions. For example, if a new product test is shown to be accurate in 70% of cases, that does not mean it has failed. It means the hypothesis should be re-worded to say it will be right 70% of the time.

Similarly, the testing of a hypothesis might show that in 90% of cases, market penetration accounted for 90% of market share. This result would not disprove the hypothesis that market penetration determined market share, it would simply define the limits of its usefulness and reliability.

A good hypothesis should generate predictions that can be tested, and could potentially prove the hypothesis to be untrue. Indeed, many people would consider somebody who generated untestable hypotheses as being little better than a charlatan.

What About Actionable Results and Storytelling?
I can imagine that some people reading this post will be saying, something like “Well that all seems very dry, I thought modern insights was all about actionable results and storytelling?” However, the scientific method is something that precedes the storytelling and the actionable outcomes.

Think of the scientific method as being like the foundations of a new house. The storytelling is like the design of the house; it will determine how happily you live in the house. The scientific method will determine if the house falls down or not.

9 thoughts on “What is the scientific method, and how does it relate to insights and market research?

  1. This is fantastic – I am teaching a STEM session on this very topic to 75 2nd grades, including my twins. May I use it?

  2. Scientific methods are and should be the core of the MR professional’s being. The reason for (client)-company leaders to order insights, answers, explanations etc is the trust related to the scientific background of the profession. Insight creation has to comply with certain scientific rules, as you clearly described. Story telling, speed, action-ability, efficiency, etc. are other very relevant criteria. We should strive to make these an intrinsic part of our profession as well, but it will not be the core of our profession. During the 70th ESOMAR Congress in Amsterdam we will discuss the future of Market research and it is likely in the future market research departments and companies are contractors in sourcing all kind of the different specialization. Story telling, project management, programming, hosting and even survey design including applying scientific rules can be in-sourced. That is just a matter of industrial organization, but the scientific methods and rigour should be an intrinsic part of the deliverable.

  3. As a Consumer Behavior Tendancy Seeker, I was really attracted by the title of your article. But going into the details of it, I got upset because this is, again, a debate about how to stay “professional” credible with existing methods we know for years. What changed is the technical support we are using….

    I think, if we really want to understand consumers in their “natural” social consuming field, we have to ask ourselves, how we can develop different MR Methods that can be considered as reliable, valid and very useful for future decision makers.
    For the moment, I only see opportunities in the qualitative field, solid theoretical frameworks have to be contructed, tested and valided within MR Communities. The potential is so high to be innovative in this field, business opportunities are getting more and more interesting… AND, consumers merit our “real field” Consumer Behavior attention!

  4. I would rather disdagree regarding the order of the key steps: The scientific method is hyothesis-driven, thus step 2 and 3 should be changed. Based on your defined problem you normally create hypothesises and then collect your data to test your hypothesis. Otherwise you can only create a hypothesis based on your collected date which doesn’t really make sense since it limits your hypothesis-options in an inadequate way.

  5. Most day to day market research is merely deductive by nature, meaning that it starts with a management question and a theory (although in most cases not clearly stated as such). Inductive research to develop a theory or model is an exception. For deductive research Stefan is right that the order is opposite.
    The past decade focus in market research has been changed from rigour to relevance Insights instead of plain information). Rigour in the meaning of reliable and valid information. The focus on relevance or even further storytelling (explaining the relevance in an attractive way) has led us away from the need for rigour. A research result can only be relevant if it meets the demands of reliability and validity.
    And yes I agree with Alexandra that we should focus on new (qualitative) techniques to understand the behaviour of consumers better as long as these new approaches will be tested on reliability and validity. This should be the core of the ‘scientific method’.

Comments are closed.