Want to answer a research question? Start by finding out what is already known

OwlPost by Ray Poynter, 10 February 2020

Quite a lot of my teaching and consulting time focuses on how to design research that will answer questions and lead to action. One of the key steps that people need to tackle when designing a research project, and when seeking to find the story in the data, is to assess what is already known.

The topic of assessing what is already known is multifaceted and includes:

  1. When the answer is already known.
  2. When part of the answer is already known.
  3. When the questions (or some of the questions) are already known.
  4. What solutions have been tried and failed?
  5. When evaluative information is known (for example benchmarks).
  6. Which things that are known are wrong?

When the answer is already known

Most research agencies will have had the situation where a client contacts them asking for X, and you have to explain to them that you have already answered that question for one of their colleagues. However, this is not the only situation where the answer is already known, and here are a few more examples.

  • A report has been published that answers the question, for example an industry or government report.
  • A report can be run from the client’s dashboard to answer the question, accessing data from transactional data.
  • A careful re-phrasing of the question makes the answer clear. For example, when asked to test if a new product, Y, is going to be viable in the market, a researcher may well need to make the question better defined. Once the question is properly defined it may quickly become apparent that Y is not viable (perhaps for cost or technical reasons).

When part of the answer is already known

For all of the situations mentioned above, there are even more cases where what already exists is a partial answer to the question. Perhaps a report has been conducted on the total customer database, but there is not enough information to be able to drill down to the target group. In these cases, there are two key consequences of the knowledge:

  1. Ensuring that the difference between the partial answer and the complete answer is sufficiently valuable for the client to spend time and money on it.
  2. Using what is already known to design the right additional research. Perhaps the partial information is a qualitative segmentation, so what is needed is a quantification study. Perhaps what is known is that 10% of customers stop using the product after the first week, so what is needed is a more qualitative or ethnographic investigation.

What solutions have been tried and failed?

When you are asked to solve a problem, try to find out if there have already been other attempts to solve the problem. For example, if the client has already tried a test market approach, you are probably not going to suggest trying it again, but you will want to learn from what happened last time. Quite often, when time consuming and relatively expensive options are used to solve a problem (for example Conjoint Analysis or Ethnography) it is because simpler options have already been tried and failed.

When the questions (or some of the questions) are already known

In some business situations the questions are well understood. For example, “If we run this advert, how many people will buy our product?” When the questions are well understood and defined, the selection of a research approach and the design of that research is relatively straightforward.

However, if the questions are ill-defined, the research design and approach needs to be more iterative, i.e. more agile. If the questions and context are not well understood, then it is often a good idea to conduct qualitative research to help define the context and variables, or use social media listening to gain a deeper understanding of the topics and language used by customers and/or the general public.

When evaluative information is known

When designing research, it is important to know how the results are going to be evaluated. Perhaps the most common evaluation is against benchmarks, i.e. against the results of previous test. For example, ad tests and concept tests will often be compared to historic norms for key measurements.

Research may also be compared against other criteria, such as:

  • The plan for the product or service. For example, an ad may have been designed to work by triggering excitement, so the research will need to report back on whether it did.
  • Action standards. For example, company Z may have a policy that an ad is not used or a new product launched if it does not reach a particular score (and that score is the action standard).
  • Live data. If an A:B test has been used or if an MVP has been test marketed, there will be actual sales or usage data.

Which things that are known are wrong?

As Mark Twain may have said “It Ain’t What You Don’t Know That Gets You Into Trouble. It’s What You Know for Sure That Just Ain’t So”.

When designing research and when looking for the story in the data, try to check that the things that are ‘known’ are actually true. Try to assess the source of truth for your assumptions and the assumptions of those around you. Think about what you can do to give those assumptions the chance of being challenged and disproved. Sometimes this can be as simple as adding ‘None of the above’ to a list in a survey. Sometimes it might mean interviewing other samples. For example, if you have designed a product specifically for Millennials, you may want to test it against other groups to see if the Millennials actually have a different desire to buy/use it.

Other posts in this series include: