Notes for a non-researcher conducting qualitative research

In November I am presenting a paper to the ESOMAR Conference on Qualitative Research, in Valencia in Spain. My paper suggests that one threat to qualitative research is the potential for damage caused by people with no training in qualitative research using one of the many DIY tools that are appearing – especially those for online discussions and instant chats.

My suggestion is to create a simple set of notes that will help put newcomers to our world on the right path. Below is my initial draft if of my notes, and I would really appreciate your feedback.

---

The Playbook

The playbook needs to be short, relevant, and easy to use if it is going to be of value to people looking to conduct their own research. Therefore, this initial draft covers the following topics:

• Evidence, not answers
• Creating a narrative
• Analysis begins at the start not the end of the project
• Creating a discussion guide
• Not everything that matters can be counted
• Data does not mean numbers
• Consider actors and agendas
• We are poor witnesses to our own motivations
• Memoing
• Enabling the participants whenever possible
• Grounding the story in the data
• Examples that inform, not ones that entertain
• The “But, I already knew that test!”

Evidence, not answers
Qualitative research, for example, online discussions, real-time chat, smartphone ethnography, or discussions gathered from social media, does not provide categorical, definitive answers. Qualitative research provides evidence, and the researcher has to interpret this evidence to produce the product of the research.

A quantitative study might discover that 10% of the population buy a product from ACME Corporation. This 10% is an answer, something discovered and provided by the research. A qualitative discussion might suggest that people seem willing to use words like respect, admire, trust about ACME, but were less willing to say love, like, associate with ACME. The researcher has to determine what that might mean and what the implications for ACME might be.

At the end of a qualitative project we can’t say things like “50% of the participants said they would try the product”, implying that 50% of the target group will buy the product. The qualitative participants are not numerous enough to forecast population-wide behaviour and the way the questions were asked will have affected the thinking and responses of the participants. A qualitative finding is more likely to describe what the people who said they would try it liked about the product, how they came to their decision that they might try it, and what was inhibiting those who did not want to try it.

A quantitative ad test might try to forecast how many people would recall it, how many would recommend the product, and how many would buy the product. A qualitative ad test tries to find out how the ad worked and to suggest how it might be improved.

Creating a narrative
The purpose of a qualitative market research project is to create a story that illuminates the topic under investigation. Qualitative researchers do not ‘discover’ the story, they create the story from what they find, potentially co-creating it with the participants and/or the client. The evidence they gather, the knowledge they have, the knowledge the client has, need to be woven together to produce the final narrative.

The narrative that is created needs to explain the evidence in a way that throws light on the subject so that it facilitates better business decisions. Qualitative researchers are aware that there is no one ‘correct’ story, there are usually many ways to tell a good/effective/useful story (and of course even more ways to tell it in ineffective or misleading ways).

Analysis begins at the start not the end of the project
Before conducting the research the researcher needs to think about what is already known, what needs to be known, and the sorts of evidence that will help create a narrative. During the data gathering phase, the text (e.g. the chat, the posts, the comments) should be reviewed to challenge the hypotheses the researcher already has and to help create new hypotheses. The researcher should seek to test hypotheses by posting questions, by assigning tasks, and by probing existing answers, in ways that will make or break the hypotheses.

For example, if the researcher feels that the participants do not trust a specific brand, the participants might be asked to write a list of all the things they like about that brand. The words that are not on the list are a clue to what people feel. The words not on the list can then be used to elicit which brands do have those characteristics.</P

Create a discussion guide
A discussion guide is a plan of what is going to be discussed during the research. Researchers vary in how detailed their guide will be. Some researchers spell out every question they plan to ask in their online chat, focus group, or discussion. Other researchers will simply map out the topics they plan to cover and the sequence which they initially expect to ask them in.

Without a discussion guide the research runs the risk of running out of time, of failing to cover all the necessary topics, or of bringing up the topics in an order that is likely to inappropriately bias the results. A discussion guide can also be a useful way of checking with other stakeholders that the research is likely to cover what is needed.

Not everything that matters can be counted
In most cases, the exact number of times a particular word is used is not directly relevant to the outcome of a qualitative research project. Simple tools, particularly word clouds, give a picture, of qualitative data, based simply on how often certain words occur. Whilst a word cloud can be a useful starting point, it is never enough.
Qualitative research is conducted by reading and considering all the material. In a modern qualitative project, that might include words, pictures, videos, audio contributions and more.

The sequence in which things are said can often matter more than the frequency of words. In an online discussion, for example, it is not unusual for several participants to comment on why they like something, until one person raises a major drawback. When this happens the conversation on that point may simply stop, because the drawback is so clear. But a word count of that conversation would treat the drawback as one comment, and the many, previous, praises for it, as being more significant. The order words are said in matters as much as the content of what is said.

Data does not mean numbers
When a qualitative researcher says ‘data’ they mean the words, pictures, videos, notes, audio recordings, and objects that have been collected. They do not mean a list of numbers in some tabular format.

There are other words that qualitative researchers use, such as text, corpus, discourse, artefacts, objects, exhibits etc. However, all of these can be subsumed in the term data. Sometimes, to reduce confusion these materials are described as qualitative data.

Consider actors and agendas
When looking at a post, an upload, or a comment, the researcher should consider who said it and why. People play roles in discussions, some are trying to be experts, while some are trying to conceal their true feelings. The researcher needs to assess who the actors in the discussion are and what they are trying to achieve, in order to place their contributions in the narrative.

In a discussion about coffee we may identify baristas, amateur experts, people with a green agenda, traditionalists, and innovators. The words cannot be separated from who said them, and ideally who said them to whom. Linking a series of contributions to the same person can increase the insight generated about narrative that is being sought.

We are poor witnesses to our own motivations
Many of the questions that researchers would like to ask are impossible for participants to answer accurately. People tend not to know why they do things. They mostly do not know the drivers of their behaviour. And, they are fairly poor at forecasting what they will do in the future. So, questions that ask “Why are you overweight?”, “Why did you buy that gym membership, knowing you’d hardly use it?” and “What is it about the ACME brand that makes you feel safe and warm?” are likely to fail.

Questions that tend to work are:

• Reporting questions – e.g. “Which cupboard do you store you cleaning products in?” and “How often do you eat in a restaurant?”
• Choice based questions. Show three items, ask “Which is the odd one out?”, and which can then lead into discussions of why.
• Asking about other people. For example, “Tell me all the reasons why some people who are on a diet drink milk shakes?”
• Asking what sorts of people do things. For example, “Tell me who might bake their own bread?”
• Lists – in online research the creation of lists can be a natural way to get participants to be active and to reveal some of their feelings and beliefs. For example, a researcher might ask “Thinking about the brand Coca-Cola, list all the non-drink things you think they would be good at making?” – again leading on to why, and asking who agrees, and who has alternative suggestions.

Asking the obvious questions, for example, “What do you like about this advert?” are always going to be part of the qualitative research process. They are often an easy way to start a discussion, and we want to know what the answers are. However, we do not place too much motivational and narrative importance on the answers to these sorts of questions. The answers should certainly not be reported as being the actual motivations and feelings.

Memoing
When analysing non-trivial amounts of qualitative information, it is really useful to annotate the material. This can be called tagging, memoing, commenting, annotating, highlighting, marking-up and probably a variety of other things. The material is read through and key themes, ideas, quotes, examples, hypotheses etc are noted.

Traditionally, this memoing process was done with scissors, copies of the transcripts, and coloured highlighter pens. Now there are a variety of software tools to help, often referred to as CAQDAS (Computer Aided Qualitative Data Analysis). Some people use specific software, whilst others find they can use Word and/or Excel to achieve what they need.

The narrative is then, typically, constructed from the memos. The source documents are often only referred back to when the story emerging from the memos needs additional evidence or appears inconsistent.

In a collaborative project, participants or ‘the crowd’ are enlisted to add their comments, tags, annotations.

Enable the participants whenever possible
The researcher will be developing hypotheses before the research, during the research, and after the research. One great way of challenging, supporting, or enriching these hypotheses is to actively involve the participants in the process.

Participants can be enabled and encouraged to tag their own comments and uploads, to tag and/or reply to other people’s contributions, and they can feedback on ideas presented by the researcher. In ‘researcher talk’, the researcher provides an outsider’s view (the etic view) whilst the participants can provide an insider’s view (the emic view). A narrative that combines the insider and outsider views is often more powerful than just a single perspective.

Grounding the story in the data
When the narrative is being created, the researcher should check that everything they are claiming can be supported by something evidenced in the data. Whilst not everything in the data should be in the narrative, everything in the narrative should be supported by something in the data.

If the researcher believes something to be true and important, but they cannot support it from the data, they should seek to introduce it to the research, in order to elicit evidence. This is could be via posts in an online discussion, or through the discussion guide for later online focus groups.

Examples that inform, not ones that entertain
There can be a temptation, when creating the research story, to include a video clip, photo, or quote that is particularly powerful, even though it is not truly relevant to the message, or perhaps is even at odds with the main element of the story. This is a bad practice and researchers need to be on their guard against it.

The researcher has to be seen as a ‘truth teller’. The role of the researcher is to tell the customers’ story. This means having the discipline to only use materials that are true to the narrative that has been created.

The “But I already knew that test!”
One test of a powerful narrative from qualitative research is that the client, when presented with the story says “But, I already knew that!” The client did already know it, but they did not know they knew it till they heard the research narrative.

This test, the “I already knew it” test goes to the heart of what qualitative research is all about. The research gathers evidence and synthesises them into a narrative that illuminates the topic under investigation. The illumination, typically, comes from revealing things we already knew, but did not realise or could not access without the research.

---

Suggestions

So, what you your thoughts and suggestions? What should be added, removed, or amended? Indeed, is the project just pure folly?