Which type of qualitative analysis do you prefer? Why not use an LLM to allow you to use multiple approaches?
Ray Poynter, 4 August, 2023
When conducting qualitative analysis, we have a wide range of choices. Popular approaches include:
- Discourse Analysis
- Grounded Theory
- Narrative Analysis
- Framework Analysis
- Hermeneutic Analysis
When I tackle a project for non-trivial cases, I tend to choose one of these, balancing the strength of the technique with the time it takes. But, with all the fuss about Large Language Models, I wondered if I could leverage multiple techniques on a single project. Here is a simple experiment using ChatGPT4.
The Kings Cross/St Pancras Travelodge and Tripadvisor
To gather some data to work with, I collected a series of reviews of a budget hotel in central London. I have stayed at this hotel several times, and I have used this hotel several times before for analysis, which means I have some insight into what any new analysis is likely to show.
To conduct the analysis, I asked ChatGPT4 to analyse the same reviews using the five approaches listed above. I then reviewed the five explications to look at the similarities and differences, that is the consonances and dissonances.
Categorising Methods
Discourse Analysis, Grounded Theory and Framework Analysis found a similar set of findings. The location was the main plus, for example Discourse Analysis said “Guests consistently appreciate the hotel’s convenient location, particularly its proximity to King’s Cross station. This aspect is emphasized as a significant advantage for travelers seeking ease of access to transportation.”. Other positives included the friendliness of the staff and the relatively low cost.
The main negatives related to the quality and cleanliness of the rooms. For example, Discourse Analysis said “Opinions on room quality and cleanliness are mixed. While some reviewers express satisfaction with modern rooms and cleanliness, others report concerns regarding unclean corridors and floors. Basement rooms are mentioned as less desirable by some guests.”. Other negatives included the amenities, noise and other disturbances.
Storytelling Methods
Both Narrative Analysis and Hermeneutic Analysis produced outputs in a more storyline format.
For example, Narrative Analysis started with “A recurring plotline in the narratives centered around the hotel’s location and accessibility. Guests highlighted their convenience in accessing King’s Cross station and the surrounding areas, portraying this as a pivotal aspect of their experience.”.
And Hermeneutic Analysis started with “The reviews reflect an appreciation for the hotel’s location, highlighting a sense of place that is intertwined with the urban experience. The proximity to King’s Cross station is not merely seen as a matter of convenience but also connects guests to the vibrancy and allure of London’s bustling city life.”
Putting it all Together
Armed with all five outputs, I can then start my analysis. Given what I already know about this hotel and on the basis of these five explications, I would veer towards the storytelling methods. However, I would also promote the importance of affordability. The ChatGPT4 analyses mentions affordability, but it tends to assign affordability as a secondary or tertiary feature. This understating of the importance of price is typical of the way people in the UK write reviews.
The use of ChatGPT4 helped me look at the text in multiple ways to ensure that I do not fall into the trap of using just perspective. It helps me to do more analysis in less time, focusing on the points of consonance and dissonance.
So What?
In my opinion, in the short-term, one of the main impacts of LLMs on qual analysis will be to enable more and deeper analysis to be conducted in the available time. As in this example, the researcher can request multiple views of the data to help highlight patterns of consonance and dissonance. All qualitative analysis requires immersion, but the options presented by LLMs offer us the opportunity to change the point of immersion. Instead of immersing in the raw text, I can immerse myself in suggested explanations AND the text. The two reasons for doing this second approach are a) in my experience, it is faster (for non-trivial projects), and b) it might reduce the risk of my biases blinding me to something.
One thought on “Which type of qualitative analysis do you prefer? Why not use an LLM to allow you to use multiple approaches?”
Comments are closed.
Hi Ray,
Thanks for tagging me in the post. I have been experimenting with generative AI in qual and quant open ends as well. While I fully appreciate that this technology will evolve and get better as most things do, let me share my experiences so far…
I like to remind myself of the qual content analysis process, approach and purpose every time I have tried something new. Content analysis does a few main things, 1) organises data in a way that helps you step up towards the bigger picture 2) It forces you to read through all the content, so you pick up on anomalies and mass themes & 3) becomes the main source of truth and/ or evidence source. Other things as well, of course.
Something has always stuck with me with analysis, if content analysis is done right then it removes the subjectivity of qual learnings and the true measure is, if we share content analysis with two equally experienced researchers, they should arrive at conclusions which are 85% similar. Sadly, we see more qual practitioners running a few interviews or groups and jumping in to write the report from back of their head.
I have used generative AI with more focused content (transcripts from interviews and groups) as opposed to reviews and feedback on websites, so my experience has been a bit different to yours.
While there is so much I can talk about here, one thing that stood out for me is around hermeneutic analysis. The AI works if the content you are pointing to is the only content that there is but if there are layers then the AI does not do justice. For example, if we were using it to summarise content from what was discussed about a new sports drink idea in a focus group, AI is fast and effective in summarizing what is working or not working (what you have referred to as categorizing in your article) but, if you want to understand why does it not work for some then you need to look at what else these consumers have shared earlier or later about their needs and lifestyle to decipher the pertinent ‘why’. Generative AI does not do that currently and that is understandable as there are multiple voices in the mix. This could also be a user error. This level of analysis is important to me as the business challenges we address are narrow and more like ‘how do I steal share from the competitor’, or ‘how can I get more people to drink sports drinks’ or ‘how do I improve the appeal of my product’, seldom are they about ‘tell us what consumers are talking about sports drinks’. With in-depth interview data I got a lot further, as I was able to adjust my questions until I got it looking at the content the right way.
With interview data, I did the analysis manually first and arrived at my summaries and conclusions for one interview, then I used the AI to replicate the process, the process revealed the right questions to ask. Once I was happy with that, I was able to scale that up quickly for multiple interviews.
With quantitative open ends, it is interesting because if you are accustomed to doing basic wordclouds then AI is a step up, but if you want to specifically look at a view and overlay operational data or survey responses from other questions then I found AI to be more cumbersome to manage. Conventional MR data analysis tools are better set up to do that. While there is so much more to talk about here, I continue to experiment and learn with every new study. I do like the speed and scale of doing some activities that take time but with other areas it leaves me more anxious when it summarises at the overall level only.
With all the said, I am enjoying the scale and speed that AI lends itself to with some tasks and particularly the translation with context when you are working with multiple languages. I believe with use I am getting better at asking better questions.
So, while I agree with you that it helps do more analysis in less time, I am not comfortable in the ability to focus on points of consonance and dissonance, in a focused data context which is generated with moderator guides and/ or multiple consumers impacting the conversation.
Cheers
Bala