Why go qualitative?

Quantitative and qualitative studies are two approaches for knowledge acquisition: the first focuses on objective numeric data, while the second focuses on subjective data that cannot be easily measured. Although quantitative research boasts numerical strength and clarity, sometimes it falls short in explaining complex phenomena. A qualitative approach, on the other hand, allows for a deeper and more nuanced understanding of the data. For example, we might find through a poll that 87% of a population prefer Tool A over Tool B, but the numbers don’t explain why. Are there certain features that make Tool A more attractive? Are there certain use cases for which one tool has an advantage? What type of users prefer which tool? A qualitative investigation, through surveys or interviews, could help to gain a more holistic understanding for user’s preferences of the tools, and perhaps even suggest future directions for tool design.  

Quantitative and qualitative studies are two valid research methodologies which have complementary strengths and weaknesses. We find that, often, the best approach is mixed methods: combining the numeric power of statistics with the context and nuance of qualitative findings.  

That said, let’s learn more about qualitative data.

Collecting Data

Qualitative data collection can take many forms.  

Typically, qualitative data comes from open/free answer responses, ones in which the subject types (or speaks) their own answer rather than choosing from a range of predefined choices. Most often, qualitative data collection actively engages the subject through surveys or interviews, but it can also take a passive form. For example, you might have artifacts from a chatbot session, where you analyze what type of questions the user asked to the bot.  

In a survey, a quantitative question on productivity might look like: “rate your productivity for the last hour on a Likert-scale from 1 to 7”, while a qualitative question might look like: “what factors affected your productivity in the last hour?”. Designing your surveys to have both types of questions (a mixed-methods approach) can give especially rich insights. 

An interview is a more resource-intensive method for both researcher and subject (as it may take 20-60 minutes of both the researcher’s and subject’s time), and is difficult to scale. However, it often results in the richest data. We often employ semi-structured interviews, with a list of predefined questions that the researcher asks to each participant. Some questions include follow-up questions, for example if the subject answered “yes” to a question of “did you run into any issues?”, the researcher can dig deeper into what happened. The semi-structured approach offers some uniformity across participants, while allowing the conversation to go off script to delve deeper into certain points (especially those which could not be foreseen).  

Other qualitative data may include artifacts from a recorded session, such as a screen capture or a video recording. We use the same qualitative analysis approach, adjusted to fit the type of data.  

Now that you have the data, let’s get to the analysis.

Thematic Analysis

There are multiple approaches for analyzing qualitative data, such as Grounded Theory, Narrative Analysis, Discourse Analysis, and more – each of which has its pros and cons for the given dataset and research type. Our research group usually employs Thematic Analysis (TA), which helps us to uncover patterns and themes in the data that help to give context and explain our statistical findings. This method was created by Braun and Clarke: a full description and resources can be found here.

What is Thematic Analysis (TA): 

• TA is a widely used method for analyzing qualitative data in various disciplines. Its versatility and accessibility make it popular across social, behavioral, and applied sciences. The goal is to identify patterns of meaning (‘themes’) in a dataset that relate to a research question. TA involves processes of data familiarization, data coding, theme development, and revision. 
• TA is not a single method but an umbrella term for a family of approaches to develop themes from data. There are broadly three types of TA approaches: coding reliability TA, codebook TA, and reflexive TA. Within our lab, we usually apply the Reflexive TA.  

Reflexive TA 

• The reflexive TA approach, as defined by Braun & Clarke, emphasizes underlying philosophy and theme development procedures. 
• It is theoretically flexible and can adapt to various research questions and theoretical frameworks. 
• Can be used to explore people’s experiences or perceptions (e.g., experiences with body hair removal or views on women in male sports). 
• Suitable for examining understanding and representation (e.g., laypeople’s understanding of therapy or representation of food in teen magazines). 

Image source

So let’s get down to business!

The Six Steps of Reflexive TA:

1. Familiarizing yourself with the dataset: This phase involves reading and re-reading the data, to become immersed and intimately familiar with its content, and making notes on your initial analytic observations and insights, both in relation to each individual data item (e.g. an interview transcript) and in relation to the entire dataset. 
2. Coding: This phase involves generating succinct labels (codes!) that capture and evoke important features of the data that might be relevant to addressing the research question. It involves coding the entire dataset, with two or more rounds of coding, and after that, collating all the codes and all relevant data extracts, together for later stages of analysis. 
3. Generating initial themes: This phase involves examining the codes and collated data to begin to develop significant broader patterns of meaning (potential themes). It then involves collating data relevant to each candidate theme, so that you can work with the data and review the viability of each candidate theme. 
4. Developing and reviewing themes: This phase involves checking the candidate themes against the coded data and the entire dataset, to determine that they tell a convincing story of the data, and one that addresses the research question. In this phase, themes are further developed, which sometimes involves them being split, combined, or discarded. In our TA approach, themes are defined as pattern of shared meaning underpinned by a central concept or idea. 
5. Refining, defining and naming themes: This phase involves developing a detailed analysis of each theme, working out the scope and focus of each theme, determining the ‘story’ of each. It also involves deciding on an informative name for each theme. 
6. Writing up: This final phase involves weaving together the analytic narrative and data extracts, and contextualizing the analysis in relation to existing literature.

[Source]

Tips:

• Share a small sample (e.g. 3 interviews) with (data-approved) research colleagues. Perform TA separately, then come together to discuss the codes. You’ll be amazed by the different insights that come up, and the discussion can really help with guiding your analysis with the rest of the dataset. 
• TA is a recursive process. Once you’ve gone through steps 1-6, you may need to repeat steps as you incorporate more data and uncover more themes. 
• Take notes as you do this analysis – it will really help with clarify coding decisions, provide a history of how the codes or themes changed, and make it easier for you (or peers) to come back to the findings later.
• Collect quotes which capture a certain code, theme, or general sentiment.  Include a selection of these quotes in the write-up.
• Often you see the statement “these themes emerged from the data”, as if the findings were hiding in the data, waiting to be uncovered. Braun and Clarke suggest to use a more active tone which says that the researcher, with their particular perspective, interpreted the data in this way.
• Do not make themes into “categories” of findings (e.g. “Features”, “Bugs”), but rather patterns of meaning that you found repeated across the data (e.g. “While the setup wizard was helpful for newbies, power users found it to be a nuisance”).

Good to Know – Different Approaches:  

• A more inductive way – coding and theme development are directed by the content of the data; 
• A more deductive way: coding and theme development are directed by existing concepts or ideas; 
• A more semantic way: coding and theme development reflect the explicit content of the data; 
• A more latent way: coding and theme development report concepts and assumptions underpinning the overt content of the data; 
• A more (critically) realist or essentialist way: analysis focuses on reporting an assumed reality evident in the data; 
• A more constructionist way: analysis focuses on exploring the realities produced within the data. 

In practice more inductive, semantic and (critical) realist approaches tend to cluster together; ditto more deductive, latent and constructionist ones. The separation between orientations isn’t always rigid. What is vitally important is that the analysis is theoretically coherent and consistent (see quality in TA). 

Our lab often employs an inductive method and a critical realist perspective, but this of course depends on the study and researcher. 

Resources:

• Clarke explains Thematic Analysis: video
• Resource for TA (see examples section especially)

Note: This guide aims to guide students in writing good theses/project reports with/for HASEL. It is subject to change. Not all the suggestions might always apply. Ask your supervisor in case of questions.