Qualitative Review Guidelines

RESEARCH ARTICLES OR RESEARCH BRIEFS DESCRIBING QUALITATIVE STUDIES

When writing your review, use line number(s) to identify where the comment refers. This checklist is for your guidance only and should not be uploaded into the review you submit. Cutting and pasting sections, accompanied with appropriate page line numbers, is acceptable.

Note: Not all items will apply to all manuscripts. For quantitative studies, please refer to the Quantitative Review Guidelines.

Considerations when reviewing the body of a Qualitative Research manuscript (Rothpletz-Puglia, 2022):

INTRODUCTION

Is there a clear statement of the aims of the research?  What was the goal of the research?  Why was it thought important or relevant? (Brice, 2018)

Is a qualitative methodology appropriate for addressing the study aim? Does the research seek to interpret or illuminate the actions and/or subjective experiences of research participants?

METHODS

Ethical: Procedural ethics (such as human subjects); situational and culturally specific ethics; exiting ethics (leaving the scene and sharing the research); transparency about assumptions, methods, and challenges (Tracy, 2010)

Research design

• Was the research design appropriate to address the aims of the research?
• Is there methodological coherence (e.g., if phenomenology, narrative analysis, case study, grounded theory, ethnography, etc., are used, is the lens that is applied for each of these methodologies consistent throughout the project? And if it is a basic qualitative study (descriptive and/or interpretive) without the additional dimensions or epistemological framework or lens of the other methodologies, did the authors provide rationale? Basic Qualitative Research description.
• If the study is mixed methods, did the authors describe the design in terms of the purpose, priority, and sequence of methods?
• Did the authors justify the methods chosen, and if mixed methods did the authors describe the justification for using mixed methods to answer the research question?
• Is the setting for data collection justified?
• Is it clear how data were collected?
• Are the methods explicit (e.g., the way interviews were conducted, transcribed, topic guide, etc.?)
• Is the process for data saturation discussed and cited in the methods (sampling/data collection) and/or results? (Brice)
  
  

Recruitment

• Is the recruitment strategy appropriate for the aims of the research?
• Did the researcher explain how and why the participants were selected?
• Is there discussion around recruitment (e.g. why some people chose not to take part)? (Brice, 2018)
  
  

Data Collection:

Is the data collected in a way that addresses the research issue? Consider:

• Is it clear how data were collected (e.g., focus group, semi-structured interview, etc.)?
• Has the researcher justified the methods chosen?
• Has the researcher made the methods explicit (e.g., for the interview method, is there an indication of how interviews were conducted or if a topic guide was used and pretested)?
• Were methods modified during the study? If so, has the researcher explained how and why?
• Is data saturation or “Information power” (Malturud, 2016) discussed (Brice, 2018) [in methods and/or results]? Though, the concept of data saturation is controversial and some experts contend that there cannot be saturation if the research is interpretive. More relevant to projects that are more concrete. https://www.quahrc.co.uk/resources/qualitative-open-mic-qualitative-conundrums-victoria-clarke-on-saturation
• Was the data collection integrated (if mixed methods – and it can be integrated at data collection and/or analysis)?

Data Analysis:

• Are the analytic procedures (see links below for more information) used clearly outlined?
• If it is mixed methods, was this done for the numeric and non-numeric data and how are they integrated)?
• Is there an in-depth description of the analysis process?
• Has the researcher discussed trustworthiness strategies? (Morse, 2015)
• Are there sufficient data presented to support the findings?
• Are contradictory data considered?
• Have the researchers critically examined their own role, potential bias, and influence during analysis and selection of data for presentation? (Brice, 2018)
• Click here to see  Data analysis for descriptive and/or interpretive basic qualitative research including content analysis or thematic analysis.
• Click here to see Data analysis in other qualitative methodologies (Creswell and Poth, 2016)
• If it is a mixed methods research (MMR) study how are the data integrated. If the data are fully integrated during analysis with conversion MMR where qualitative data is converted to quantitative or vice versa, have these procedures been thoroughly described and a citation to a reference provided?
  
  

RESULTS

Is there a clear statement of findings? Consider:

• Are the findings explicit?
• Is it clear how the categories/themes are derived from the data?

• Are the themes reported like categories or domain summaries rather than fully realized themes? Are categories confused for themes?
• Have the data collection questions been used as themes instead of themes generated from the data? Themes are interpretations that conceptually overarch the different categories of data generated.
• Are categories or domain summaries appropriate to the purpose of the research?
• Is non-thematic contextualizing information presented as a theme instead of a descriptive category?
• Would the manuscript benefit from further analysis being undertaken and the reporting of fully realized themes? (Braun)

Is there evidence of weak or unconvincing analysis? Consider:

• Are there too many or too few themes?
• Are there too many theme levels?
• Is there confusion between codes and themes?
• Is there a mismatch between data extracts and analytic claims?
• Are there too few or too many data extracts?
• Is there overlap between themes? (Braun, 2019)

If it is MMR, are the data integrated?

DISCUSSION

Meaningful coherence:  The study (1) Achieves what it purports to be about, (2) Uses methods and procedures that fit its stated goals, and (3) Meaningfully interconnects theory, philosophical assumptions, literature, research questions/foci, findings, and interpretations with each other (Tracy 2010).

• Is there an adequate discussion of the evidence both for and against the researcher’s arguments?
• Are the findings discussed in relation to the original research question (Brice, 2018)

• Are the findings relevant and meaningful beyond the study setting? (Walther et al., 2013)

Do authors make problematic statements about the lack of generalizability of their results, and implicitly conceptualize generalizability as statistical generalizability? (Braun, 2019)

• Should be discussing transferability versus generalizability (unless it is a mixed study and there are data that can be generalized).
  
  

IMPLICATIONS FOR RESEARCH AND PRACTICE

• Did the researchers discuss the contribution the study makes to existing knowledge or understanding (e.g., do they consider the findings in relation to current practice, theory development or policy or relevant research-based literature)?
• Did they discuss how the themes generated lead to new areas where research is necessary?
• Did they discuss whether or how the findings can be transferred to other populations or considered other ways the research may be used? (Brice, 2018)
  
  

REFERENCES

Braun V, Clark V. Checklist for reviewers and editors evaluating thematic analysis manuscripts. 2019. https://cdn.auckland.ac.nz/assets/psych/about/our-research/documents/Checklist%20for%20reviewers%20and%20editors%20evaluating%20thematic%20analysis%20manuscripts.pdf.

Brice R. CASP Checklist: 10 questions to help you make sense of a qualitative research. 2018. CASP-Qualitative-Checklist-2018_fillable_form. chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/viewer.html?pdfurl=https%3A%2F%2Fcasp-uk.b-cdn.net%2Fwp-content%2Fuploads%2F2018%2F01%2FCASP-Qualitative-Checklist-2018.pdf&clen=450217&chunk=true.

Creswell JW, Poth CN. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. Sage Publications; 2016.

Malterud K, Siersma VD, Guassora AD. Sample size in qualitative interview studies: guided by information power. Qual Health Res. 2016;26:1753-1760.

Morse JM. Critical analysis of strategies for determining rigor in qualitative inquiry. Qual Health Res. 2015;25:1212-1222.

O'Cathain A, Murphy E, Nicholl J. (2008): The quality of mixed methods studies in health services research. J Health Serv Res Pol. 2008;13:92–98. https://doi.org/10.1258/jhsrp.2007.007074.

Rothpletz-Puglia P. Qualitative and Mixed Methods Research Appraisal Tool. Rutgers University; 2022. https://docs.google.com/document/d/1ANHM-WBI_f8_8mS4B6NSFJgAWQPbWPaK/edit 

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int. J. Qual. Health Care. 2010;9:349-357.

Tracy SJ. Qualitative quality: eight “big-tent” criteria for excellent qualitative research. Qual Inq. 2010;16:837-851. https://10.1177/1077800410383121.

Walther J, Sochacka NW, Kellam NN. (2013): Quality in interpretive engineering education research: reflections on an example study. J. Eng. Educ. 2013;102:626-659. https://doi.org/10.1002/jee.20029.