More than 100 critical appraisal tools for qualitative studies have been developed. However, still few have been validated.
Some reviews on critical appraisal tools for qualitative studies:
- Munthe-Kaas, H. M., Glenton, C., Booth, A., Noyes, J., & Lewin, S. (2019). Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Medical Research Methodology, 19(1), 113. doi:10.1186/s12874-019-0728-6
- Majid, U., & Vanstone, M. (2018). Appraising Qualitative Research for Evidence Syntheses: A Compendium of Quality Appraisal Tools. Qualitative Health Research, 28(13), 2115-2131. doi:10.1177/1049732318785358
- Santiago-Delefosse, M., Gavin, A., Bruchez, C., Roux, P., & Stephen, S. (2016). Quality of qualitative research in the health sciences: Analysis of the common criteria present in 58 assessment guidelines by expert users. Social Science and Medicine, 148(1), 142-151. doi:10.1016/j.socscimed.2015.11.007
Checklist from Boejie et al (2011)
| Study designs | Qualitative studies |
| Number of items | 10 |
| Rating | 0 (item is absent), 1 (item is dealt with but weak), and 2 (item is satisfactorily dealt with). |
| Validity | This tool was developed from existing critical appraisal tools for qualitative studies. Two reviewers rated 17 articles using expert judgement (1 to 10) and the tool. The correlation between the overall expert judgement and the checklist appraisal was 0.867. |
| Reliability | Two reviewers appraised 17 articles. The ICC was 0.94 . |
| Other information | N/A |
| Main references | Boeije, H. R., van Wesel, F., & Alisic, E. (2011). Making a difference: towards a method for weighing the evidence in a qualitative synthesis. Journal of evaluation in clinical practice, 17(4), 657-663. |
Checklist from Vermeire et al (2002)
| Study designs | Focus group |
| Number of items | 13 |
| Rating | yes, no, can’t tell |
| Validity | Tool developed based upon a review of the literature and expert panel discussions among primary care researchers experienced in focus group research (the six authors). To test the feasibility of using the checklist, primary care researchers not experienced in focus group research were asked to use the draft checklist to assess five focus group research articles randomly chosen among a sample of 70 papers. The median time to appraise a study using the pilot version was 30 minutes (with mean of 68 minutes). The applicability varied among the studies. The rating scale was considered too rigid. |
| Reliability | Three papers were appraised by 9 primary care researchers and there was 68% agreement, 16% contradiction and 16% of discrepency because of doubt. |
| Other information | N/A |
| Main references | Vermeire, E., Van Royen, P., Griffiths, F., Coenen, S., Peremans, L., & Hendrickx, K. (2002). The critical appraisal of focus group research articles. European Journal of General Practice, 8(3), 104-108. |
Checklist from Reis et al (2007)
| Study designs | Qualitative studies |
| Number of items | 15 |
| Rating | 0 (unable to rate); 1 (low/barely); 2 (moderate/moderately); 3 (high/clearly) |
| Validity | Tool developed by research group base on existing literature. Pilot tested by analysing a set of qualitative papers and refined the tool until content, clarity and wording was satisfactory. |
| Reliability | The interrater reliability was tested for the two pairs of investigators(ranged from 0.22 to 0.77) and for one investigator compared to the other three (ranged from 0.38 to 0.60). Agreement on the item of global rating ranged from 0.36 to 1.00. |
| Other information | N/A |
| Main references | Reis, S., Hermoni, D., Van-Raalte, R., Dahan, R., & Borkan, J. M. (2007). Aggregation of qualitative studies—From theory to practice: Patient priorities and family medicine/general practice evaluations. Patient Education and Counseling, 65(2), 214-222. |
CAMELOT
| Study designs | Checklist to assess the quality of qualitative studies. |
| Number of items | 12 domains |
| Rating | no or minimal concerns, minor concerns, moderate concerns or serious concerns |
| Validity | Tool developed based on four stages: (1) systematic literature review of existing tools, (2) identification of evidence to support potential domains (3) consensus survey, and (4) human-centered design approach by exploring user experience. |
| Reliability | N/A |
| Other information | https://camelotapproach.wordpress.com/ http://thecamelotplot.pbworks.com/ |
| Main references | Munthe-Kaas, H. M., Booth, A., Sommer, I., Cooper, S., Garside, R., Hannes, K., Noyes, J., & Group, T. C. D. (2024). Developing CAMELOT for assessing methodological limitations of qualitative research for inclusion in qualitative evidence syntheses. Cochrane Evidence Synthesis and Methods, 2(6), e12058. https://doi.org/https://doi.org/10.1002/cesm.12058 |