AMSTAR-2 (A MeaSurement Tool to Assess systematic Reviews)
| Study designs | Systematic reviews of randomized and non-randomized studies. |
| Number of items | 16 |
| Rating | yes and no (other response options are available for some items such as partial yes and no meta-analysis conducted) |
| Validity | A face to face meeting was held with a group of experts to uptate the tool from the results of updated literature reviews on relevant critical appraisal tools, surveys of AMSTAR users, experience of participants in AMSTAR workshops at conferences, feedback from the AMSTAR website, and published critiques of the original tool (Shea et al 2017). Studies on validity: • Lorenz, R. C. (2020). AMSTAR 2 overall confidence rating: lacking discriminating capacity or requirement of high methodological quality? Journal of Clinical Epidemiology, 119, 142. • Lorenz, R. C., Matthias, K., Pieper, D., Wegewitz, U., Morche, J., Nocon, M., et al. (2019). A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool. Journal of Clinical Epidemiology, 114, 133-140. |
| Reliability | Studies on reliability: • Bühn, S., Ober, P., Mathes, T., Wegewitz, U., Jacobs, A., & Pieper, D. (2021). Measuring test-retest reliability (TRR) of AMSTAR provides moderate to perfect agreement–a contribution to the discussion of the importance of TRR in relation to the psychometric properties of assessment tools. BMC medical research methodology, 21(1), 1-7. • García-Alamino, J. M., López-Cano, M., Kroese, L., Helgstrand, F., & Muysoms, F. (2019). Quality Assessment and Risk of Bias of Systematic Reviews of Prophylactic Mesh for Parastomal Hernia Prevention Using AMSTAR and ROBIS Tools. World Journal of Surgery, 43(12), 3003-3012. • Gates, M., Gates, A., Duarte, G., Cary, M., Becker, M., Prediger, B., et al. (2020). Quality and risk of bias appraisals of systematic reviews are inconsistent across reviewers and centers. Journal of Clinical Epidemiology, 125, 9-15. • Gates, A., Gates, M., Duarte, G., Cary, M., Becker, M., Prediger, B., et al. (2018). Evaluation of the reliability, usability, and applicability of AMSTAR, AMSTAR 2, and ROBIS: protocol for a descriptive analytic study. Systematic reviews, 7(1), 85. • Inuganti, B., Inuganti, A., Vsn, M., Hyderboini, R., Chakrawarthy, M., Chidirala, S., et al (2018). A Comparison of Amstar and Robis Tools for Methodological Quality Assessment of Systematic Reviews of Alzheimer’s Disease. Value in Health, 21, S230. • Leclercq, V., Beaudart, C., Tirelli, E., & Bruyère, O. (2020). Psychometric measurements of AMSTAR 2 in a sample of meta-analyses indexed in PsycINFO. Journal of Clinical Epidemiology, 119, 144-145. • Lorenz, R. C., Matthias, K., Pieper, D., Wegewitz, U., Morche, J., Nocon, M., et al. (2019). A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool. Journal of Clinical Epidemiology, 114, 133-140. • Pieper, D., Buechter, R. B., Li, L., Prediger, B., & Eikermann, M. (2015). Systematic review found AMSTAR, but not R (evised)-AMSTAR, to have good measurement properties. Journal of Clinical Epidemiology, 68(5), 574-583. • Pieper, D., Jacobs, A., Weikert, B., Fishta, A., & Wegewitz, U. (2017). Inter-rater reliability of AMSTAR is dependent on the pair of reviewers. BMC Medical Research Methodology, 17(1), 98. • Pieper, D., Puljak, L., González-Lorenzo, M., & Minozzi, S. (2019). Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. Journal of Clinical Epidemiology, 108, 26-33. • Swierz, M. J., Storman, D., Zajac, J., Koperny, M., Weglarz, P., Staskiewicz, W., et al. (2021). Similarities, reliability and gaps in assessing the quality of conduct of systematic reviews using AMSTAR-2 and ROBIS: systematic survey of nutrition reviews. BMC Medical Research Methodology, 21(1), 261. |
| Other information | Previous version: AMSTAR developed in 2007 The second version of this tool was developed in 2017. https://amstar.ca/Amstar-2.php |
| Main references | • Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., et al. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. British Medical Journal, 358, j4008. • Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., et al. (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 62(10), 1013-1020. |
ROBIS (Risk of Bias in Systematic Reviews)
| Study designs | Systematic reviews |
| Number of items | 4 domains (5 to 6 items in each domain) |
| Rating | yes, probably yes, probably no, no, no information |
| Validity | A steering group of 11 experts defined the scope of the tool. Items were identified from reviewing 40 existing tools, a review of overviews and from the Methodological Expectations of Cochrane Intervention Review guidelines. A first draft of the tool was developed from a face to face meeting was held during a conference. The scope and content of the tool was refined during a Delphi study. The tool was piloted during conferences (Whiting et al 2016). Studies on validity: • Jaca, A., Ndze, V. N., Wiysonge, C. S. (2019). Assessing the methodological quality of systematic reviews of interventions aimed at improving vaccination coverage using AMSTAR and ROBIS checklists. Human Vaccines & Immunotherapeutics, 15(12), 2824-2835. |
| Reliability | Studies on reliability: • Banzi, R., Cinquini, M., Gonzalez-Lorenzo, M., Pecoraro, V., Capobussi, M., & Minozzi, S. (2018). Quality assessment versus risk of bias in systematic reviews: AMSTAR and ROBIS had similar reliability but differed in their construct and applicability. Journal of Clinical Epidemiology, 99, 24-32. • Bühn, S., Mathes, T., Prengel, P., Wegewitz, U., Ostermann, T., Robens, S., & Pieper, D. (2017). The risk of bias in systematic reviews tool showed fair reliability and good construct validity. Journal of Clinical Epidemiology, 91, 121-128. • García-Alamino, J. M., López-Cano, M., Kroese, L., Helgstrand, F., & Muysoms, F. (2019). Quality Assessment and Risk of Bias of Systematic Reviews of Prophylactic Mesh for Parastomal Hernia Prevention Using AMSTAR and ROBIS Tools. World Journal of Surgery, 43(12), 3003-3012. • Gates, A., Gates, M., Duarte, G., Cary, M., Becker, M., Prediger, B., et al. (2018). Evaluation of the reliability, usability, and applicability of AMSTAR, AMSTAR 2, and ROBIS: protocol for a descriptive analytic study. Systematic Reviews, 7(1), 85. • Gates, M., Gates, A., Duarte, G., Cary, M., Becker, M., Prediger, B., et al. (2020). Quality and risk of bias appraisals of systematic reviews are inconsistent across reviewers and centers. Journal of Clinical Epidemiology, 125, 9-15. • Inuganti, B., Inuganti, A., Vsn, M., Hyderboini, R., Chakrawarthy, M., Chidirala, S., et al. (2018). A Comparison of Amstar and Robis Tools for Methodological Quality Assessment of Systematic Reviews of Alzheimer’s Disease. Value in Health, 21, S230. • Pieper, D., Puljak, L., González-Lorenzo, M., & Minozzi, S. (2019). Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. Journal of Clinical Epidemiology, 108, 26-33. • Swierz, M. J., Storman, D., Zajac, J., Koperny, M., Weglarz, P., Staskiewicz, W., et al. (2021). Similarities, reliability and gaps in assessing the quality of conduct of systematic reviews using AMSTAR-2 and ROBIS: systematic survey of nutrition reviews. BMC Medical Research Methodology, 21(1), 261. |
| Other information | https://www.bristol.ac.uk/population-health-sciences/projects/robis/ |
| Main references | Whiting, P., Savovic, J., Higgins, J. P., Caldwell, D. M., Reeves, B. C., Shea, B., et al. (2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69(1), 225-234. |
Mixed Methods Systematic Reviews Appraisal Tool (MMSR)
| Study designs | Mixed methods systematic reviews |
| Number of items | 27 items (on methods to identify and include studies, to analyse the findings in quantitatives and qualitatives analysis and to analyse the causal chain and reach conclusions). |
| Rating | yes, partially, no, not applicable |
| Validity | Tool developed from frameworks for mixed methods research, and existing systematic review critical appraisal tools and checklists. The tool was reviewed by mixed methods experts and piloted by two independent reviewers on 7 papers (Jimenez et al 2018). |
| Reliability | N/A |
| Other information | N/A |
| Main references | • Jimenez, E., Waddington, H., Goel, N., Prost, A., & Pullin, A. (2018) Mixing and Matching: Using Qualitative Methods to Improve Quantitative Impact Evaluations (IEs) and Systematic Reviews (SRs) of Development Outcomes. London, UK: Centre of Excellence for Development Impact and Learning (CEDIL) Inception Paper 5. • Jimenez, E., Waddington, H., Goel, N., Prost, A., Pullin, A., White, H., . . . Bhatia, R. (2018). Mixing and matching: using qualitative methods to improve quantitative impact evaluations (IEs) and systematic reviews (SRs) of development outcomes. Journal of Development Effectiveness, 10(4), 400-421. |
Quality standards for realist synthesis
| Study design | Realist synthesis |
| Number of criteria | 9 criteria |
| Rating | Inadequate, adequate, good, excellent |
| Validity | Tool developed from several sources such as email discussion list gathering more than 300 members, methodological experts, and methods training workshops (Wong et al 2014). |
| Reliability | N/A |
| Other information | https://www.ramesesproject.org/Standards_and_Training_materials.php |
| Main references | • Wong, G., Greenhalgh, T., Westhorp, G., & Pawson, R. (2014). Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: The RAMESES (Realist And Meta-narrative Evidence Syntheses - Evolving Standards) project. Health Services and Delivery Research, 2(30). |
Quality standards for meta-narrative synthesis
| Study design | Meta-narrative synthesis |
| Number of criteria | 10 criteria |
| Rating | Inadequate, adequate, good, excellent |
| Validity | Tool developed from several sources such as email discussion list gathering more than 300 members, methodological experts, and methods training workshops (Wong et al 2014). |
| Reliability | N/A |
| Other information | https://www.ramesesproject.org/Standards_and_Training_materials.php |
| Main references | • Wong, G., Greenhalgh, T., Westhorp, G., & Pawson, R. (2014). Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: The RAMESES (Realist And Meta-narrative Evidence Syntheses - Evolving Standards) project. Health Services and Delivery Research, 2(30). |
Critical Appraisal Tool for Health Promotion and Prevention Reviews (CAT HPPR)
| Study designs | Checklist to assess the quality of different review types (systematic reviews, rapid reviews, scoping reviews, etc.) in a review database for health promotion and prevention. |
| Number of items | 15 items |
| Rating | yes, no, not applicable |
| Validity | The CAT HPPR was developed from a review of 14 existing appraisal tools. It was piloted with 14 reviews to obtain feedback and clarification requests from users and experts. |
| Reliability | N/A |
| Other information | |
| Main references | • Heise, T. L., Seidler, A., Girbig, M., Freiberg, A., Alayli, A., Fischer, M., Haß, W., & Zeeb, H. (2022). CAT HPPR: a critical appraisal tool to assess the quality of systematic, rapid, and scoping reviews investigating interventions in health promotion and prevention. BMC Medical Research Methodology, 22(1), 1-10. https://doi.org/10.1186/s12874-022-01821-4 |