Systematic Reviews

Systematic review: a method for making informed decisions based on the best available body of evidence.

According to the Cochrane Handbook, a systematic review collates all eligible evidence for a focused research question using explicit, pre-specified methods designed to minimize bias, so that the findings are more reliable for drawing conclusions and informing decisions.

In this sense, a systematic review is:

• A robust way to build confidence in scientific knowledge over time, by synthesizing results across multiple studies.

• A method for assessing the reliability and validity of bodies of research, by evaluating consistency, quality, and risk of bias across studies.

(Committee on Reproducibility and Replicability in Science, 2019)

References

• Committee on Reproducibility and Replicability in Science. (2019). Reproducibility and Replicability in Science. Washington, DC: National Academies Press. https://doi.org/10.17226/25303
• Higgins, J., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M., & Welch, V. A. (Eds.). (2024). Cochrane Handbook for Systematic Reviews of Interventions (Version 6.5, updated August 2024). Cochrane. Retrieved from https://training.cochrane.org/handbook/current 

To determine which type of knowledge synthesis best suits your project, consult the decision tree below, designed by Professors Ana Loritz de Guinea and Guy Paré.

You can also use these tools to help you make appropriate methodological choices for your project:

• Right Review
• Literature Reviews Explained (LITR-EX)
• Center for Open Science
• SystematicReview_DecisionTree

Reference 

• Ortiz de Guinea, A., & Paré, G. (2018). What Literature Review Type Should I Conduct? In Robert Galliers & Mari-Kara Stein (Eds.), The Routledge Companion to Management Information systems, p. 78. Routledge.

Systematic reviews often address the question of whether an intervention is effective or not.

• Does this intervention work in this population?
• Or is there insufficient evidence?

A typical systematic review question could be formulated as follows:

What is the effect of intervention X on population Y in order to achieve outcome Z?

Systematic reviews can also be used to answer various well-formulated research questions, such as:

• how a concept is defined and measured;
• questions about rates, trends, and prevalence;
• determining whether variables are correlated;
• exploring cause-and-effect relationships.

Reference

• Evidence Synthesis Institute (2024). « Exploring different review methodologies », présentée à la Evidence Synthesis Institute Conference.

The methods used in a systematic literature review must be transparent, reproducible, and replicable.

Reproducibility refers to the ability of a researcher to duplicate the results of a previous study using the same materials and procedures employed by the original researcher. (Bollen et al., 2015) Other researchers can thus verify the work conducted, demonstrating that studies were not selected arbitrarily.

Replicability refers to the ability of a researcher to duplicate the results of a previous study if the same procedures are followed, but with newly collected data. (Bollen et al., 2015) It allows a researcher to redo or update their own research as well as replicate and extend analyses carried out by others. Replicability can lead to a living systematic review.

Even if a literature review has been conducted using transparent methods, the reproducibility and replicability of results may be affected by factors beyond the researchers’ control, particularly in relation to the use of databases, such as:

• syntax changes;
• algorithm changes;
• retrospective cataloging;
• reassignment of subject terms;
• addition or removal of journals;
• default settings that vary across institutions;
• discontinuation of a database;
• the researcher attempting to reproduce the search strategy may not have access to databases or proprietary platforms used in the original review.

(Evidence Synthesis Institute, 2024)

References

• Bollen, Kenneth, John T. Cacioppo, Robert M. Kaplan, Jon A. Krosnick et James L. Olds (2015). Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science.
• Cram, W. A., Templier, M., & Paré, G. (2020). (Re)considering the Concept of Literature Review Reproducibility. Journal of the Association for Information Systems, 21(5), 1103–1114. https://doi.org/10.17705/1jais.00630 
• Evidence Synthesis Institute (2024). « Reproducibility, Transparent Methods and Pre-Registration », présentée à la Evidence Synthesis Institute Conference.
• Subcommittee on Replicability in Science Advisory Committee to the National Science Foundation Directorate for Social,  Behavioral, and Economic Sciences (2015). Repéré de https://www.datamanagement.hms.harvard.edu/sites/default/files/assets/documents/SBE_Robust_and_Reliable_Research_Report.pdf

In a systematic review, guidelines for research—including standards or checklists—are documents containing rules, principles, or standards that researchers are required to follow when conducting and presenting their research. They help to standardize systematic reviews, facilitating the assessment of research quality and enabling readers to follow the procedures and evaluate the conclusions.

Guidelines for conducting research

These serve as user manuals for researchers in knowledge synthesis, providing direction on establishing inclusion and exclusion criteria, best practices for study screening, data extraction, and other essential steps to be followed when conducting a review. Guidelines should be selected at the beginning of the research process.

Examples include:

• Joanna Briggs Institute (JBI) Manuals:
  • JBI, an international nonprofit organization dedicated to research and development, based in Australia, provides resources offering guidance for conducting systematic reviews.
• Agency for Healthcare Research and Quality (AHRQ) Methods Guide for Comparative Effectiveness Reviews:
  • The intended audience includes researchers and clinicians who use systematic review methods. These reviews focus on medical treatments and interventions.

Guidelines for reporting results

These provide a list of items that researchers must use when documenting study details in the IMRAD structure (Introduction, Methods, Results, and Discussion) as well as in abstracts, appendices, and supplements. They specify what must be reported transparently, thereby promoting better understanding among readers and increasing the likelihood that other researchers can reproduce the study.

Examples include:

• Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Guidelines, Checklists, Flow Diagrams & Extensions
  • PRISMA for Searching (PRISMA-S)
• Reporting standards of the American Psychological Association (APA)
  • Journal Article Reporting Standards – Quantitative Research Design (JARS–Quant), which include reporting standards not limited to systematic reviews.
  • Meta-Analysis Reporting Standards (MARS)
• Collaboration for Environmental Evidence’s Reporting standards for Systematic Evidence Syntheses (ROSES)

Guidelines for conducting & reporting

• Methodological Expectations of Cochrane Intervention Reviews (MECIR)
• Methodological Expectations of Campbell Collaboration Intervention Reviews (MECCIR)
• Collaboration for Environmental Evidence (CEE) Guidelines for Systematic Reviews in Environmental Management

Factors to consider when selecting guidelines:

• Type of review: determines the appropriate guidelines, as each review type has specific methodologies and reporting expectations.
• Discipline: may have its own guidelines reflecting its practices, standards, and research needs, ensuring recommendations are relevant and tailored to the field.
• Population: some standards may be designed for specific demographic groups or research subjects (e.g., medical treatments for children or environmental studies on trees).
• Journal requirements: regarding reporting and methodologies. Meeting these requirements is necessary for submission and publication.
• Collaborative research networks (e.g., Campbell Collaboration): often provide tailored guidelines for their members, helping to harmonize practices within a field and improve the quality of systematic reviews.

(Evidence Synthesis Institute, 2024)

Reference

• Evidence Synthesis Institute (2024). « Systematic Review Guidelines, Checklists & Reporting Standards », présentée à la Evidence Synthesis Institute Conference.

Pre-registration helps to enhance the reproducibility of results in the context of a systematic review. Pre-registration involves stating your research plan, including your hypotheses, methods, and analyses, before collecting data.

This approach helps to:

• distinguish exploratory research from confirmatory research by clearly defining whether the study aims to generate hypotheses or to test them;
• avoid the practice of formulating hypotheses after results are known, which can bias conclusions;
• follow rigorous methods defined in advance rather than being guided by results (e.g., selecting only favorable outcomes);
• promote transparency and reduce bias.

The pre-registration of a systematic review is carried out by formally registering its protocol (fr).

Reference

• Evidence Synthesis Institute (2024). « Systematic Review Guidelines, Checklists & Reporting Standards », présentée à la Evidence Synthesis Institute Conference.