Systematic Reviews

At the study selection stage, the MECIR guidelines recommend in particular:

• Using (at least) two independent reviewers to determine whether each study meets the eligibility criteria, with a predefined process for resolving disagreements (C39);
• Including studies in the analysis regardless of whether the outcome data are reported in a "usable" way (C40);
• Documenting the selection process in sufficient detail to complete a PRISMA flow diagram and a table of “characteristics of excluded studies” (C41).

PRISMA flow diagram

The PRISMA flow diagram (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) is a visual tool used to document and present the steps of a systematic review transparently, from the identification of initial sources through to the final inclusion of studies.

It details the number of studies identified, excluded, and included—along with reasons for exclusion. The diagram can be adapted to the type of review and the sources used. It ensures rigor, transparency, and reproducibility in the process.

• Select the appropriate template for your review and sources.
• You can generate a PRISMA flow diagram using the Shiny App.

Selection process
Search results should be systematically assessed for relevance according to eligibility criteria in a two-step process:

1. Title and abstract screening: Review titles and abstracts to exclude clearly irrelevant records.
2. Full-text screening: Examine the full text of articles to identify those that meet all inclusion criteria.

Suggested workflow for rigorous selection

1. Pilot title and abstract screening

• Objective: Test and refine inclusion/exclusion criteria and establish an initial level of agreement among reviewers.
• Process:
  • Use a random sample of 25–50 records.
  • Aim for at least 75% agreement between reviewers.
  • Consider practical aspects: where screening will occur (e.g., software platform) and how many reviewers are involved.

2. Title and abstract screening

• Objective: Systematically review all retrieved records.
• Process:
  • Use screening software where possible (e.g., Covidence).
  • Provide training or demonstrations and set clear eligibility parameters.
  • Track the number of records: # Included and # Excluded for the PRISMA diagram.

3. Full-text retrieval

• Objective: Retrieve the full texts for records retained after title and abstract screening.
• Process:
  • Use tools such as reference management software (EndNote, Zotero), library catalogues (e.g., Sofia), or interlibrary loan services.
  • Establish a system for:
    • Storing PDFs in a centralized repository.
    • Naming PDFs consistently (e.g., using a standardized convention).
  • Document:
    • # of PDFs retrieved.
    • # of PDFs not retrieved, for the PRISMA diagram.

4. Pilot full-text screening

• Objective: Refine inclusion/exclusion criteria and establish consensus on reasons for exclusion.
• Process:
  • Use a random sample of about 5 PDFs, ensuring that at least one is included if possible.
  • Define and test a hierarchy of exclusion reasons (to avoid conflicts in tools like Covidence).
  • Aim for a high level of agreement among reviewers.

5. Full-text screening

• Objective: Conduct a complete review of all full-text articles.
• Process:
  • Use software to track decisions (e.g., Covidence and its PRISMA diagram).
  • Define and apply exclusion reasons consistently.
  • Record:
    • # Included.
    • # Excluded (with reasons) for the PRISMA diagram.

(Evidence Synthesis Institute, 2024)

Using Covidence to streamline the process

Covidence is an online platform that simplifies the screening and analysis of articles for systematic reviews and other types of evidence syntheses, such as scoping reviews, umbrella reviews, and meta-analyses.

The research team sets the project parameters and criteria used to:

• Import bibliographic references from article databases;
• Decide whether to include or exclude each article based on its title and abstract;
• Retrieve the full text of retained articles and evaluate their quality and relevance;
• Extract and export references in different formats.

After import, Covidence provides several tools for managing and processing references:

• Automatic deduplication: Covidence identifies and removes duplicate references, leaving a clean dataset for subsequent steps.
• PRISMA flow tracking: Covidence generates a PRISMA flow diagram documenting each stage of the article selection process, including duplicates removed and exclusions at each phase.
• Title and abstract screening: Allows for an initial triage of studies, with each reference assessed against inclusion and exclusion criteria.
• Full-text screening: After title and abstract triage, the selected references move on to full-text assessment.

To retrieve full texts, you can use reference management software such as EndNote or Zotero. The recommended procedure is:

• Export filtered references from Covidence: Export the remaining references from Covidence in RIS format and import them into EndNote or Zotero.
• Use the “Find Full Text” function: In EndNote or Zotero, use the automatic full-text search options (via DOI, open access, and library database connections). These tools will locate and download available PDFs. Make sure the settings are configured properly to maximize retrieval success.
  • EndNote
  • Zotero
• Re-import full texts into Covidence: Once PDFs are found, they can be re-imported into Covidence for annotation and centralized analysis.
  • See the “Bulk download” section in the Covidence guide.

Reference 

• Evidence Synthesis Institute (2024). « Wirting a Search Strategy and Translating it to Multiple Databases », présentée à la Evidence Synthesis Institute Conference.

The risk of bias refers to systematic errors in the results of studies or their synthesis, which can lead to inaccurate conclusions about the true effect of an intervention. Bias can arise from:

• flaws in study design;
• reporting practices (e.g., publication bias or selective outcome reporting);
• conflicts of interest,

and can result in either an overestimation or underestimation of effects. Cochrane reviews address risk of bias at two levels: within individual studies and across the overall synthesis, ensuring that these risks are carefully assessed and integrated into the analysis to strengthen the reliability and validity of conclusions.

(Cochrane Handbook, section 7.1)

Why assess risk of bias in systematic reviews?

Assessing risk of bias in systematic reviews is essential because empirical evidence shows that certain aspects of study design, methodology, and reporting can introduce systematic errors (bias) that affect the reliability of results. While it is often impossible to precisely measure the extent of bias in a given study, evaluating the risk of bias provides a structured way to assess its potential impact on the conclusions.

• Risk of bias or imprecision?
  Bias is a systematic error that distorts results, even after repeated replications of a study. Imprecision, on the other hand, is due to random variation in effect estimates caused by sample size or number of events.

• Risk of bias or external validity?
  Bias affects study conclusions by deviating from the truth. External validity, however, relates to whether the results can be generalized to other populations or settings, without necessarily affecting the effect estimate in the studied population.

A rigorous risk of bias assessment helps detect and mitigate systematic errors, ensuring reliable conclusions. It is also essential for distinguishing methodological bias from other factors such as imprecision or generalizability.

(Cochrane Handbook, section 7.1)

Types of bias

Bias in study design and conduct

• Randomization and allocation: Inadequate sequence generation or allocation concealment can lead to overestimation of intervention effects.
• Blinding: Lack of blinding (double blinding or blinding of outcome assessors) increases the risk of overestimating effects, particularly for subjective outcomes (e.g., pain).

Trial characteristics

• Sample size: Small studies are more likely to overestimate effects due to random variability and potential bias.
• Single-center vs. multicenter trials: Single-center trials often report larger effects, possibly due to selective reporting bias or less rigorous control.

Non-publication bias

• Publication bias: Studies with significant or positive results are more likely to be published.
• Time-lag bias: Positive results are often published more quickly than non-significant findings.
• Language bias: Positive results are more often published in English-language journals.
• Citation bias: Significant results are cited more frequently.
• Multiple publication bias: Studies with significant findings are more likely to be published in multiple formats.
• Location bias: Research published in widely accessible or well-indexed journals tends to present smaller or more reliable effect estimates.

Selective reporting

• Outcome reporting bias: Only favorable outcomes are reported, while non-significant outcomes may be omitted.
• Incomplete reporting: Results may be reported vaguely (e.g., “p > 0.05”) without detailed data.

These biases affect the validity of individual studies and the reliability of conclusions in systematic reviews or meta-analyses. Understanding and addressing them is crucial to ensure robust evidence synthesis.

(Cochrane Handbook, section 7.2)

To assess risk of bias, information may be collected from:

• Published articles: Commonly used, but often missing key methodological details.
• Trial registries: Provide information on unpublished studies but usually with limited methodological detail.
• Protocols: Offer more detailed information on study design and methods, increasingly available online.
• Clinical study reports: Highly detailed reports from pharmaceutical companies, valuable for bias assessment.
• Contacting investigators: Reaching out for clarification on missing or uncertain information can reduce response bias.

(Cochrane Handbook, section 7.3)

Here are some tools and repositories designed to assess risk of bias. This list is not exhaustive, and it is important to verify the appropriateness of the tools used to ensure rigorous evaluations.

Tools for risk of bias assessment

• Cochrane RoB 2 (Risk of Bias 2)
  • Designed to assess bias in randomized controlled trials (RCTs). It covers five domains (bias from randomization, deviations from intended interventions, missing outcome data, outcome measurement, and selective reporting). Provides a structured framework for evaluating bias and helps interpret synthesized results.
• ROBINS-I (Risk of Bias in Non-randomized Studies – of Interventions)
  • Assesses bias in non-randomized intervention studies (cohort, case-control, etc.). Suitable for identifying design- and analysis-related biases.
• AMSTAR 2 (A Measurement Tool to Assess Systematic Reviews)
  • Evaluates the methodological quality of systematic reviews. Includes 16 critical domains covering study selection, data extraction, bias assessment, heterogeneity, and publication bias.
• GRADE (Grading of Recommendations, Assessment, Development, and Evaluations)
  • A widely used framework to assess the certainty of evidence and strength of recommendations in systematic reviews. Considers risk of bias, inconsistency, indirectness, imprecision, and publication bias.

Repositories of appraisal tools

• Quality Assessment and Risk of Bias Tool Repository
  • A collection of diverse tools for assessing study quality and risk of bias, covering multiple study types with comprehensive evaluation criteria.
• Critical appraisal tools (Joanna Briggs Institute)
  • Checklists for assessing study quality across a wide range of study types (RCTs, cohort studies, qualitative research, etc.).
• CASP checklists (Critical Appraisal Skills Programme)
  • Structured checklists to critically appraise studies (RCTs, cohort, qualitative). Particularly useful for beginners.
• SIGN checklists (Healthcare Improvement Scotland)
  • Checklists designed to evaluate systematic reviews, RCTs, cohort studies, and other health research designs.
• Resources filtered to Appraisal (National Collaborating Centre for Methods and Tools)
  • A repository of filtered appraisal resources focused on tools for assessing the quality of research evidence, especially for evidence-informed decision-making.

Suggested Reading

Research Techniques Made Simple : Assessing Risk of Bias in Systematic Reviews

• Aaron M. Drucker, Patrick Fleming et An-Wen Chan (2016). Research Techniques Made Simple : Assessing Risk of Bias in Systematic Reviews, Journal of Investigative Dermatology, vol. 136, no 11, p. 109‑114. https://doi.org/10.1016/j.jid.2016.08.021 

Here are some key elements of synthesis, as presented in Chapter 9 of the Cochrane Handbook:

• Synthesis is the process of bringing together data from a set of included studies in order to draw conclusions across the evidence. This includes both the synthesis of study characteristics and, where appropriate, statistical synthesis of study results.
• A general framework for synthesis can guide the planning of comparisons, preparation for synthesis, conducting the synthesis, and interpreting and describing results.
• Tabulating study characteristics allows reviewers to examine and compare PICO elements across studies, supports the synthesis of these characteristics, and enables grouping of studies for statistical synthesis.
• Tabulating extracted data helps assess the number of studies contributing to a particular meta-analysis and informs the choice of other synthesis methods when meta-analysis is not possible.

Covidence Extraction Template

Here is how Covidence can support the synthesis process:

• Data extraction: Covidence enables reviewers to extract relevant data from studies, including study characteristics (e.g., PICO elements) and outcomes. Structured extraction supports comparison and synthesis of study results.
• Tabulation and organization: Covidence helps organize extracted data into structured tables that summarize key study characteristics. This facilitates the identification of patterns and differences across studies.
• Study grouping: Covidence’s filtering and tagging features allow reviewers to group studies by specific characteristics, which is helpful in preparing for statistical synthesis or subgroup analysis.
• Collaboration: Covidence supports multiple reviewers working together on data extraction and synthesis, ensuring consistency and transparency—both critical for reliable evidence synthesis.

Chapter 5 of the Cochrane Handbook also provides detailed guidance on data collection.

Qualitative data

Qualitative data are explored using narrative methods, such as thematic synthesis, to identify, group, and interpret key concepts and emerging themes across studies.

Software available at HEC:

• NVivo
• Atlas.ti

Quantitative data

Quantitative data are analyzed using statistical methods, such as meta-analysis, to aggregate numerical results and provide overall estimates of identified relationships or effects.

Support and software available at HEC:

• Mathematics and Statistics Help Centre
  • Mathematics, Excel, Python, MATLAB, Stata, R
• Data Processing and Analysis Laboratory (LACED)
  • MATLAB, SAS, Stata, SPSS, EViews, Scientific Workplace