This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License (https://creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).
This article aims to provide an overview of the structure, form and content of systematic reviews. It focuses in particular on the literature searching component, and covers systematic database searching techniques, searching for grey literature and the importance of librarian involvement in the search. It also covers systematic review reporting standards such as PRISMA-P and PRISMA, critical appraisal and tools and resources to support the review and ensure it is conducted efficiently and effectively. Finally, it summarizes the requirements when screening search results for inclusion in the review, and the statistical synthesis of included studies’ findings.
Keywords: Libraries, PRISMA, PRISMA-P, Research support, Systematic reviewsProvenance and Peer review: Solicited contribution; Peer reviewed; Accepted for publication 24 January 2021.
A systematic review collects secondary data, and is a synthesis of all available, relevant evidence which brings together all existing primary studies for review (Cochrane 2016). A systematic review differs from other types of literature review in several major ways. It requires a transparent, reproducible methodology which indicates how studies were identified and the criteria upon which they were included or excluded. As well as synthesis of these studies' findings, there should be an element of evaluation and quality assessment. The systematic review methodology originated in medical and healthcare research, but it has now been adopted by other disciplines, such as engineering, education, economics and business studies. The processes and requirements for conducting a systematic review can seem arduous or time consuming, but with the use of appropriate tools and resources, and with thorough planning undertaken before beginning the review, researchers will be able to conduct their systematic reviews efficiently and smoothly.
This article provides an overview of the structure, form and content of systematic reviews, with a particular focus on the literature searching component. It will also discuss tools and resources – including those relating to reporting standards and critical appraisal of the articles included in the review – which will be of use to researchers conducting a systematic review.
In recent years, there has been an explosion in the number of systematic reviews conducted and published (Chalmers & Fox 2016, Fontelo & Liu 2018, Page et al 2015) – although a systematic review may be an inappropriate or unnecessary research methodology for answering many research questions. Systematic reviews can be inadvisable for a variety of reasons. It may be that the topic is too new and there are not enough relevant published papers to synthesise and analyse for a systematic review, or, conversely, that many other researchers have already published systematic reviews on the topic. However, if a scoping search appears to yield sufficient relevant studies for evidence synthesis, and indicates that no previous systematic reviews have been published (or that those previously published require an update or have methodological flaws), systematic reviews are likely to be appropriate.
Most systematic reviews take between six and 18 months to complete, and require a minimum of three authors to independently screen search results. Although many university modules require students to complete systematic reviews, due to this time and authorship requirement, it would be better to describe such student reviews as ‘reviews with systematic literature searches,’ as it is not possible to fulfil all the methodological requirements of a systematic review in a piece of work with a single author. Researchers without the available time or number of potential co-authors may prefer to adopt a different approach, such as narrative, scoping, or umbrella reviews. The systematic, transparent searching techniques outlined in this article can be adopted and adapted for use in other forms of literature review (Grant & Booth 2009), for example, while the critical appraisal tools highlighted are appropriate for use in other contexts in which the reliability and applicability of medical research require evaluation.
Once it has been determined that a systematic review is the appropriate methodology for the research, and that there is sufficient time and resources to conduct it, researchers should then spend some time developing their review topic. It is appropriate at this point to do some scoping searches in relevant subject databases, first to ensure that the proposed review is unique, and meets a research need, and second to obtain a broad overview of the literature that exists, and which is likely to be included in the eventual systematic review. Based on this scoping work, the review topic may need to be refined or adapted, possibly to broaden or narrow it in focus. Once reviewers are satisfied with their chosen topic, the next step is to prepare a protocol which states transparently the methodology they intend to follow when conducting their review.
A protocol is a description of the proposed systematic review, including methods, the rationale for the review, and steps which will be taken to eliminate bias while conducting the review. Registering the protocol stakes a claim on the research, and it also means that researchers have done a significant portion of the work required before they formally begin the review, as they will have written the Methods section in draft form and planned what will be necessary to document and report by the time the protocol is finished.
Most protocols are registered with PROSPERO (2020), although it is also possible to upload your protocol on an institutional or subject repository, or publish the protocol in a journal. Guidance for creating a protocol can be found at PRISMA-P (The PRISMA Group et al 2015), or by working through the online training on protocols available at the Cochrane Library (Cochrane Interactive Learning 2019).
PRISMA (the Preferred Reporting Items for Systematic Reviews and Meta-analyses) is 'an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses' (Moher et al 2009). The PRISMA checklist is a useful guideline of content that should be reported and included in the final published version of the systematic review, and will help when in the planning stages as well. Most systematic reviews will be written up using the PRISMA checklist as their underlying structure, so familiarity with this checklist and the content required when reporting the findings of the systematic review should be established at the earliest planning stages of the research.
PRISMA-P (The PRISMA Group et al 2015) is the reporting guidelines for protocols. The EQUATOR Network lists reporting standards for multiple different types of study design (EQUATOR Network 2020). Researchers can search for the right guideline for their type of study. Those undertaking a Cochrane review should select the correct Cochrane Handbook (Cochrane Training 2020) for their review type.
The search strategy for systematic reviews is the main method of collecting the data which will underpin the review's findings. This means that the search must be sufficiently robust – both sensitive and specific – to capture all relevant articles. Ideally, multiple databases and other sources of information should be searched, using a consistent, predetermined search string. Generally, this will involve multiple synonyms for each theme of the review's topic, and a multifield search including freetext terms in (at minimum) the title and abstract, and the controlled vocabulary in the database thesaurus. These words are then combined with the Boolean operators AND, OR and NOT so that search results are both sensitive and specific.
It is likely that systematic reviews will need to include a search of grey literature as well as the peer-reviewed journal articles found through database searching. Grey literature includes unpublished theses, conference proceedings, government reports, unpublished trial data and more. Leaving grey literature out can run the risk of biasing the reviews results (Goldacre 2011).
Searching grey literature can be challenging. Most sources of grey literature cannot be searched with complex Boolean operators and myriad synonymous keywords in the manner of a database. Likewise, the websites and other sources used to search for grey literature are unlikely to have a controlled vocabulary thesaurus. The Canadian Agency for Drugs and Technologies in Health (CADTH) tool is designed to help adapt complex systematic database search strategies for use when searching for grey literature (CADTH 2009).
Sometimes it may be appropriate to 'snowball' a search. This involves screening all the articles that cite included papers (the articles which meet the inclusion criteria after screening). Search for the titles of each included article in Web of Science or Scopus (or both), and any listed citing article which meets your inclusion criteria should also be included in the review.
Hand searching involves looking back through the tables of contents of key journals, conference proceedings, or lists of conference presentations relevant to the systematic review topic. Once key journals have been identified, reviewers should plan how many years back they will look – this will need to be done consistently across all journals that are hand-searched.
After reviewers have screened all the papers identified by the database and grey literature searches, and agreed on which will be included in the review, they should check through these articles' reference lists. Any articles in their reference lists which meet all inclusion criteria should also be included in the review.
There is some evidence that having a librarian co-author on a systematic review can improve the review's quality. A number of recent studies have indicated that librarian involvement improves the reproducibility of the literature searching (Hameed et al 2020, Koffel 2015, Rethlefsen et al 2015). Reviews without librarian involvement often have problems with their search strategies – for example Boolean operators used incorrectly, inappropriate search syntax, or a lack of sufficient synonyms for each search term, meaning that relevant studies might be missed (Golder et al 2008, Li et al 2014). Unfortunately, in some instances, systematic reviews without librarian co-authors will still be published, even if their search strategies have significant methodological flaws (Brasher & Giustini 2020). Librarian involvement will help ensure that the search strategy is robust, and that it is described accurately in the methodology to ensure that the systematic review is reproducible. Generally, if a librarian is developing the search terms, running the searches in databases and writing the search methods, they should be a co-author of the systematic review, whereas if the librarian supports researchers who then conduct the searches themselves, co-authorship is not necessary. This also aligns with the Vancouver recommendations on co-authorship (International Committee of Medical Journal Editors 2019).
After database and grey literature searches are completed, and researchers have identified other papers through hand-searching, they will need to screen the titles and abstracts to determine if they meet the criteria for inclusion. These criteria should be pre-defined (ie: stated in the protocol before searches have begun). Inclusion criteria might relate to the following:
Date range of publication.
Study design type.
Whether a study focuses on the review's specific disease, condition, or patient population.
Whether a study focuses mainly on the review's specific intervention.
Whether a study focused on a certain country, region, or healthcare context (for example primary care, outpatient department, critical care unit, or similar).
This list is not exhaustive, and there are many other inclusion criteria to apply, depending on the scope of the topic of the systematic review. It is important that these criteria are stated clearly in the Methods section of both the protocol and systematic review, and that all co-authors understand them.
Generally, articles are screened against these criteria independently by at least two authors. Initially they should screen the titles and abstracts, and then move on to screening the full text for any articles which could not be judged as fulfilling (or not fulfilling) all inclusion criteria on the basis of the information in their titles and abstracts.
Referencing software such as Endnote, EndnoteWeb, Mendeley or Zotero can be used for screening, or reviewers may prefer to use systematic review screening software such as Covidence or Rayyan.
There are a number of tools and checklists available to help assess the quality of studies to be included in a review. Studies included in a systematic review should be assessed for their quality and reliability. While poor quality studies should not be excluded if they fulfil predefined inclusion criteria, the systematic review should make clear that all included studies have been assessed according to consistent principles of critical appraisal, and the results of that appraisal should be included in the review.
Most critical appraisal tools consist of different checklists to apply to different types of study design. If a systematic review includes multiple types of study design, it is advisable that researchers are consistent about which tools they use – it is preferable to use different checklists from a single source, rather than picking and choosing from a variety of sources.
If the systematic review is only including peer-reviewed, published journal articles, the checklists from either CASP (Critical Appraisal Skills Programme), Centre for Evidence-Based Medicine, SIGN (Scottish Intercollegiate Guidelines Network), or Joanna Briggs Institute will be appropriate (Brice 2020, Centre for Evidence-Based Medicine 2020, Joanna Briggs Institute 2020, SIGN 2020). Reviews which include grey literature should use a grey literature appraisal tool, such as AACODS (Tyndall 2008). There are also risk of bias assessment tools, such as RoBiS for evaluating systematic reviews, and RoB 2 for evaluating randomized controlled trials (Bristol Medical School 2020, Sterne et al 2019).
One of the main advantages of systematic reviews is that they combine the analysis of the data from a number of primary studies. Most commonly, this is done through meta-analysis – the statistical combination of results from two or more studies. As outlined in the Cochrane Handbook, in interventional studies, a systematic review meta-analysis will seek to answer these three main questions:
What is the direction of effect?
What is the size of effect?
Is the effect consistent across [all included] studies? (Higgins et al 2019)
The researchers will then make a judgement as to the strength of evidence for the effect. If the systematic review is assessing the effectiveness of a variety of different interventions, it may not be possible to combine all studies for meta-analysis as the studies may be sufficiently different to make meta-analysis inappropriate. Researchers should ensure that when interpreting the results they consider the limitations and potential biases of included studies. When reporting the findings it is also usually necessary to consider applicability, and make recommendations – such as for a change in practice.
Systematic reviews – when an appropriate approach to the topic being researched – are a way to synthesize and evaluate the range of evidence available in multiple primary studies. Their methodology is complex, but if the correct reporting guidelines are followed, and researchers make use of tools, resources and the support of librarians and other information specialists, the process will be more straightforward. Planning is key: researchers should have a clear picture of what is involved, and what will need to be documented and reported in any resulting publications, and put measures in place to ensure that they capture all of this essential information.
No competing interests declared.
Brasher PMA, Giustini D. Rapid response: Effect of breakfast on weight and energy intake: Systematic review and meta-analysis of randomised controlled trials. British Medical Journal . 2020; 364 :142. [Google Scholar]
Brice R.2020. CASP CHECKLISTS CASP – Critical Appraisal Skills Programme. Available at: https://casp-uk.net/casp-tools-checklists/ (Accessed 30 October 2020)
Canadian Agency for Drugs and Technologies in Health (CADTH) 2009Grey matters: A practical tool for searching health-related grey literature. Available at: https://www.cadth.ca/resources/finding-evidence/grey-matters (Accessed 30 October 2020)
Centre for Evidence-Based Medicine 2020Critical Appraisal Tools. Available at: https://www.cebm.ox.ac.uk/resources/ebm-tools/critical-appraisal-tools (Accessed 30 October 2020)
Chalmers I, Fox DM.2016 Increasing the incidence and influence of systematic reviews on health policy and practice American Journal of Public Health 106 11–13 [PMC free article] [PubMed] [Google Scholar]
Cochrane 2016What are systematic reviews? Available at: https://www.youtube.com/watch?v=egJlW4vkb1Y&feature=emb_logo (Accessed 11 February 2021)
Cochrane Interactive Learning 2019Module 2: Writing the review protocol. Available at: https://training.cochrane.org/interactivelearning/module-2-writing-review-protocol (Accessed 23 September 2020)
Cochrane Training 2020Guides and handbooks. Available at: https://training.cochrane.org/handbooks (Accessed 23 September 2020)
Fontelo P, Liu F. A review of recent publication trends from top publishing countries. Systematic Reviews. 2018; 7 :147. [PMC free article] [PubMed] [Google Scholar]
Goldacre B.2011. Battling bad science.Golder S, Loke Y, Mcintosh HM.2008 Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects Journal of Clinical Epidemiology 61 440–448 [PubMed] [Google Scholar]
Grant MJ, Booth A.2009 A typology of reviews: An analysis of 14 review types and associated methodologies Health Information & Libraries Journal 26 91–108 [PubMed] [Google Scholar]
Hameed I, Demetres M, Tam DY, et al. 2020 An assessment of the quality of current clinical meta-analyses BMC Medical Research Methodology 20 105. [PMC free article] [PubMed] [Google Scholar]
Higgins JPT, Thomas J, Chandler J, et al. 2019Cochrane Handbook for Systematic Reviews of Interventions London, The Cochrane Collaboration
International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Annals of Internal Medicine . 2019 [PubMed] [Google Scholar]
Joanna Briggs Institute 2020Critical Appraisal Tools. Available at: https://joannabriggs.org/critical-appraisal-tools (Accessed 30 October 2020)
Koffel JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: A cross-sectional survey of recent authors. PLoS One . 2015; 10 :e0125931. [PMC free article] [PubMed] [Google Scholar]
Li L, Tian J, Tian H, et al. 2014 Network meta-analyses could be improved by searching more sources and by involving a librarian Journal of Clinical Epidemiology 67 1001–1007 [PubMed] [Google Scholar]
Moher D, Liberati A, Tetzlaff J, Altman DG, The Prisma Group 2009 Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement PLoS Medicine 6 e1000097. [PMC free article] [PubMed] [Google Scholar]
National Institute for Health Research 2020PROSPERO. Available at: https://www.crd.york.ac.uk/prospero/ (Accessed 30 October 2020)
Page MJ, Shamseer L, Alman DG, et al. Epidemiology and reporting characteristics of systematic reviews: 2014 update | Colloquium Abstracts. Reward/Equator Conference, 2015, Edinburgh
Rethlefsen ML, Farrell AM, Trzasko LCO, Brigham TJ.2015 Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews Journal of Clinical Epidemiology 68 617–626 [PubMed] [Google Scholar]
Sterne JAC, Savović J, Page MJ, et al. 2019 RoB 2: A revised tool for assessing risk of bias in randomised trials British Medical Journal 366 l4898. [PubMed] [Google Scholar]
The Prisma Group, Moher D, Shamseer L, et al. 2015 Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement Systematic Reviews 4 1. [PMC free article] [PubMed] [Google Scholar]