Source: Foster, M. (2018). Systematic reviews service: Introduction to systematic reviews. Retrieved September 18, 2018, from
Volume 70, 2019, review article, how to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.
Article metrics loading...
Full text loading...
Literature Cited
Most cited most cited rss feed, job burnout, executive functions, social cognitive theory: an agentic perspective, on happiness and human potentials: a review of research on hedonic and eudaimonic well-being, sources of method bias in social science research and recommendations on how to control it, mediation analysis, missing data analysis: making it work in the real world, grounded cognition, personality structure: emergence of the five-factor model, motivational beliefs, values, and goals.
Created by health science librarians.
A simplified process map, how can the library help, publications by hsl librarians, systematic reviews in non-health disciplines, resources for performing systematic reviews.
Check our FAQ's
Email us
Call (919) 962-0800
Make an appointment with a librarian
Request a systematic or scoping review consultation
A systematic review is a literature review that gathers all of the available evidence matching pre-specified eligibility criteria to answer a specific research question. It uses explicit, systematic methods, documented in a protocol, to minimize bias , provide reliable findings , and inform decision-making. ¹
Before beginning a systematic review, consider whether it is the best type of review for your question, goals, and resources. The table below compares a few different types of reviews to help you decide which is best for you.
Systematic Review | Scoping Review | Systematized Review |
---|---|---|
Conducted for Publication | Conducted for Publication | Conducted for Assignment, Thesis, or (Possibly) Publication |
Protocol Required | Protocol Required | No Protocol Required |
Focused Research Question | Broad Research Question | Either |
Focused Inclusion & Exclusion Criteria | Broad Inclusion & Exclusion Criteria | Either |
Requires Large Team | Requires Small Team | Usually 1-2 People |
Systematic reviews follow established guidelines and best practices to produce high-quality research. Librarian involvement in systematic reviews is based on two levels. In Tier 1, your research team can consult with the librarian as needed. The librarian will answer questions and give you recommendations for tools to use. In Tier 2, the librarian will be an active member of your research team and co-author on your review. Roles and expectations of librarians vary based on the level of involvement desired. Examples of these differences are outlined in the table below.
Tasks | Tier 1: Consultative | Tier 2: Research Partner / Co-author |
---|---|---|
Guidance on process and steps | Yes | Yes |
Background searching for past and upcoming reviews | Yes | Yes |
Development and/or refinement of review topic | Yes | Yes |
Assistance with refinement of PICO (population, intervention(s), comparator(s), and key questions | Yes | Yes |
Guidance on study types to include | Yes | Yes |
Guidance on protocol registration | Yes | Yes |
Identification of databases for searches | Yes | Yes |
Instruction in search techniques and methods | Yes | Yes |
Training in citation management software use for managing and sharing results | Yes | Yes |
Development and execution of searches | No | Yes |
Downloading search results to citation management software and removing duplicates | No | Yes |
Documentation of search strategies | No | Yes |
Management of search results | No | Yes |
Guidance on methods | Yes | Yes |
Guidance on data extraction, and management techniques and software | Yes | Yes |
Suggestions of journals to target for publication | Yes | Yes |
Drafting of literature search description in "Methods" section | No | Yes |
Creation of PRISMA diagram | No | Yes |
Drafting of literature search appendix | No | Yes |
Review other manuscript sections and final draft | No | Yes |
Librarian contributions warrant co-authorship | No | Yes |
The following are systematic and scoping reviews co-authored by HSL librarians.
Only the most recent 15 results are listed. Click the website link at the bottom of the list to see all reviews co-authored by HSL librarians in PubMed
Researchers conduct systematic reviews in a variety of disciplines. If your focus is on a topic outside of the health sciences, you may want to also consult the resources below to learn how systematic reviews may vary in your field. You can also contact a librarian for your discipline with questions.
Environmental Topics
Social Sciences
Social Work
Software engineering
Sport, Exercise, & Nutrition
Updating reviews
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Email citation, add to collections.
Your saved search, create a file for external citation management software, your rss feed.
Affiliations.
Background: The role of evidence-based medicine in sports medicine and orthopaedic surgery is rapidly growing. Systematic reviews and meta-analyses are also proliferating in the medical literature.
Purpose: To provide the outline necessary for a practitioner to properly understand and/or conduct a systematic review for publication in a sports medicine journal.
Study design: Review.
Methods: The steps of a successful systematic review include the following: identification of an unanswered answerable question; explicit definitions of the investigation's participant(s), intervention(s), comparison(s), and outcome(s); utilization of PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines and PROSPERO registration; thorough systematic data extraction; and appropriate grading of the evidence and strength of the recommendations.
Results: An outline to understand and conduct a systematic review is provided, and the difference between meta-analyses and systematic reviews is described. The steps necessary to perform a systematic review are fully explained, including the study purpose, search methodology, data extraction, reporting of results, identification of bias, and reporting of the study's main findings.
Conclusion: Systematic reviews or meta-analyses critically appraise and formally synthesize the best existing evidence to provide a statement of conclusion that answers specific clinical questions. Readers and reviewers, however, must recognize that the quality and strength of recommendations in a review are only as strong as the quality of studies that it analyzes. Thus, great care must be used in the interpretation of bias and extrapolation of the review's findings to translation to clinical practice. Without advanced education on the topic, the reader may follow the steps discussed herein to perform a systematic review.
Keywords: PRISMA; PROSPERO; evidence-based medicine; meta-analysis; systematic review.
© 2013 The Author(s).
PubMed Disclaimer
Full text sources.
NCBI Literature Resources
MeSH PMC Bookshelf Disclaimer
The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .
Regina kunz.
1 German Cochrane Centre, Freiburg and Department of Nephrology, Charité, Berlin, Germany
2 Centre for Reviews and Dissemination, York, UK
3 German Cochrane Centre, Freiburg, Germany
Systematic reviews and meta-analyses are a key element of evidence-based healthcare, yet they remain in some ways mysterious. Why did the authors select certain studies and reject others? What did they do to pool results? How did a bunch of insignificant findings suddenly become significant? This paper, along with a book 1 that goes into more detail, demystifies these and other related intrigues.
A review earns the adjective systematic if it is based on a clearly formulated question, identifies relevant studies, appraises their quality and summarizes the evidence by use of explicit methodology. It is the explicit and systematic approach that distinguishes systematic reviews from traditional reviews and commentaries. Whenever we use the term review in this paper it will mean a systematic review . Reviews should never be done in any other way.
In this paper we provide a step-by-step explanation—there are just five steps—of the methods behind reviewing, and the quality elements inherent in each step (Box 1). For purposes of illustration we use a published review concerning the safety of public water fluoridation, but we must emphasize that our subject is review methodology, not fluoridation.
You are a public health professional in a locality that has public water fluoridation. For many years, your colleagues and you have believed that it improves dental health. Recently there has been pressure from various interest groups to consider the safety of this public health intervention because they fear that it is causing cancer. Public health decisions have been based on professional judgment and practical feasibility without explicit consideration of the scientific evidence. (This was yesterday; today the evidence is available in a York review 2 , 3 , identifiable on MEDLINE through the freely accessible PubMed clinical queries interface [ http://www.ncbi.nlm.nib.gov/entrez/query/static/clinical.html ], under ‘systematic reviews’.)
The research question may initially be stated as a query in free form but reviewers prefer to pose it in a structured and explicit way. The relations between various components of the question and the structure of the research design are shown in Figure 1 . This paper focuses only on the question of safety related to the outcomes described below.
Structured questions for systematic reviews and relations between question components in a comparative study
Box 1 The steps in a systematic review
The problems to be addressed by the review should be specified in the form of clear, unambiguous and structured questions before beginning the review work. Once the review questions have been set, modifications to the protocol should be allowed only if alternative ways of defining the populations, interventions, outcomes or study designs become apparent
The search for studies should be extensive. Multiple resources (both computerized and printed) should be searched without language restrictions. The study selection criteria should flow directly from the review questions and be specified a priori . Reasons for inclusion and exclusion should be recorded
Study quality assessment is relevant to every step of a review. Question formulation (Step 1) and study selection criteria (Step 2) should describe the minimum acceptable level of design. Selected studies should be subjected to a more refined quality assessment by use of general critical appraisal guides and design-based quality checklists (Step 3). These detailed quality assessments will be used for exploring heterogeneity and informing decisions regarding suitability of meta-analysis (Step 4). In addition they help in assessing the strength of inferences and making recommendations for future research (Step 5)
Data synthesis consists of tabulation of study characteristics, quality and effects as well as use of statistical methods for exploring differences between studies and combining their effects (meta-analysis). Exploration of heterogeneity and its sources should be planned in advance (Step 3). If an overall meta-analysis cannot be done, subgroup meta-analysis may be feasible
The issues highlighted in each of the four steps above should be met. The risk of publication bias and related biases should be explored. Exploration for heterogeneity should help determine whether the overall summary can be trusted, and, if not, the effects observed in high-quality studies should be used for generating inferences. Any recommendations should be graded by reference to the strengths and weaknesses of the evidence
Is it safe to provide population-wide drinking water fluoridation to prevent caries?
To capture as many relevant citations as possible, a wide range of medical, environmental and scientific databases were searched to identify primary studies of the effects of water fluoridation. The electronic searches were supplemented by hand searching of Index Medicus and Excerpta Medica back to 1945. Furthermore, various internet engines were searched for web pages that might provide references. This effort resulted in 3246 citations from which relevant studies were selected for the review. Their potential relevance was examined, and 2511 citations were excluded as irrelevant. The full papers of the remaining 735 citations were assessed to select those primary studies in man that directly related to fluoride in drinking water supplies, comparing at least two groups. These criteria excluded 481 studies and left 254 in the review. They came from thirty countries, published in fourteen languages between 1939 and 2000. Of these studies 175 were relevant to the question of safety, of which 26 used cancer as an outcome.
Design threshold for study selection.
Adequate study design as a marker of quality, is listed as an inclusion criterion in Box 1. This approach is most applicable when the main source of evidence is randomized studies. However, randomized studies are almost impossible to conduct at community level for a public health intervention such as water fluoridation. Thus, systematic reviews assessing the safety of such interventions have to include evidence from a broader range of study designs. Consideration of the type and amount of research likely to be available led to inclusion of comparative studies of any design. In this way, selected studies provided information about the harmful effects of exposure to fluoridated water compared with non-exposure.
After studies of an acceptable design have been selected, their in-depth assessment for the risk of various biases allows us to gauge the quality of the evidence in a more refined way. Biases either exaggerate or underestimate the ‘true’ effect of an exposure. The objective of the included studies was to compare groups exposed to fluoridated drinking water and those without such exposure for rates of undesirable outcomes, without bias. Safety studies should ascertain exposures and outcomes in such a way that the risk of misclassification is minimized. The exposure is likely to be more accurately ascertained if the study was prospective rather than retrospective and if it was started soon after water fluoridation rather than later. The outcomes of those developing cancer (and remaining free of cancer) are likely to be more accurately ascertained if the follow-up was long and if the assessment was blind to exposure status.
When examining how the effect of exposure on outcome was established, reviewers assessed whether the comparison groups were similar in all respects other than their exposure to fluoridated water. This is because the other differences may be related to the outcomes of interest independent of the drinking-water fluoridation, and this would bias the comparison. For example, if the people exposed to fluoridated water had other risk factors that made them more prone to have cancer, the apparent association between exposure and outcome might be explained by the more frequent occurrence of these factors among the exposed group. The technical word for such defects is confounding. In a randomized study, confounding factors are expected to be roughly equally distributed between groups. In observational studies their distribution may be unequal. Primary researchers can statistically adjust for these differences, when estimating the effect of exposure on outcomes, by use of multivariable modelling.
Put simply, use of a prospective design, robust ascertainment of exposure and outcomes, and control for confounding are the generic issues one would look for in quality assessment of studies on safety. Consequently, studies may range from satisfactorily meeting quality criteria, to having some deficiencies, to not meeting the criteria at all, and they can be assigned to one of three prespecified quality categories as shown in Table 1 . A quality hierarchy can then be developed, based on the degree to which studies comply with the criteria. None of the studies on cancer were in the high-quality category, but this was because randomized studies were non-existent and control for confounding was not always ideal in the observational studies. There were 8 studies of moderate quality and 18 of low quality.
Description of quality assessment of studies on safety of public water fluoridation
Prospective design | Prospective | Prospective | Prospective or retrospective |
Ascertainment of exposure | Study began within 1 year of fluoridation | Study began within 3 years of fluoridation | Study began >3 years after fluoridation |
Ascertainment of outcome | Follow-up for at least 5 years and blind assessment | Long follow-up and blind assessment | Short follow-up and unblinded assessment |
Control for confounding | Adjustment for at least three confounding factors (or use of randomization) | Adjustment for at least one confounding factor | No adjustment for confounding factors |
To summarize the evidence from studies of variable design and quality is not easy. The original review 3 provides details of how the differences between study results were investigated and how they were summarized (with or without meta-analysis). This paper restricts itself to summarizing the findings narratively. The association between exposure to fluoridated water and cancer in general was examined in 26 studies. Of these, 10 examined all-cause cancer incidence or mortality, in 22 analyses. Of these, 11 analyses found a negative association (fewer cancers due to exposure), 9 found a positive one and 2 found no association. Only 2 studies reported statistically significant differences. Thus no clear association between water fluoridation and increased cancer incidence or mortality was apparent. Bone/joint and thyroid cancers were of particular concern because of fluoride uptake by these organs. Neither the 6 studies of osteosarcoma nor the 2 studies of thyroid cancer and water fluoridation revealed significant differences. Overall no association was detected between water fluoridation and mortality from any cancer. These findings were also borne out in the moderate-quality subgroup of studies.
In the fluoridation example, the focus was on the safety of a community-based public health intervention. The generally low quality of available studies means that the results must be interpreted with caution. However, the elaborate efforts in searching an unusually large number of databases provide some safeguard against missing relevant studies. Thus the evidence summarized in this review is likely to be as good as it will get in the foreseeable future. Cancer was the harmful outcome of most interest in this instance. No association was found between exposure to fluoridated water and specific cancers or all cancers. The interpretation of the results may be generally limited because of the low quality of studies, but the findings for the cancer outcomes are supported by the moderate-quality studies.
After having spent some time reading and understanding the review, you are impressed by the sheer amount of published work relevant to the question of safety. However, you are somewhat disappointed by the poor quality of the primary studies. Of course, examination of safety only makes sense in a context where the intervention has some beneficial effect. Benefit and harm have to be compared to provide the basis for decision making. On the issue of the beneficial effect of public water fluoridation, the review 3 reassures you that the health authority was correct in judging that fluoridation of drinking water prevents caries. From the review you also discovered that dental fluorosis (mottled teeth) was related to concentration of fluoride. When the interest groups raise the issue of safety again, you will be able to declare that there is no evidence to link cancer with drinking-water fluoridation; however, you will have to come clean about the risk of dental fluorosis, which appears to be dose dependent, and you may want to measure the fluoride concentration in the water supply and share this information with the interest groups.
The ability to quantify the safety concerns of your population through a review, albeit from studies of moderate to low quality, allows your health authority, the politicians and the public to consider the balance between beneficial and harmful effects of water fluoridation. Those who see the prevention of caries as of primary importance will favour fluoridation. Others, worried about the disfigurement of mottled teeth, may prefer other means of fluoride administration or even occasional treatment for dental caries. Whatever the opinions on this matter, you are able to reassure all parties that there is no evidence that fluoridation of drinking water increases the risk of cancer.
With increasing focus on generating guidance and recommendations for practice through systematic reviews, healthcare professionals need to understand the principles of preparing such reviews. Here we have provided a brief step-by-step explanation of the principles. Our book 1 describes them in detail.
IMAGES
VIDEO
COMMENTS
Literature reviews establish the foundation of academic inquires. However, in the planning field, we lack rigorous systematic reviews. In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in literature ...
Screen the literature. Assess the quality of the studies. Extract the data. Analyze the results. Interpret and present the results. 1. Decide on your team. When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis.
Background. A systematic review, as its name suggests, is a systematic way of collecting, evaluating, integrating, and presenting findings from several studies on a specific question or topic.[] A systematic review is a research that, by identifying and combining evidence, is tailored to and answers the research question, based on an assessment of all relevant studies.[2,3] To identify assess ...
Abstract. Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in ...
Examples of literature reviews. Step 1 - Search for relevant literature. Step 2 - Evaluate and select sources. Step 3 - Identify themes, debates, and gaps. Step 4 - Outline your literature review's structure. Step 5 - Write your literature review.
The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information.
Registration can be done on platforms like PROSPERO 5 for health and social care reviews or Cochrane 3 for interventions. Step 3: search. In the process of conducting a systematic review, a well-organized literature search is a pivotal step.
A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer. Example: Systematic review. In 2008, Dr. Robert Boyle and his colleagues published a systematic review in ...
Image by TraceyChandler. Steps to conducting a systematic review. Quick overview of the process: Steps and resources from the UMB HSHSL Guide. YouTube video (26 min); Another detailed guide on how to conduct and write a systematic review from RMIT University; A roadmap for searching literature in PubMed from the VU Amsterdam; Alexander, P. A. (2020).
Documenting grey literature and/or hand searches. If you have also searched additional sources, such as professional organization websites, cited or citing references, etc., document your grey literature search using the flow diagram template version 1 PRISMA 2020 flow diagram for new systematic reviews which included searches of databases, registers and other sources or the version 2 PRISMA ...
A preliminary review, which can often result in a full systematic review, to understand the available research literature, is usually time or scope limited. Complies evidence from multiple reviews and does not search for primary studies. 3. Identifying a topic and developing inclusion/exclusion criteria.
Method details Overview. A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12].An SLR updates the reader with current literature about a subject [6].The goal is to review critical points of current knowledge on a ...
This article provides a step-by-step approach to conducting and reporting systematic literature reviews (SLRs) in the domain of healthcare design and discusses some of the key quality issues associated with SLRs. SLR, as the name implies, is a systematic way of collecting, critically evaluating, integrating, and presenting findings from across ...
Systematic Review Components. Starts with a clearly articulated question. Uses explicit, rigorous methods to identify, critically appraise, and synthesize relevant studies. Appraises relevant published and unpublished evidence for validity before combining and analyzing data. Reports methodology, studies included in the review, and conclusions ...
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to ...
Systematic reviews: Structure, form and content. This article aims to provide an overview of the structure, form and content of systematic reviews. It focuses in particular on the literature searching component, and covers systematic database searching techniques, searching for grey literature and the importance of librarian involvement in the ...
A Step-by-Step Guide. ospital of Philadelphia, Philadelphia, PAIntroductionA systematic review attempts to comprehensively and reproducibly collect, appraise, and synthesize all available empirical evidence that meets pre-d. fined criteria in order to answer a research question. The quantitative combination and statistical synthesis of the systema.
Systematic literature reviews (SRs) are a way of synthesising scientific evidence to answer a particular ... The issue with such reviews is that they leave it up to the expert author to decide what should be included or not, and do not allow readers to track and ... was established to support more standardised SR protocol writing (Moher et al ...
A systematic review is a literature review that gathers all of the available evidence matching pre-specified eligibility criteria to answer a specific research question. It uses explicit, systematic methods, documented in a protocol, to minimize bias, provide reliable findings, and inform decision-making.
Our goal with this paper is to conduct a narrative review of the literature about systematic reviews and outline the essential elements of a systematic review along with the limitations of such a review. Keywords: systematic reviews, meta-analysis, narrative literature review, prisma checklist. Go to: A literature review provides an important ...
he simplest thing of all—structure. Everything you write has three components: a beginning, a middle and an e. d and each serves a different purpose. In practice, this means your review will have an introduction, a main body where you review the literature an. a conclusion where you tie things up.
Background: The role of evidence-based medicine in sports medicine and orthopaedic surgery is rapidly growing. Systematic reviews and meta-analyses are also proliferating in the medical literature. Purpose: To provide the outline necessary for a practitioner to properly understand and/or conduct a systematic review for publication in a sports medicine journal.
Reasons for inclusion and exclusion should be recorded. Step 3: Assessing the quality of studies. Study quality assessment is relevant to every step of a review. Question formulation (Step 1) and study selection criteria (Step 2) should describe the minimum acceptable level of design.