Doing A Systematic Review of the Literature
How to Locate, Appraise and Perform a Systematic Review of the Literature
A Systematic Review is a summary that attempts to address a focused clinical question using methods designed to reduce the likelihood of bias. Systematic reviews provide a summary of the available literature using "explicit methods to systematically search, critically appraise, and synthesize the world literature on a specific issue" (Straus et al, 2005). A meta-analysis is a type of systematic review that uses statistical, quantitative methods to describe the results.
Unlike simple review articles, a systematic review is based on a systematic review of the evidence and requires an exhaustive search of the literature. Review articles are more narrative and, while a good way to start your research or learn about a topic, do not follow the strict methodology demanded of a systematic review. A systematic review uses numerical or cross-study analysis to synthesize the findings of original studies and are more clinically based. Systematic reviews must clearly state why the research is being done and what methods were used to find the primary studies. These methods must be described so that the search process can be replicated.
Systematic reviews often require a significant time investment. Allen and Olkin, in 1999, reported 1139 hours were required for a review, on average, though the time investment was often dependent on the number of citations retrieved for a search.
Here is a section of the Methods part of a paper on Positive End-Expiratory Pressure in patients with acute lung injury. By using the strategy outlined below, the search could be replicated by the reader.
Levels of Evidence
A Systematic Review (and Meta-Analysis) is considered to be the pinnacle of research. Clinical questions that can be answered using Systematic Reviews would be considered to have more validity than questions that can only be answered using case reports or animal research.
Pyramid courtesy of SUNY
Steps in performing a Systematic Review: The following steps outline how to conduct a systematic review (from Wright et al, 2007).
To begin, researchers define the focus of the review using a research question. Research questions are usually created using the PICO components (for more information on PICO, please read the next section). Beginning with a precise, well-reasoned research question will reduce the amount of time required to retrieve relevant literature citations and will also reduce bias compared to approaches in which the question is shaped in response to literature already reviewed by the authors. The type of question (e.g. Therapy, Prevention, Prognosis, Test/Diagnosis, Etiology) will often lend itself to a specific type of article (e.g. a Randomized Clinical Trial, cohort studies, Case Control Trial). Reviewers should identify what level of evidence is appropriate for the type of question they are asking.
Next, the reviewers should outline the methods they will use for the search. A written document should outline what methods and terms will be used to search the literature, extract data, and the inclusion and exclusion criteria. The quality of the literature that is selected for inclusion will impact the quality of the final review. For example, cohort and case studies are considered a much lower quality of evidence than RCTs (see the Pyramid of Evidence, above). Your search strategy may use indexing terms or keywords.
Reviewers should now conduct a literature search. This is an exhaustive process, requiring the use of several search strategies, that results in a comprehensive list of potentially relevant studies. The use of a bibliographic management software (e.g. EndNote, ProCite, Referance Manager) can be crucial for managing the results of this step. Processes appropriate for a literature search include:
- Search multiple, general and specific bibliographic databases (MEDLINE (e.g. PubMed or Ovid MEDLINE), EMBASE, PsychInfo, Current Contents Connect, DARE, Cochrane, PubMed Clinical Queries, etc.)
- Use Science Citation Index or Social Science Citation Index (Cited References) to see which studies have cited an article (or study)
- Hand search specific journals
- Hand search conference proceedings and meeting abstracts
- Contact experts in the field
- Contact professional organizations or governmental agencies (i.e. US Center for Disease Control and Prevention)
- Contact manufacturers
- Scan bibliographies of review articles, selected studies, book chapters, clinical practice guidelines, and theses
- Use the PubMed “Related Articles” feature
Once the studies are obtained, data from each study is put into tabular format to manage the range of studies included and provide raw data for statistical calculations (in step 6). The items extracted often include:
- Reference (citation)
- Objective of the study
- Type of Trial
- Control or Comparison
Next, reviewers should evaluate the quality of the studies found for quality and applicability. Quality factors may include, but are not limited to:
- Patient numbers and characteristics at baseline
- Study design/type of trial
- Follow-up duration and numbers
- Outcomes and how they are defined and assessed
- Risk factors
- Adverse effects
- Year of publication
Using both common sense and statistical computations, decide whether the data are similar enough to be combined statistically and mathematically. If the data are not similar enough to combine, no further statistical analysis should be done. If the data can be combined, the systematic review article becomes a “meta-analysis”. (A meta-analysis is a subcategory of a systematic review.)
Last, reviewers will need to display their results in a coherent manner. Base your conclusions on the best evidence that is available from your review. Systematic reviews include data in several different formats. The raw data are presented in a table form and are accompanied with text. You will also want to make recommendations for future studies, particularly by identifying areas which have not had sufficient study.
Using PICO to structure a Question:
Finding Evidence-Based Literature:
Appraising Systematic Reviews:
Not all systematic reviews are of the same quality. As a creator of systematic reviews, or as a consumer, keep in mind the following guidelines for appraising a systematic review:
- Does the article address a focused question? Was the question well-defined and described in the beginning of the article?
- Were all relevant studies included? Did the authors describe the databases searched along with the search strategy used?
- Were the inclusion criteria used to select articles appropriate? These criteria may vary according to the population studied, interventions, outcomes, and methods of each study. Did the authors state inclusion (or exclusion) criteria along with an explanation of their rationale?
- Was the validity of the included studies assessed? The pyramid of evidence can be used to assess the quality of the studies included. A double-blind randomized controlled study is more valid than a study that is not double-blind. Was the follow-up of patients sufficiently long and complete?
- Were the assessments of the studies reproducible? Even when explicit criteria are used to include studies in a review and evaluate their quality, the judgment of the review’s authors is still required. If the authors performed each of the review steps independently and in duplicate and then reported their level of agreement, one can access how open to judgment each of the steps was.
- Were the results similar from study to study? Synthesizing the results of studies requires assessing the similarity of the studies to each other. Patients (or populations), interventions (or exposures), and outcomes must be considered. If all of the studies appear to be similar in the initial assessment, it is then important to evaluate whether the results are similar.
For further information, contact the Reference Librarian. Last Updated:09/17/2010