How to read a systematic review.
Over the past year I have blogged several times on systematic reviews and I have occasionally been critical of the methods adopted by some investigators. It seems that the number of published reviews is increasing. We are also coming under pressure to keep up with a large amount of published information of various quality. As a result, I thought that it would be helpful if I wrote a short blog on “how to read a systematic review”. This is a post that I first published several years ago and I thought that it was worth revisiting. I have targetted this at orthodontists, but I hope that it is relevant to all dental care providers.
The first step
Firstly, you need to make an overall assessment of the quality of the systematic review. My feeling is that Cochrane systematic reviews tend to provide the most useful information on clinical questions. This is because of the strict editorial and methodological steps that need to be taken to satisfy the Cochrane editors. I have now completed several Cochrane reviews and I think that they have the toughest editorial control than any other publisher!
Another important criteria that set Cochrane reviews apart from others is that the authors are committed to updating the review periodically. In theory, this results in findings changing as new research is published. One example of this is the Cochrane review of Class II treatment that I helped produce. As additional evidence became available on the effect of early treatment on trauma, the conclusions changed.
I would like to point out that I am not suggesting that other reviews are not of value, but we need to bear in mind the overall quality of a review when we evaluate this form of literature.
Now that I have set out the context of the review, I will move to the main features that I look at when I read a review. This list is not exhaustive it is simply a set of tips that I hope are useful to you. I would like to use the review on Temporary Anchorage Devices that I blogged about previously as an example.
Check the inclusion criteria.
The systematic review should clearly outline the inclusion criteria. The authors should state whether the studies that they included were RCTs and/or other types of study. In some reviews, RCTs and Controlled clinical trials are accepted. However, other reviews contain low-quality studies, for example, retrospective designs with historic or convenience controls that are characterised by selection bias. When reading a review that includes retrospective studies you need to appreciate that the strength of any findings is diminished. If the authors put poor quality data in, they will get poor quality data out!
How were the papers selected?
Look closely at this section. In a good review, the authors will provide a flow chart of how they obtained their papers. Ideally, they should provide information on each paper that they included and excluded.
The meta-analysis and Forest Plot
The analysis of the data derived from the included papers is frequently presented as a Forest plot. This is a clear way of presenting the data, but it can at first glance be confusing. I will go through an example Forest Plot from the TADS review. This is the plot that illustrates the effectiveness of palatal implants and TADS compared to conventional forms of anchorage.
I have concentrated on the results for TADS. When you look at this plot I have highlighted the relevant sections in red.
This figures shows that on the left hand side of the Forest plot is a summary of the data on sample size etc from each included study. You can look at this to build a picture of the number of studies and the subjects enrolled.
Now look at the data for each study, this shows the effect size and 95% confidence intervals.
In this figure I have highlighted the graphics around the vertical line. This is the line of “no effect”. Now look at the highlighted graphics that are dispersed around the line. These repretesent the means and confidence intervals for each study. If the CI line crosses the line of no effect, then there is no statistically significant effect of the intervention.
Now look at the trapezoid shaped figure at the bottom of the plot. This represents the combined data. The vertical axis is the mean and the horizontal is the confidence intervals. If the shape crosses the line, the difference is not statistically significant. You can see, in this example, that the shape does not cross the line and reveals that TADS are more effective than other methods of anchorage.
Finally, look at the bottom right numbers, this is the combined effect size and confidence intervals. Even if the result is statistically significant, you need to evaluate whether this is clinically significant. I have outlined this previously in this blog.
Strength of recommendations
Many reviews now include a statement on the strength of the recommendations that can be made from the review. Several now use the GRADE approach. I am not going to go through this in detail but I have included a brief outline of these in this table.
| Quality level | Definition |
|---|---|
| High | We are very confident that the true effect is close to the estimate of the effect |
| Moderate | We are moderately confident in the effect estimate |
| Low | Our confidence is the effect estimate is limited |
| Very low | We have little confidence in this estimate |
You will see that they are concerned with looking at the level of confidence in the effect size included in their review. In the TADS review, the effect size was -2.17 mm. We concluded that we had moderate confidence in the findings.
Summary
I have tried to keep this post as concise as possible. As a result, I have only outlined the features that I evaluate when I read a systematic review, I have not mentioned other features that are important and I appreciate that some people may criticise this rather basic approach. However, realistically this is all that I do when I read the literature. I hope that you feel that this is useful.
Emeritus Professor of Orthodontics, University of Manchester, UK.
Very useful, thank you.
nicely summarized
I do agree. There has been some inflation in systematic reviews and what we need instead is clinical research of good quality, instead of systematic reviews of questionable quality that repeatedly shows us what we already know i.e. we lack a lot of knowledge.
never really taught this in uni. which is ludicrous considering we have to be critical thinkers.