How to read a systematic review: Some tips and tricks

How I read a systematic review

Over the past year I have blogged several times on systematic reviews and I have occasionally been critical of the methods adopted by some investigators. It seems that the number of published reviews is increasing and I feel that we are coming under pressure to keep up with the large amount of information that is coming our way. As a result, I thought that it would be helpful if I wrote a short blog on “how to read a systematic review”. This is targeted, to some degree, at orthodontists, but I hope that it is relevant to all dental care providers.

The first step

This is to make an overall assessment of the quality of the review. My feeling is that Cochrane systematic reviews tend to provide the most useful information on clinical questions. This is because of the strict editorial and methodological steps that need to be taken to satisfy the Cochrane editors. I have now completed several Cochrane reviews and I think that they have the toughest editorial control than any other publisher!

Another important criteria that sets Cochrane reviews apart from others is that the authors are committed to updating the review periodically. In theory, this results in the finding changing as new research is published. One example of this is the Cochrane review of Class II treatment that I helped produce. As additional evidence became available on the effect of early treatment on trauma, the conclusions changed.

I would like to point out that I am not suggesting that other reviews are not of value, but we need to bear in mind the overall quality of a review when we evaluate this form of literature.

Now that I have set out the context of the review, I will move to the main features that I look at when I read a review. This list is not exhaustive it is simply a set of tips that I use and I hope are useful to you.  I would like to use  the recent review on Temporary Anchorage Devices that I blogged about last week as an example.

Check the inclusion criteria.

The review should clearly outline the inclusion criteria. The authors should state whether the studies that they included were RCTs and/or other types of study. In some reviews RCTs and Controlled clinical trials are accepted. However, other reviews contain low quality studies, for example, retrospective designs with historic or convenience controls that are characterised by selection bias. When reading a review that includes retrospective studies you need to appreciate that the strength of any findings is diminished.

How were the papers selected?

Look closely at this section. In a good review the authors will provide a flow chart of how they obtained their papers. Ideally they should provide information on each paper that they included and excluded.

The meta analysis and Forest Plot

The analysis of the data derived from the included papers is frequently presented as a Forest plot. This is a clear way of presenting the data, but it can at first glance be confusing. I will go through an example Forest Plot from the  TADS review. This is the plot that illustrates the effectiveness of palatal implants and TADS compared to conventional forms of anchorage.

I have concentrated on the results for TADS. When you look at this plot I have highlighted the relevant sections in red. If you click on each one they should expand so that you can read them more easily.

This figures shows that on the left hand side of the Forest plot is a summary of the data on sample size etc from each included study. You can look at this to build a picture of the number of studies and the subjects enrolled.

Now look at the data for each study, this shows the effect size and 95% confidence intervals.

In this figure I have highlighted the graphics around the vertical line. This is the line of “no effect”. Now look at the graphics that are dispersed around the line. The upper ones are the means and confidence intervals for each study. If the CI line crosses the line of no effect, then there is no statistically significant effect of the intervention.

Now look at the trapezoid shaped figure at the bottom of the plot. This represents the combined data. The vertical axis is the mean and the horizontal is the confidence intervals. If the shape crosses the line, the the difference is not statistically significant. You can see, in this example, that the shape does not cross the line and reveals that TADS are more effective than other methods of anchorage.

Finally, look at the bottom right numbers, this is the combined effect size and confidence intervals. Even if the result is statistically significant, you need to evaluate whether this is clinically significant. I have outlined this previously in this blog.

Strength of recommendations

Many reviews now include a statement on the strength of the recommendations that can be made from the review. Several now use the GRADE approach. I am not going to go through this in detail but I have included a brief outline of these in this table.

You will see that they are concerned with looking at level of confidence in the effect size included in they review. In the TADS review the effect size was -2.17 mm. We concluded that we had moderate confidence in the findings.

 Summary

I have tried to keep this post as concise as possible. As a result, I have only outlined the features that I evaluate when I read a systematic review, I have not mentioned other features that are important and I appreciate that some people may criticise this rather basic approach. However, realistically this is all that I do when I read the literature. I hope that you feel that this is useful.