How to read a systematic review: Some tips and tricks
How I read a systematic review
Over the past year I have blogged several times on systematic reviews and I have occasionally been critical of the methods adopted by some investigators. It seems that the number of published reviews is increasing and I feel that we are coming under pressure to keep up with the large amount of information that is coming our way. As a result, I thought that it would be helpful if I wrote a short blog on “how to read a systematic review”. This is targeted, to some degree, at orthodontists, but I hope that it is relevant to all dental care providers.
The first step
This is to make an overall assessment of the quality of the review. My feeling is that Cochrane systematic reviews tend to provide the most useful information on clinical questions. This is because of the strict editorial and methodological steps that need to be taken to satisfy the Cochrane editors. I have now completed several Cochrane reviews and I think that they have the toughest editorial control than any other publisher!
Another important criteria that sets Cochrane reviews apart from others is that the authors are committed to updating the review periodically. In theory, this results in the finding changing as new research is published. One example of this is the Cochrane review of Class II treatment that I helped produce. As additional evidence became available on the effect of early treatment on trauma, the conclusions changed.
I would like to point out that I am not suggesting that other reviews are not of value, but we need to bear in mind the overall quality of a review when we evaluate this form of literature.
Now that I have set out the context of the review, I will move to the main features that I look at when I read a review. This list is not exhaustive it is simply a set of tips that I use and I hope are useful to you. I would like to use the recent review on Temporary Anchorage Devices that I blogged about last week as an example.
Check the inclusion criteria.
The review should clearly outline the inclusion criteria. The authors should state whether the studies that they included were RCTs and/or other types of study. In some reviews RCTs and Controlled clinical trials are accepted. However, other reviews contain low quality studies, for example, retrospective designs with historic or convenience controls that are characterised by selection bias. When reading a review that includes retrospective studies you need to appreciate that the strength of any findings is diminished.
How were the papers selected?
Look closely at this section. In a good review the authors will provide a flow chart of how they obtained their papers. Ideally they should provide information on each paper that they included and excluded.
The meta analysis and Forest Plot
The analysis of the data derived from the included papers is frequently presented as a Forest plot. This is a clear way of presenting the data, but it can at first glance be confusing. I will go through an example Forest Plot from the TADS review. This is the plot that illustrates the effectiveness of palatal implants and TADS compared to conventional forms of anchorage.
I have concentrated on the results for TADS. When you look at this plot I have highlighted the relevant sections in red. If you click on each one they should expand so that you can read them more easily.
This figures shows that on the left hand side of the Forest plot is a summary of the data on sample size etc from each included study. You can look at this to build a picture of the number of studies and the subjects enrolled.
Now look at the data for each study, this shows the effect size and 95% confidence intervals.
In this figure I have highlighted the graphics around the vertical line. This is the line of “no effect”. Now look at the graphics that are dispersed around the line. The upper ones are the means and confidence intervals for each study. If the CI line crosses the line of no effect, then there is no statistically significant effect of the intervention.
Now look at the trapezoid shaped figure at the bottom of the plot. This represents the combined data. The vertical axis is the mean and the horizontal is the confidence intervals. If the shape crosses the line, the the difference is not statistically significant. You can see, in this example, that the shape does not cross the line and reveals that TADS are more effective than other methods of anchorage.
Finally, look at the bottom right numbers, this is the combined effect size and confidence intervals. Even if the result is statistically significant, you need to evaluate whether this is clinically significant. I have outlined this previously in this blog.
Strength of recommendations
Many reviews now include a statement on the strength of the recommendations that can be made from the review. Several now use the GRADE approach. I am not going to go through this in detail but I have included a brief outline of these in this table.
|Outcome domain||Number of studies (%)|
|Disease activity (morphologic change)||84 (63)|
|Health Service Utilization||43 (32)|
|Adverse effects of treatment||43 (32)|
|Social outcomes||12 (9)|
|Functional status||10 (8)|
|Physical consequences of malocclusion||3 (2)|
You will see that they are concerned with looking at level of confidence in the effect size included in they review. In the TADS review the effect size was -2.17 mm. We concluded that we had moderate confidence in the findings.
I have tried to keep this post as concise as possible. As a result, I have only outlined the features that I evaluate when I read a systematic review, I have not mentioned other features that are important and I appreciate that some people may criticise this rather basic approach. However, realistically this is all that I do when I read the literature. I hope that you feel that this is useful.
Emeritus Professor of Orthodontics, University of Manchester, UK.
Have your say!
Excellent, as usual. Very helpful. Thank you.
Thanks for the comments, I enjoyed doing this one, even though it took ages to get the graphics right. WordPress is not the best platform for putting graphics into a blog!
I play a very small part in the teaching of registrars in the West Midlands. As an exercise we looked through all the English speaking journals to see if we could find a single case where using TADs:
1. class II molars had been corrected to class I(without lower extractions moving the lower molars forward OR
2. more than one units space clossure in the lower
3.clossing an AOB with reduction of MM angle by more than 4 degrees
WE FAILED TO FIND A SINGLE CASE.
At first I was relieved by your systemic review but then looking at the figures as I read it:
two TADs move the molars on average 0.35mm forward (n=58)
control 2.4mm forward (n=60)
surgical anchorage move the molars forward 0.69 (n=92)
control 1.9mm forward (n=98)
SO THE BENIFITS OF THE TADS IS SMALLER THAN THE DIFFERENCE BETWEEN THE CONTROLS
This would make no difference to actual orthodontic treatment. Am I right to be concened that a multi-million pound industry may be doing nothing?
You are the best with all respect
Thank you for taking the time out to explain how one can go through a literature review without reading all the contents of the review. This helps me out a lot with the 61 literature systemic reviews I have to read. Although I am still confused on the graphs presented in these reviews, somehow, I will manage. I will also share this site with others.