Are there too many orthodontic systematic reviews?
During the current period of isolation for many of us, I have decided to increase my blog posts. I will do this by re-publishing older posts that I feel are interesting. This is one from 2015 on systematic reviews. I still think that my comments are relevant.
Over the past 10 years, there have been many orthodontic systematic reviews. I am now wondering if there are too many.
I will start this discussion with my own experiences. In the early 1990s, Manchester was an exciting place to be. The City had become a hotbed of new music with the first work of The Smiths and the Stone Roses. Manchester United had recovered from a thirty-year slumber and had started to play good football. In our small world of orthodontic research, we were beginning to plan randomised trials and enrolling nearly all our patients into trials. Things then got more exciting when we discovered the new research method called the ‘systematic review”. We were going to find out the truth about everything and change orthodontics forever!
Unfortunately, when we started systematically reviewing, we concluded “further research is needed” too many times. So, we went back to trials. At the same time, the Stone Roses and The Smiths imploded, and Manchester United starting dominating English and European football, but all things must pass.
Since then investigators have published many more trials. These were later included in systematic reviews. This led to an increase in our evidence base. But, are we reaching saturation with systematic reviews? For example, when I did a simple PubMed search, I found that there have been 110 orthodontic systematic reviews published in the last 10 years.
The increases in systematic reviewing
I have been thinking about this for a while, and I came across a paper by John Ioannidis. He is a methodology expert who challenges research methods and philosophy. In this paper, he asked whether systematic reviews have been overproduced. I am clearly not as talented as John, and I am only going to consider whether some of his points are relevant to orthodontic systematic reviews. I will also add some of my own.
The overall conclusion from his paper was;
“The production of systematic reviews has reached epidemic proportions. Possibly, most systematic reviews are unnecessary, misleading and/or conflicted”.
This is pretty strong stuff. So, what do I think about the current state of orthodontic systematic reviews? This is just my interpretation of this issue.
Several reviews of the same clinical question.
I did another PubMed search on Class II treatments. I found 30 systematic reviews; this is almost one for every cephalometric measurement. The reviews were about skeletal change, trauma, root resorption, TMD, mandibular change, maxillary movement and even a systematic review of systematic reviews! I have not looked at all of them, but I do wonder if we are overdoing it.
One of the consequences of the multiple reviewing of one orthodontic problem is that this adds to confusion and increases uncertainty. In some ways, this fuels the clinical controversy and the wheel keeps turning. While it may be good practice to repeat some reviews, I am not convinced that it is necessary.
This commonly occurs when the authors that they will not find many trials and then decide to include non- randomised trials or retrospective studies. We all know that one of the main benefits of carrying out a systematic review is to provide the highest level of evidence. As a result, systematic reviews should only include randomised trials. If we include retrospective studies, this leads to the introduction of bias into the systematic review. This reduces the level of evidence, and the review is of limited value. In many ways, the decision on whether to include this type of review in a journal is an editorial decision, and I hope that editors are aware of this problem. However, this does not seem the case, and we are currently becoming overloaded with the publication of poor-quality systematic reviews all concluding that we need more evidence.
The “no evidence and we need more research review.”
As I mentioned before, when systematic reviews were first developed, it was not unusual for reviews to conclude that “further research is needed”. This conclusion was not always unhelpful, as researchers could use this to generate questions for research.
However, several years later, I cannot help feeling that we need to move on. While systematic reviews are an essential part of question generation. We should ask whether every review should be published, particularly those that do not add to knowledge?
The selection of outcomes
This is not a problem that is unique to orthodontic treatment. However, we are a speciality that can measure everything. This ranges from obscure measurements in our favourite cephalometric analysis to more valuable outcomes, such as self-esteem.
The over measurement and non-standardised outcomes results in significant problems when we carry out systematic reviews. This results in a conclusion of “we could not carry out a meta-analysis because of the variation in outcomes”. Can we solve this problem by selecting common outcomes and abandonment of cephalometrics?
Pressure to publish
This is an essential issue for academic orthodontics. We are all under pressure to publish as part of our career progression. There is no doubt that an excellent systematic review is a valuable addition to a person’s cv. Furthermore, while a useful review is difficult to carry out, it is not as hard (or expensive) as a clinical trial. I wonder if the expansion in reviews has occurred because of the need for academics to publish. I fully understand this pressure. Nevertheless, the overproduction of systematic reviews is not a solution to this problem.
Final thoughts and solutions?
At the end of my reflections. I need to acknowledge that I am the author of five systematic reviews, but I hope that these are of a high standard. I cannot help thinking that we need to be more critical of the many reviews that are being done. As investigators, we need to ask if another systematic review is really going to add to knowledge? Editors need to assess the quality of reviews more closely. Finally, readers and clinicians should ask if a new review of a subject that has already been done or an appliance looking at an obscure measurement, or question, is going to add to their clinical knowledge. I think that we need to move on…
Emeritus Professor of Orthodontics, University of Manchester, UK.