Are there too many orthodontic systematic reviews?
Are there too many orthodontic systematic reviews?
Over the past 10 years there have been many orthodontic systematic reviews. I am now wondering if there are too many…..just a thought in a short blog post….
I will start this discussion with my own experiences. In the early 1990’s Manchester was an exciting place to be. The City had become a hotbed of new music with the early work of The Smiths and the Stone Roses. Manchester United had recovered from a thirty year slumber and had started to play good football. In our small world of orthodontic research, we were starting to plan randomised trials and enrolling nearly all our patients into trials. Things then got more exciting when we discovered the great new research method called the ‘systematic review”. We were going to find out the truth about everything and change orthodontics forever!
Unfortunately, when we started systematically reviewing we concluded “further research is needed” too many times. So we went back to trials. At the same time the Stone Roses and The Smiths imploded, and Manchester United starting dominating English and European football, but all things must pass…..
Since then investigators have published many more trials. These were then included into systematic reviews. This has led to an increase in our evidence base. But, are we reaching saturation with systematic reviews? For example, when I did a simple PubMed search I found that there have been 110 orthodontic systematic reviews published in the last 10 years.
Some concerns about the increase in systematic reviewing
I have been thinking about this for while and I came across a paper by John Ioannidis. He is a methodology expert who challenges research methods and philosophy. In this paper he asked whether systematic reviews have been overproduced. I am clearly not as clever as John (or Brian Cox) and I am simply going to consider whether some of his points are relevant to orthodontic systematic reviews. I will also add some of my own.
The overall conclusion from his paper was;
“The production of systematic reviews has reached epidemic proportions. Possibly, most systematic reviews are unnecessary, misleading and/or conflicted”.
This is pretty strong stuff. So, what do I think about the current state of orthodontic systematic reviews? This is just my interpretation of this issue.
Several reviews of the same clinical question.
I did another PubMed search on Class II treatments. I found 30 systematic reviews, this is almost one for every cephalometric measurement. The reviews were about skeletal change, trauma, root resorption, TMD, mandibular change, maxillary movement and even a systematic review of systematic reviews! I have not looked at all of them, but I do wonder if we are overdoing it.
One of the consequence of the multiple reviewing of one orthodontic problem is that this adds to confusion and increases uncertainty. In some ways this fuels the clinical controversy and the wheel keeps turning. While it may be good practice to repeat some reviews, I am not convinced that is necessary.
This commonly occurs when the authors do not find any trials and then decide to include non- randomised trials or retrospective studies. We all know that one of the main benefits of carrying out a systematic review is to provide the highest level of evidence. As a result, systematic reviews should only include randomised trials. If we include retrospective studies this leads to the introduction of bias into the systematic review. This reduces the level of evidence and the review is of limited value. In many ways, the decision on whether to include this type of review in a journal is an editorial decision and I hope that editors are aware of this problem.
The “no evidence and we need more research review”
As I mentioned before, when systematic reviews were first developed, it was not unusual for reviews to conclude that “further research is needed”. This conclusion was not always unhelpful, as researchers could use this to generate questions for research.
However, several years later, I cannot help feeling that we need to move on. While systematic reviews are an essential part of question generation. We should ask whether every review should be published, particularly those that do not add to knowledge?
The selection of outcomes
This is not a problem that is unique to orthodontic treatment. However, we are a specialty that can measure everything. This ranges from obscure measurements in our favourite cephalometric analysis to more valuable outcomes, such as self esteem.
The over measurement and non standardised outcomes results in major problems when we carry out systematic reviews. This results in a conclusion of “we could not carry out a meta analysis because of the variation in outcomes”. We can solve this problem by the selecting common outcomes and an abandonment of cephalometrics?
Pressure to publish
This is an important issue for academic orthodontics. We are all under pressure to publish, as part of our career progression. There is no doubt that a good systematic review is a valuable addition to a person’s cv. Furthermore, while a good review is difficult to carry out, it is not as hard (or expensive) as a clinical trial. As a result, I wonder if the expansion in reviews has occurred because of the need for academics to publish. I fully understand this pressure. Nevertheless, the over production of systematic reviews is not a solution to this problem.
Final thoughts and solutions?
At the end of my reflections. I need to acknowledge that I am the author of three systematic reviews, but I hope that these are of a high standard. This is the solution. I cannot help thinking that we need to be more critical of the many reviews that are being done. As investigators, we need to ask if another systematic review is really going to add to knowledge? Editors need to assess the quality of reviews more closely. Finally, readers and clinicians should ask if a new review of an appliance looking at an obscure measurement, or question, is going to add to their clinical knowledge. I think that we need to move on…