Are there too many orthodontic systematic reviews?
Are there too many orthodontic systematic reviews?
Over the past 10 years there have been many orthodontic systematic reviews. I am now wondering if there are too many…..just a thought in a short blog post….
I will start this discussion with my own experiences. In the early 1990’s Manchester was an exciting place to be. The City had become a hotbed of new music with the early work of The Smiths and the Stone Roses. Manchester United had recovered from a thirty year slumber and had started to play good football. In our small world of orthodontic research, we were starting to plan randomised trials and enrolling nearly all our patients into trials. Things then got more exciting when we discovered the great new research method called the ‘systematic review”. We were going to find out the truth about everything and change orthodontics forever!
Unfortunately, when we started systematically reviewing we concluded “further research is needed” too many times. So we went back to trials. At the same time the Stone Roses and The Smiths imploded, and Manchester United starting dominating English and European football, but all things must pass…..
Since then investigators have published many more trials. These were then included into systematic reviews. This has led to an increase in our evidence base. But, are we reaching saturation with systematic reviews? For example, when I did a simple PubMed search I found that there have been 110 orthodontic systematic reviews published in the last 10 years.
Some concerns about the increase in systematic reviewing
I have been thinking about this for while and I came across a paper by John Ioannidis. He is a methodology expert who challenges research methods and philosophy. In this paper he asked whether systematic reviews have been overproduced. I am clearly not as clever as John (or Brian Cox) and I am simply going to consider whether some of his points are relevant to orthodontic systematic reviews. I will also add some of my own.
The overall conclusion from his paper was;
“The production of systematic reviews has reached epidemic proportions. Possibly, most systematic reviews are unnecessary, misleading and/or conflicted”.
This is pretty strong stuff. So, what do I think about the current state of orthodontic systematic reviews? This is just my interpretation of this issue.
Several reviews of the same clinical question.
I did another PubMed search on Class II treatments. I found 30 systematic reviews, this is almost one for every cephalometric measurement. The reviews were about skeletal change, trauma, root resorption, TMD, mandibular change, maxillary movement and even a systematic review of systematic reviews! I have not looked at all of them, but I do wonder if we are overdoing it.
One of the consequence of the multiple reviewing of one orthodontic problem is that this adds to confusion and increases uncertainty. In some ways this fuels the clinical controversy and the wheel keeps turning. While it may be good practice to repeat some reviews, I am not convinced that is necessary.
Flawed reviews
This commonly occurs when the authors do not find any trials and then decide to include non- randomised trials or retrospective studies. We all know that one of the main benefits of carrying out a systematic review is to provide the highest level of evidence. As a result, systematic reviews should only include randomised trials. If we include retrospective studies this leads to the introduction of bias into the systematic review. This reduces the level of evidence and the review is of limited value. In many ways, the decision on whether to include this type of review in a journal is an editorial decision and I hope that editors are aware of this problem.
The “no evidence and we need more research review”
As I mentioned before, when systematic reviews were first developed, it was not unusual for reviews to conclude that “further research is needed”. This conclusion was not always unhelpful, as researchers could use this to generate questions for research.
However, several years later, I cannot help feeling that we need to move on. While systematic reviews are an essential part of question generation. We should ask whether every review should be published, particularly those that do not add to knowledge?
The selection of outcomes
This is not a problem that is unique to orthodontic treatment. However, we are a specialty that can measure everything. This ranges from obscure measurements in our favourite cephalometric analysis to more valuable outcomes, such as self esteem.
The over measurement and non standardised outcomes results in major problems when we carry out systematic reviews. This results in a conclusion of “we could not carry out a meta analysis because of the variation in outcomes”. We can solve this problem by the selecting common outcomes and an abandonment of cephalometrics?
Pressure to publish
This is an important issue for academic orthodontics. We are all under pressure to publish, as part of our career progression. There is no doubt that a good systematic review is a valuable addition to a person’s cv. Furthermore, while a good review is difficult to carry out, it is not as hard (or expensive) as a clinical trial. As a result, I wonder if the expansion in reviews has occurred because of the need for academics to publish. I fully understand this pressure. Nevertheless, the over production of systematic reviews is not a solution to this problem.
Final thoughts and solutions?
At the end of my reflections. I need to acknowledge that I am the author of three systematic reviews, but I hope that these are of a high standard. This is the solution. I cannot help thinking that we need to be more critical of the many reviews that are being done. As investigators, we need to ask if another systematic review is really going to add to knowledge? Editors need to assess the quality of reviews more closely. Finally, readers and clinicians should ask if a new review of an appliance looking at an obscure measurement, or question, is going to add to their clinical knowledge. I think that we need to move on…
Emeritus Professor of Orthodontics, University of Manchester, UK.
Agree with your assessment 100%
Dear Kevin, the question is not ”Are there too many orthodontic systematic reviews?” but ”Are there too many low quality orthodontic systematic reviews?” And the answer to the latter question is YES.
We face a new problem now: how to judge the quality of a published systematic review. This is something we have to learn our residents and future clinicians now from the very beginning.
I completely agree with you that the basis of a good systematic review is a number of good RCTs. Unfortunately, there is a tendency to give priority to synthesizing (often low quality) evidence rather than producing evidence through a randomized controlled trial. A reason you did not mention might be that it is very difficult to get funding for orthodontic trials as orthodontics is not a major health issue. Performing a SR is less costly.
Kevin I would like to add a little food for thought.
First I can be consider one source of this overproduction of RSs. I have published quite a few. I have to say that I used to published a lot more in the past than nowadays.
As an associated editor in a couple of orthodontic journals I agree with your overall conclusion that there are too many. SRs keep being submitted and most of the times the question while reviewing them is the “so what” question. The end conclusion may be the typical “more and better CTs are needed”, but I have to say that sometimes those reviews discussion generate a very productive inner thought process. There are certainly a lot brighter minds than my own around. I certainly enjoy a good discussion every time. As clinician I learned or question my self a lot more based on the manuscript discussions than the actual conclusions.
Maybe we simply have too many orthodontic journals. With less the selection process would be tougher and only the highest quality will survive. In any event the same can be said about clinical trials. A lot published but certainly with a lot of areas to improve.
I personally find the exercise of generating a SR, very loose definition due to manpower and time issues, a very productive step for our students to properly justify their research projects. The question is if all of those attempts should be published. They are certainly not. So I agree that we may be publishing a lot of unnecessary SRs, but the decision about which topic is worth or not is problematic. Each one of use has its own biases as personal preferences.
Finally, it is more likely that a busy clinician will read a SR than a few individual CTs. The onus is at the end of each of us to make decisions based on what other opinion. That will not change – we are responsible form the level of care we provide. With a good individual critical training training we should be able to screen the useless from the useful. Is that a responsibility of the editors, authors, researchers?
Again thanks for fomenting all this helpful discussion.
Dear Kevin,
It was nice to meet you in Queenstown.
I personally agree with you that many SRs are redundant and of low-quality. For some orthodontic research questions, there are more SRs than RCTs. Furthermore, there is a new trend to carry out SRs of SRs, so that the same data are squeezed and squeezed again (let’s torture the data until they confess what we want) in an attempt to find out new answers to old questions, but these will never come from those data.
SRs are cheap, relatively fast and easy to carry out, their acceptance rate is relatively high, and they often get more citations than other research articles. Therefore, they are very attractive to academics, as they can quickly increase their celebrity and boost their career. The are also attractive to Journal Editors, as they can quickly increase the IF of their journals. I much welcome your post as it will increase awareness of this problem in Editors, Educators, Students, and Clinicians. Personally I wish to see more original research published in orthodontic journals and less systematic reviews, especially those of low quality.