Are there too many orthodontic systematic reviews?
During the current period of isolation for many of us, I have decided to increase my blog posts. I will do this by re-publishing older posts that I feel are interesting. This is one from 2015 on systematic reviews. I still think that my comments are relevant.
Over the past 10 years, there have been many orthodontic systematic reviews. I am now wondering if there are too many.
I will start this discussion with my own experiences. In the early 1990s, Manchester was an exciting place to be. The City had become a hotbed of new music with the first work of The Smiths and the Stone Roses. Manchester United had recovered from a thirty-year slumber and had started to play good football. In our small world of orthodontic research, we were beginning to plan randomised trials and enrolling nearly all our patients into trials. Things then got more exciting when we discovered the new research method called the ‘systematic review”. We were going to find out the truth about everything and change orthodontics forever!
Unfortunately, when we started systematically reviewing, we concluded “further research is needed” too many times. So, we went back to trials. At the same time, the Stone Roses and The Smiths imploded, and Manchester United starting dominating English and European football, but all things must pass.
Since then investigators have published many more trials. These were later included in systematic reviews. This led to an increase in our evidence base. But, are we reaching saturation with systematic reviews? For example, when I did a simple PubMed search, I found that there have been 110 orthodontic systematic reviews published in the last 10 years.
The increases in systematic reviewing
I have been thinking about this for a while, and I came across a paper by John Ioannidis. He is a methodology expert who challenges research methods and philosophy. In this paper, he asked whether systematic reviews have been overproduced. I am clearly not as talented as John, and I am only going to consider whether some of his points are relevant to orthodontic systematic reviews. I will also add some of my own.
The overall conclusion from his paper was;
“The production of systematic reviews has reached epidemic proportions. Possibly, most systematic reviews are unnecessary, misleading and/or conflicted”.
This is pretty strong stuff. So, what do I think about the current state of orthodontic systematic reviews? This is just my interpretation of this issue.
Several reviews of the same clinical question.
I did another PubMed search on Class II treatments. I found 30 systematic reviews; this is almost one for every cephalometric measurement. The reviews were about skeletal change, trauma, root resorption, TMD, mandibular change, maxillary movement and even a systematic review of systematic reviews! I have not looked at all of them, but I do wonder if we are overdoing it.
One of the consequences of the multiple reviewing of one orthodontic problem is that this adds to confusion and increases uncertainty. In some ways, this fuels the clinical controversy and the wheel keeps turning. While it may be good practice to repeat some reviews, I am not convinced that it is necessary.
Flawed reviews
This commonly occurs when the authors that they will not find many trials and then decide to include non- randomised trials or retrospective studies. We all know that one of the main benefits of carrying out a systematic review is to provide the highest level of evidence. As a result, systematic reviews should only include randomised trials. If we include retrospective studies, this leads to the introduction of bias into the systematic review. This reduces the level of evidence, and the review is of limited value. In many ways, the decision on whether to include this type of review in a journal is an editorial decision, and I hope that editors are aware of this problem. However, this does not seem the case, and we are currently becoming overloaded with the publication of poor-quality systematic reviews all concluding that we need more evidence.
The “no evidence and we need more research review.”
As I mentioned before, when systematic reviews were first developed, it was not unusual for reviews to conclude that “further research is needed”. This conclusion was not always unhelpful, as researchers could use this to generate questions for research.
However, several years later, I cannot help feeling that we need to move on. While systematic reviews are an essential part of question generation. We should ask whether every review should be published, particularly those that do not add to knowledge?
The selection of outcomes
This is not a problem that is unique to orthodontic treatment. However, we are a speciality that can measure everything. This ranges from obscure measurements in our favourite cephalometric analysis to more valuable outcomes, such as self-esteem.
The over measurement and non-standardised outcomes results in significant problems when we carry out systematic reviews. This results in a conclusion of “we could not carry out a meta-analysis because of the variation in outcomes”. Can we solve this problem by selecting common outcomes and abandonment of cephalometrics?
Pressure to publish
This is an essential issue for academic orthodontics. We are all under pressure to publish as part of our career progression. There is no doubt that an excellent systematic review is a valuable addition to a person’s cv. Furthermore, while a useful review is difficult to carry out, it is not as hard (or expensive) as a clinical trial. I wonder if the expansion in reviews has occurred because of the need for academics to publish. I fully understand this pressure. Nevertheless, the overproduction of systematic reviews is not a solution to this problem.
Final thoughts and solutions?
At the end of my reflections. I need to acknowledge that I am the author of five systematic reviews, but I hope that these are of a high standard. I cannot help thinking that we need to be more critical of the many reviews that are being done. As investigators, we need to ask if another systematic review is really going to add to knowledge? Editors need to assess the quality of reviews more closely. Finally, readers and clinicians should ask if a new review of a subject that has already been done or an appliance looking at an obscure measurement, or question, is going to add to their clinical knowledge. I think that we need to move on…
Emeritus Professor of Orthodontics, University of Manchester, UK.
It is interesting to see dental authority lawyers in Alberta, Canada investing over a decade and over $500,000 zigzagging in disciplining orthodontists and the odd general dentist over claims of quicker treatment- picking one study that suggests, for example that self-ligating brackets don’t reduce treatment time, then one that does and then back to no again…and next suggesting all methods whether they help or not are ‘implying superiority’ over other practitioners so are a violation of ethical and advertising regulation.
Two lawyers with no dental background have now beaten back close to 25 dental professionals over this niche. This goes on while the authority granted continuing education credits to attendees at the coronavirus-infected dental conference last month who may have attended a lecture by an Alberta-based orthodontist who promotes ultrasound orthodontics as FASTER and SAFER. If true, this brings a breach of public trust into the bullying of orthodontic professionals by the ‘Word Police’ of this dental authority. While dental manufacturers should be called out for exaggerated product claims, in this province select professionals bare the burden of for example DAMON’s positioning/justification for high costs.
It would be interesting to see what you would consider as a solid reference to guide dental boards/colleges/authorities on current orthodontic claims- speed of treatment is low on the scale of risks to the public, safety and claims of jaw growth would be more important but perhaps a more sensitive area for some believers. The public is not served by resources wasted on trivial advertising claims when they are being butchered by dentists using voodoo bite philosophies to justify full mouths of porcelain as headache cures. Orthodontics has also been contaminated with ‘neuromuscular’ beliefs and may be leading to harm by flying below the radar of scrutiny.
This is an interesting topic, Kevin – thanks for bringing it up. Despite the downsides to systematic reviews you’ve noted, there is an air of inevitably. We live in an information age. The more (good) info we have, the better the decision we can make. The number of studies in the literature has exploded and it’s difficult to keep track. So an advantage of the systematic review is that is gives the reader a broad overview of the current summated findings. If linked with a meta-analysis, these ‘wholesale’ findings can help swing decisions in one direction or another. The conclusion that ‘further research is necessary’ is a self-fulfilling prophecy, which leads to the propagation of further systematic reviews ad infinitum.
Thank-you for raising this topic again Kevin. My eye roll when I read a research paper with “systematic review” in the title. I cant help but wonder how can we possibly glean more accurate information on which to base clinical decision or increase academic knowledge from simply summarizing, “statistically cleansing” and even worse, cherry picking from a muddy pool?