Are all orthodontic systematic reviews equal?

Over the last few years, I have highlighted the problems with publications that are in predatory journals.  I have raised concerns with the legitimacy of the findings of these papers. Importantly, the risk that these findings influence practice increases if these papers are non-critically included in new systematic reviews. Martyn Cobourne has written a great blog post about a new paper examining the effects of poor-quality predatory research on the conclusions of some systematic reviews.  It is an essential read for everyone who reads the orthodontic research literature.

Introduction

This is a really interesting paper produced by a well-established international team of researchers based in Dublin, Bern and Riyadh, investigating an issue of fundamental importance to the practice of evidence-based orthodontics. The novelty, importance and quality of this study are reflected in its publication in the Journal of Dental Research, a high-impact journal that is very difficult to publish in.

Predatory journals

Predatory academic journals are a product of the online publishing industry and purport to be legitimate, adhering to high standards of publishing ethics and facilitating rapid open-access publication, but often they do not. Many of these journals aggressively solicit submissions, charge high publication fees, and rely on high-volume, rapid publication output for their financial model. For these reasons, the peer review and editorial processes are often less than robust, leading to the publication of papers of low methodological quality.

What did they ask?

They did this study to ask;

“What is the prevalence of predatory papers included in meta-analyses of orthodontic systematic reviews and to describe the prevalence and the characteristics of recent systematic reviews published in legitimate, suspect, or predatory journals within the field of orthodontics”.

What did they do?

This study has investigated the influence of predatory publishing in orthodontics, focusing specifically on orthodontic systematic reviews, which represent a fundamental component of evidence-based practice. The authors searched various databases for English-language orthodontic systematic reviews published between 2020 and 2023, and then analysed these reviews for a number of parameters, including their general characteristics (type, topic, authors), the type of journal in which they were published, and, importantly, whether the journal was regarded as legitimate, suspect, or predatory.

They further investigated one meta-analysis from each of these systematic reviews and extracted data on the primary studies included in each (again, assessing whether any had been published in predatory or suspect journals). Specifically, any meta-analyses containing at least one study published in a suspect or predatory journal underwent a further subset analysis (effect measure, sample size, heterogeneity), including analyses of the data using either all studies or only those published in legitimate journals.

What did they find?

There were some really interesting findings, albeit slightly depressing. They identified 340 eligible systematic reviews, and among these, 114 (33%) contained primary data from at least one predatory or suspect journal. The majority of these systematic reviews (60%) were published in broader-scope medical and general science journals, not orthodontic – and 16% of these journals were classified as predatory or suspect. Among the 114 systematic reviews that had their primary meta-analyses re-analysed, the median sample size within the included studies was only n=20, with a median I2 score of 85% (demonstrating that studies with small sample sizes and substantial heterogeneity often populate these meta-analyses). Recalculation using all included studies, or only those published in legitimate journals, resulted in pooled estimates decreasing in 55% of meta-analyses and increasing in 45%.

What did I think?

The importance of this study is that it provides data supporting the perception many of us who spend time reading these systematic reviews have held for years – that they often include primary studies of questionable validity and probably should not have been published. Perhaps most surprising is the sheer scale of data that is out there, emanating from predatory or suspect journals. One in three orthodontic meta-analyses contains primary data from these journals, and almost one in six orthodontic systematic reviews. The effect of data derived from predatory publishing on pooled estimates in meta-analyses is unpredictable and therefore adds considerable uncertainty to the evidence.

The harsh reality is that the current orthodontic evidence base, derived from systematic reviews, is likely to be significantly distorted by predatory publishing practices, with over- and under-estimation of treatment effects commonplace. There are too many journals, too many systematic reviews, and too many unreliable studies. Read them and interpret them with caution…………………..