Do authors exaggerate (spin) the findings of orthodontic trials?
We assume that the results of orthodontic trials are reported as accurately as possible. This new study suggests that this does not always happen. I thought that this was a significant paper for all of us who are interested in interpreting research.
While we have clear guidelines for reporting trials, authors have significant freedom to structure their papers. Importantly, they can highlight specific results in the search for finding significant findings. They, perhaps, do this because there is a tendency for journals to accept papers that report significant or positive findings. Furthermore, readers are more likely to read articles reporting substantial treatment effects instead of no difference in interventions or simply “more research is needed”.
This “highlighting” is called spin. It has been described as
“the use of specific reporting strategies, from whatever motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or to distract the reader from statistically nonsignificant results”.
Does this occur in orthodontic publications? This team from China set out to discover if this was the case. The European Journal of Orthodontics published their paper.
Feiyang Guo et al. doi:10.1093/ejo/cjab044
What did they ask?
They did the study to
“Identify the presence and characteristics of spin within abstracts of RCTs published in five leading journals”.
What did they do?
First, the authors selected the five top orthodontic journals according to journal citation reports. These were:
- European Journal of Orthodontics
- American Journal of Orthodontics
- Progress in Orthodontics
- Angle Orthodontist
- Orthodontics and Craniofacial Research
Then they did a manual search of the journals to identify abstracts from January 2015 to December 2020. They included abstracts that clearly stated statistically non-significant primary outcomes.
For their next step, the reviewers used a framework to identify any spin strategies. They based this is on the following categories.
- Focusing on statistically significant results and within-group comparisons
- Verbiage that implied significance. For example, stating that there was a trend towards significance.
- Claiming significance or non-inferiority for statistically non-significant results.
- Claiming efficacy when the primary outcomes were non-significant
- Mentioning statistically non-significant results for the primary outcome, but they stated the treatment was beneficial.
- Recommending that the treatment was acceptable for clinical use.
What did they find?
The team identified 111 RCT abstracts. The AJO or EJO published most of them.
- They found spin-in 69 ((62%) of the abstracts.
- In the results sections, they found spin-in 42% of abstracts
- In the conclusions, there was spin-in 51%
- When they looked at the results and conclusions together, they found spin in 31% of the text.
The most common reason for spin in the results was “focussing on significant within group comparisons for the primary outcome”. For example, discussing selected pre and post-treatment ceph measurement in one treatment arm.
When they looked at similar data for the conclusions. They found that the most common spin strategy was “claiming equivalence or non-inferiority for statistically non-significant results”.
Their overall conclusions were:
“The prevalence of spin in orthodontic trial abstracts was high”.
What did I think?
This is a significant paper for those who are interested in interpreting the literature.
I have had a good look at their methodology. The investigators used a standard method for this work. Their approach was good. Nevertheless, I was not sure why they only selected abstracts of trials that reported non-significant findings. I wonder if the results would have been different if the authors had simply included all published trials in the time period?
The results are very relevant for orthodontics. It is clear that authors tend to spin their findings. Unfortunately, this practice appears to be widespread.
When we consider the type of spin. One method concentrates on within-group comparisons. This is easily done for orthodontics because we tend to use cephalometric measurements. In this respect, the authors draw attention to the before and after treatment ceph measurements within one treatment. Unfortunately, this creates a degree of “white noise”. As a result, we find it difficult to see and interpret the critical between-group measurements. Importantly, these may not be significant.
Another practice they highlighted was stating that one intervention had a treatment effect even though the difference was not statistically significant. We need to remember that if a difference is not statistically significant, it may have occurred by chance.
The authors wrote a clear discussion outlining these issues. I suggest that you read this.
Final comments
The most important take-home message is that we need to take care in reading the abstracts of published RCTs. It is up to us, as readers, to evaluate the literature. This is something that we need to consider as the practice of spin is widespread. Referees and journal editors should also take steps to identify spin and reduce it.
I now need to look at some of my past papers to see if I have spun the results. I will keep my fingers crossed!
Emeritus Professor of Orthodontics, University of Manchester, UK.
Very interesting study – and well-spotted, Kevin. This trend is also seen in other (medical) journals. Recently, a paper was published in a high-ranking journal where spin was obvious (at least to me). I wrote the Editor who published my response and the authors were big enough to accept the correction in their published reply. Now I know what a ‘spin doctor’ is!
It’s very sad in a way. It’s enough that we have to run the gauntlet of claims constantly being thrown at us with no basis in research or science at all. Now, even within very respected journals, we still have to carefully compare the claims to the statistical relevance.
I remember a time when it was commonplace to publish the raw data in research. My chairman in residency, Dr. Richard Smith, had a masters in statistics. Sometimes he would take an author’s raw data and use it to come to the exact opposite conclusion of that in the paper.
His question at the time was, what will happen when they stop publishing raw data? He thought that it would remove a valuable double check on the authors. Now that raw data is no longer published,, and this issue has been revealed, it seems that another leak has been found in the cauldron.
I think one root of this problem (spin in reporting results) is the pressure Institutions bring to bear on staff to publish and to publish papers with ‘impact’ and of course, there is not much impact in a neutral finding.
I suggest that research is to some extent an artistic endeavour which needs time to bear fruit, not an enterprise which is tractable simply to more effort (like building a railway).
Would an academic dean threaten Beethoven with dismissal because he had not written enough symphonies? Perhaps they might, nowadays …
In my opinion, Academics need space and time to do good reliable work and, in such circumstances, the temptation to spin data might loose its force?
I think the journals need to take a tighter reign on this. not every reader will be proficient in interpreting results and will have a natural bias towards ones own treatment philosophy. extreme findings are usually due to methodological mistakes. it is rare to see RCTs published where they declare all results non significant. also some areas of scientific publishing are more rigorous than others with publishing standards.