Do authors exaggerate (spin) the findings of orthodontic trials?
We assume that the results of orthodontic trials are reported as accurately as possible. This new study suggests that this does not always happen. I thought that this was a significant paper for all of us who are interested in interpreting research.
While we have clear guidelines for reporting trials, authors have significant freedom to structure their papers. Importantly, they can highlight specific results in the search for finding significant findings. They, perhaps, do this because there is a tendency for journals to accept papers that report significant or positive findings. Furthermore, readers are more likely to read articles reporting substantial treatment effects instead of no difference in interventions or simply “more research is needed”.
This “highlighting” is called spin. It has been described as
“the use of specific reporting strategies, from whatever motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or to distract the reader from statistically nonsignificant results”.
Does this occur in orthodontic publications? This team from China set out to discover if this was the case. The European Journal of Orthodontics published their paper.
Feiyang Guo et al. doi:10.1093/ejo/cjab044
What did they ask?
They did the study to
“Identify the presence and characteristics of spin within abstracts of RCTs published in five leading journals”.
What did they do?
First, the authors selected the five top orthodontic journals according to journal citation reports. These were:
- European Journal of Orthodontics
- American Journal of Orthodontics
- Progress in Orthodontics
- Angle Orthodontist
- Orthodontics and Craniofacial Research
Then they did a manual search of the journals to identify abstracts from January 2015 to December 2020. They included abstracts that clearly stated statistically non-significant primary outcomes.
For their next step, the reviewers used a framework to identify any spin strategies. They based this is on the following categories.
- Focusing on statistically significant results and within-group comparisons
- Verbiage that implied significance. For example, stating that there was a trend towards significance.
- Claiming significance or non-inferiority for statistically non-significant results.
- Claiming efficacy when the primary outcomes were non-significant
- Mentioning statistically non-significant results for the primary outcome, but they stated the treatment was beneficial.
- Recommending that the treatment was acceptable for clinical use.
What did they find?
The team identified 111 RCT abstracts. The AJO or EJO published most of them.
- They found spin-in 69 ((62%) of the abstracts.
- In the results sections, they found spin-in 42% of abstracts
- In the conclusions, there was spin-in 51%
- When they looked at the results and conclusions together, they found spin in 31% of the text.
The most common reason for spin in the results was “focussing on significant within group comparisons for the primary outcome”. For example, discussing selected pre and post-treatment ceph measurement in one treatment arm.
When they looked at similar data for the conclusions. They found that the most common spin strategy was “claiming equivalence or non-inferiority for statistically non-significant results”.
Their overall conclusions were:
“The prevalence of spin in orthodontic trial abstracts was high”.
What did I think?
This is a significant paper for those who are interested in interpreting the literature.
I have had a good look at their methodology. The investigators used a standard method for this work. Their approach was good. Nevertheless, I was not sure why they only selected abstracts of trials that reported non-significant findings. I wonder if the results would have been different if the authors had simply included all published trials in the time period?
The results are very relevant for orthodontics. It is clear that authors tend to spin their findings. Unfortunately, this practice appears to be widespread.
When we consider the type of spin. One method concentrates on within-group comparisons. This is easily done for orthodontics because we tend to use cephalometric measurements. In this respect, the authors draw attention to the before and after treatment ceph measurements within one treatment. Unfortunately, this creates a degree of “white noise”. As a result, we find it difficult to see and interpret the critical between-group measurements. Importantly, these may not be significant.
Another practice they highlighted was stating that one intervention had a treatment effect even though the difference was not statistically significant. We need to remember that if a difference is not statistically significant, it may have occurred by chance.
The authors wrote a clear discussion outlining these issues. I suggest that you read this.
The most important take-home message is that we need to take care in reading the abstracts of published RCTs. It is up to us, as readers, to evaluate the literature. This is something that we need to consider as the practice of spin is widespread. Referees and journal editors should also take steps to identify spin and reduce it.
I now need to look at some of my past papers to see if I have spun the results. I will keep my fingers crossed!