Short steps on how to read a paper: Part 10. Outcomes
This short step is about the outcomes of a study. As with other sections, understanding how to interpret outcomes is essential to the reading and interpretation of a study.
The correct selection of outcomes is crucial to a research project. If the investigators choose to measure outcomes that are not relevant. Then the findings of the paper are not useful to anyone. This is a problem on many levels. Not least because clinical research is often expensive and arduous. Selection of inappropriate outcomes is a leading cause of waste in research.
Irrelevant outcomes
The use of irrelevant and obscure outcomes is sadly very true for orthodontic research.
Regular readers of my blog will know that I am not a fan of some traditional orthodontic measures. For example, cephalometric analysis. This is because these measurements are often misused. This is done when investigators generate many related measurements and run simple univariate tests along all of them. They then spot the ones that are significant and base the paper on these. This can distort the interpretation of study findings very significantly. Furthermore, investigators can get excited about differences of less than 1 degree, and this is clinically meaningless.
This approach may have been acceptable when orthodontic research was in its early development. In the present time, these papers are not useful to anyone, apart from the person building their CV. Unfortunately, this practice is getting worse with the development of CBCT, which enables investigators to measure ad infinitum to the smallest degree.
What should you look for?
So, when you are reading a paper, I think that you should look for these properties of the outcomes.
- Is the outcome relevant to my clinical practice? For example, in a study looking at the effectiveness of a new bracket do the authors investigate the duration of treatment. Other clinically relevant outcomes are occlusal index scores, harms due to treatment, socio-psychological measures and appearance of the teeth and face,
- Does the outcome have meaning to our patients? For example, discomfort, the appearance of appliances and effects on self-esteem. Ideally, all outcomes should have value to patients. If they do not, we may be missing some of the critical impacts of treatment.
- Is the effect size that the authors report clinically relevant? This is a significant problem in orthodontics. It does not matter what outcome the authors select if there is no indication of the clinical relevance of the effect size, the paper does not help us.
In summary, if the outcomes lack relevance to your practice or to your patients or we are measuring minute differences, you should start to question the value of the paper.
We have expanded on orthodontic outcomes and the measures we could all use in this blog post and associated paper.
Emeritus Professor of Orthodontics, University of Manchester, UK.
We also have to remember that outcomes are presented as means of a group. The variation, often presented as standard deviation, can be fairly large even if the results are statistically significant. So, what will the outcome be on the patient in our dental chair. The mean patient does not exist!
We patiently wait for Bayesian analyses to become part of clinical protocols. They won’t be cheap or easy to do but they may allow the clinician to adapt their treatment based on the response to the first intervention. I would hope they provide a more nuanced approach allowing for more acknowledgment that response a mean response could be thought of but flexibility in planning for those who respond more or less than the mean.
There needs to be some consensus on what is “clinically significant” in orthodontics since multivariate analysis can gauge treatment effect. But, Dr Hansen raises the most relevant question. Here, we have to bypass descriptive statistics and venture into (predictive) modeling.
Although I agree that the use of cephalometrics to find “something” is a trivial pursuit, there are specific questions that are perhaps answerable only by way of cephalometrics. For example, asking the patient how they feel won’t tell whether or not functional appliances can grow mandibles. Does extraction cause distal mandibular displacement? On and on, world without end. I would suggest that a blanket condemnation of cephalometrics is inappropriate and “evidence based” only in specific, albeit numerous, instances.