Can we reduce uncertainty in orthodontic treatment?
Sense about Science: Reducing uncertainty in orthodontic treatment
I spend a large amount of time browsing websites looking for interesting articles on research that is relevant or even not relevant to orthodontics. One of the most interesting sites is Sense about Science (http://www.senseaboutscience.org). Their twitter account is @senseaboutsci. This is a charitable trust that has the aim of helping people make sense of science and evidence. It is directed at lay and clinical people. I suggest that you have a look at the website.
While having a look at their work, I came across the report of a symposium called “Making sense of Uncertainty” (http://www.senseaboutscience.org/pages/uncertainty.html). I felt that this was a really interesting document as it explains the role of uncertainty in research, media and public policy. When I read this it became clear to me that most of their discussion was relevant to orthodontic research and I thought that I should address this in this post and this is my orthodontic interpretation of some of the issues that they raised.
The paper starts of by pointing out that there is rarely such a thing as 100% certainty and scientific investigation aims to reduce uncertainty as much as possible. When we consider orthodontics, there is no doubt that there is a large amount of uncertainty in almost everything that we do. This is reinforced by the common experience that 10 orthodontists will come up with 10 different treatment plans for one patient! There have also been several research projects that have been carried out to illustrate the diversity of clinical opinion when examining records of patients. For example, the papers by Han et al, and Ribarevski et at.
Uncertainty and confidence
A very nice quote is from Michael Rawlins (who is the Chair of Nice) he states that “Uncertainty is fraught with misinterpretation and he prefers to consider our level of confidence in a finding or decision”. I felt that this was a clinically relevant way of addressing the problem and we should consider the level of confidence that we have in our “evidence based ” decisions. This brings us to the interpretation of research papers and the statistics that indicate levels of confidence (or uncertainty). Importantly, they indicate whether a finding is not simply a random event.
One method of evaluating confidence is to interpret confidence intervals and we are only just getting round to including these in orthodontic research. One way that I can explain these is to consider a study in which we want to identify the average overjet of 11 year old children in the UK. We cannot make this measurement on all the children, so we select a sample and come up with a mean overjet measurement. Because this is a sample we are uncertain on the accuracy of this measurement and we calculate the 95% confidence interval. This will indicate the range of values that we would expect the overjet to fall within for 95 out of 100 repeats of the data collection. The narrower the confidence interval the less the uncertainty and this helps us come to conclusions the level of confidence that we have in a piece of research.
I would like to illustrate this with the results from a recent systematic review into methods of moving molars distally. I reviewed this in an earlier post (Distal movement of molars: A case of swings and roundabouts?). This is an open access paper (http://goo.gl/0wVqTC) and I will reproduce a table illustrating the amount of distal movement achieved with different distalising appliances.
If we evaluate this table, we can see that four studies have been included in a meta analysis and this shows that for a total sample of 75 patients that the intra oral appliance is more effective than headgear in moving molars distally by 1.45 mm. It is clear that this difference is small and not very exciting, nevertheless, we also need to look at the confidence intervals. These range from -2.74 to -0.15. This means that if we repeated this study 100 times then 95 times out of 100 then the “true mean” will fall between -2.74 to -0.15. We can interpret this as representing a high degree of uncertainty in this area of our treatment. This is because the values represent a wide range from nearly 3.0 which is clinically significant to 0.15 which is of no value. I can, therefore, conclude that the mean difference between treatments is not great and the finding has a high level of uncertainty. In reality, we do not know much about the comparative effectiveness of distalising appliances and headgear. As a result, our decisions should be based on other factors such as our tolerance of risk in providing headgear with its inherent serious but rare risks.
Can we tolerate uncertainty?
So to move on from statistics, if we accept that we cannot eliminate uncertainty then we have to consider if we know enough or do we need to carry our additional research using larger and larger samples and complex studies that are designed to eliminate bias and reduce uncertainty?
This really depends on how we assess the the quality of the evidence and consider the effects of our treatment. We could argue, that if we are providing a “low risk” treatment then we can operate with relatively large levels of uncertainty. An example of this could be a course of simple Class I non-extraction treatment in which all we need to achieve is alignment of the incisors with no change in buccal segment relationship. This type of treatment could be provided on a non extraction basis with any fixed appliance system. The next level of “risk” could be more complex treatment involving irreversible decisions, for example, the extraction of teeth. Finally, the highest level of risk is probably orthognathic surgery when the incorrect treatment decisions could result in major problems for our patients. Ironically, when we evaluate orthodontic research, it appears we have concentrated on reducing the uncertainty about various treatment mechanics, bracket selection and the use of screws to help maintain anchorage. We have not, yet, addressed the areas in which there is maximum uncertainty and risk. Clearly, we need to move on….
I have just heard that our Cochrane review into Class II treatment has just been published (http://goo.gl/W2YyZ). I will return to this subject and use the review to illustrate our confidence in the research into the treatment of Class II problems.
There was an interesting paper from Chapel Hill wrt ‘how much’ of the proposed tooth movement an appliance delivered. It compared Incognitio v Invisalign. So you draw the conclusion that Incognito removes a lot of uncertainty as it delivers 90 odd percent of planned tooth move net. However, theres always operator influence – just like driving a car…..
Yes, the operator influence does have an effect on the results of a study, however, if the study is sufficiently powered and structured, then this can be evaluated. We tried to do this in our early Class II treatment study but did not have enough power to come to firm conclusions. As regards the uncertainty, this is a new area for me and I will be continuing to look at this when I review other studies, starting with our Class II cochrane review next week
Kevin,
I think this posting on uncertainty and interpretation is very important and relevant and it is great that you are bringing it up. On a study we did a few years ago only 4-8% of articles in major orthodontic journals were reporting CIs and interpretation was based solely on p-values!!
Confidence intervals are also important for interpretation purposes when the results are not significant. I will borrow an example from medicine to further elaborate on this. Imagine that a new medication comes out that is less expensive and easier to deliver compared to the standard. The outcome is death and we compare the risk of dying between new vs. standard medication and find that the risk ratio is 0.71 in favor of the new medication with a 95% CI: 0.34 to 1.50. This result is not statistically significant since the 95% CI includes the value of 1. The RR=0.71 is telling us that the risk of dying is 29% (1-0.71) lower in the new medication group vs. the standard. However, the range of the 95% CI is telling us that the effect of the new medication can range from 66% (1-0.34, lower CI bound) protective to 50% (upper CI bound 1.50-1) harmful. The 1.50 value would indicate that the risk of dying is 50% higher in the new medication group vs. the standard. Therefore, interpretation on a non-significant p-value will be misleading and would indicate that there is no difference between the 2 medications, whereas the 95% CI is telling that the evidence is inconclusive.
Nick
This is a great example of why the orthodontic literature needs to catch up with the medical literature. We get excited about small values and tend to be wedded to p<0.05 and forget that we need to interpret the statistics. I will expand this when I discuss the updated Class II review that we got published and I will emphasise the uncertainty around some of the findings.
Kevin, I continue trying to post comments on this site, but keep batting zero. You do some great reviews, and I greatly enjoy reading your cogent remarks. I would also like to encourage you to eliminate the word distalize. Editors have allowed this linguistic abomination to increase without questioning it, e.g., we don’t lingualize, buccalize, intrudilize or extrudilize. So why distalize. In fact, we retract the molars or protract them.
Larry White
Hi Larry, Thanks for the comments. I will check and see if any of your posts are deep within the complexities of this blog site? You have made a very helpful point about tooth movement and its description and I agree with you. I will try to do this in future as it does make more sense
Best wishes Kevin