The Randomised Trial and the Retrospective investigation: Looking forward or back?
Looking forward or back? The randomised trial and the retrospective investigation
At the last European Orthodontic Congress they held a debate on the value of the randomised trial versus the retrospective investigation.
When I initially heard about this I felt that this was a redundant topic, as people have debated this many times in the past. I could not attend the meeting, but I was very pleased to see that the EOS has put the debate on their website for open access.
The debate was called “Randomized Controlled Trial: The gold standard or an unobtainable fallacy”? and it can be found here: http://www.oxfordjournals.org/our_journals/eortho/ejovideo.html
The two speakers, who led the discussion, were Lars Bondemark who has carried out several trials and he spoke for the value of trials and the opposing view was expressed by Sabine Ruf, who has reported many retrospective case control studies, who spoke for the use of case control studies.
I thought that the debate was very interesting and they went over many of the “classic” arguments for and against trials. In summary, the conclusions were that trials should be carried out to minimize bias, but they are difficult to successfully run but this does not mean that we should not do them! The opposing view, put forwards by Sabine, was that while we accept that retrospective investigations do have high levels of bias, they are easier to run, inexpensive and if we acknowledge the deficits and ensure that our collection of records is good, then they provide us with useful information. I agree with both of these viewpoints. However, we need to be cautious when we consider case control studies, in that the ideal situation as suggested by Sabine very rarely exists because of the way that we collect the records that to include in retrospective studies. I shall illustrate this further
The Classic Retrospective study.
In my early research years I carried out several retrospective studies. This was because this was the way we did research and we had not really considered carrying out trials. So I have first hand experience of these methods. It is also clear that this type of study tends to be done as a Masters level as they can be completed in a short length of time.
Classically the method that is adopted is to develop a study question based on the availability of records that have been previously collected. While this is convenient, there is a problem because the quality of the study relies on the completeness of the records. This is very relevant when we consider that there is a tendency for orthodontists to collect records of the patients whose treatment went well, as we like to collect the “precious things” and look at them from time to time. Furthermore, we do not always collect records for the patients whose treatment did not go well. As a result, unless we can guarantee that all the records have been collected on every patient who is eligible for the study, then we have to assume that bias is present. It could be suggested that this bias is going to be towards the positive because the records do not include those treatments that “did not do so well”.
Can I prove this?
I fully appreciate that over the years I have put forwards this argument based on the results of studies carried out in other areas of health care. So, does this happen in orthodontics? I have done a small pilot study to illustrate this concept and like all pilots this is a very low level of evidence. Nevertheless, this is what I did.
I took the data on overjet change from the three classic RCTs on the effectiveness of early treatment for Class II malocclusion. These were the studies carried out in North Carolina, Florida and Manchester. Details of these are found in the Class II Cochrane Review http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD003452.pub2/abstract
I then took the same results from three retrospective case control studies into the same question and put them into a simple meta analysis. These studies are found on
The analysis revealed that for the RCTs the mean overjet reduction was 3.2mm and for the retrospective studies this was 4.3mm. This is a difference of only 1mm but we need to remember that orthodontic research deals with this level of measurement. I am going to expand this study in a more systematic way, as I am sure that it will lead to some interesting data. I will report back in due course on this blog and in the published literature.
What about a collection of completed cases?
In the question and answer section of the EOS debate, several people made the suggestion that we should all collect the records of completed treatments and then enter these in a database. This could then be used for research purposes. It is clear that one advantage of this approach is that it does not involve randomisation and is an observational study of treatments that are already being provided by experienced clinicians. However, the only advantage of this approach over the retrospective study would arise if the patients were enrolled at the start of their treatment. In reality, this would be a prospective cohort investigation. I have been involved in several of these studies, when it was not possible to randomise, and they involve a similar amount of work to a trial, as patients need to be entered and followed through to completion. This methodology clearly has a place when it is not ethical to randomise,unfortunately, they are subject to bias because the treatment is not allocated randomly. As a result, they are not an alternative to a trial when the operators are in equipoise and are prepared to randomise.
What are the other advantages of the prospective investigation?
Aside from the compelling advantage that in a trial, the study question is clearly defined and drives the randomization and data collection to minimize bias. One other advantage of a prospective study is that we can plan it in advance to include patient values. We can measure these using standard patient questionnaires on important variables, such as self-esteem etc, but it is also possible to build in a qualitative component to trials. This can yield incredibly useful information that is relevant to our patients. Whereas, in a retrospective study it is not possible to obtain this information, as all we have is study models and cephalograms. I know that some orthodontists treat their collection of models as their closest friends but they cannot talk to them!
Importantly, if we include patient values in our research, we can move orthodontic research away from the collection and reporting of tables of figures from cephalograms that lack meaning, towards research that informs us of our patients perceptions and feelings. This can only be done in prospective investigations. This convinces me that the question that we considered earlier on the value of trials vs retrospective studies is even more redundant.
I know that my views are not universally accepted and it would be great if I could have some comments back and we can have a debate on this blog?