To the Editor:
Verhagen et al. (Reference Verhagen, de Vet and Vermeer5) suggest, in a study done in 2002, that methodologic quality of individual trials do not influence the conclusions of a landmark meta-analysis on thrombolytic therapy. This meta-analysis was studied because it was believed that it represents the true effect because its conclusions remain valid 15 years after publication. They incorporated the results of quality assessment in five different ways in the calculation of the pooled odd ratios (ORs): (i) component analysis, (ii) visual plot, (iii) quality score as a threshold score, (iv) quality score as a weighting factor, and (v) cumulative pooling. They did not find much discrepancy using either of these methods of quality assessment.
My concern is that, in a meta-analysis, the pooled effect size is a weighted average. The weight of one study in a meta-analysis is dependent on the weights allocated to other studies as overall they add up to 100 percent. As such, looking for the effects of quality by breaking up studies into subgroups is not a valid approach, because this will change the way the inverse variance weight is allocated and is a more important factor than quality per se. As such, using a component analysis or threshold score or cumulative pooling is not a valid approach to the assessment of the impact of quality.
Only two methods do not break up the studies into groups and one of these, the visual plot of quality score versus effect size, is also not useful because, as mentioned previously, the precision of the studies take precedence in a meta-analysis. As such if a high quality study has a bigger effect size but little precision, it makes little impact on the pooled estimate. The only valid approach then seems to be using the quality score as a weighting factor. This was done by the authors by adjusting the study variance by a factor 1/Qi (where Qi is the probability from zero to one that study i is credible) and then performing a standard fixed effects analysis (Reference Tritchler4). This is akin to downplaying the inverse variance weights of more biased studies, but leads to two problems. First, this form of adjustment fails to account for the direction of the bias induced by a quality deficiency and may end up nullifying the value of a quality score applied in this way (Reference Greenland and O'Rourke3). Second, Tritchler has demonstrated that the mean-squared error and confidence interval coverage are poor and there is a systematic bias in outcome with this form of adjustment (Reference Tritchler4). This is probably because the adjustments were made to individual study variances without consideration of the variability of the variance across studies. We rectified this by introducing a model that does away with these two limitations using a using a quality effects (QE) approach (Reference Doi2) and then applying this to this meta-analysis.
We now are able to show that, indeed, quality has an effect on the outcome of this meta-analysis. For example, if we remove the three European studies from the seventeen intravenous studies, the meta-analysis results (using the MIX version 1.7 software) (Reference Bax, Yu, Ikeda, Tsuruta and Moons1) are as depicted in Figure 1. These three studies had an average quality (score 5/9) and high precision. Only the quality effects model was resistant to bias induced by this imbalance introduced. It is clear, therefore, that quality has an impact on this meta-analysis, but only if assessed by means of an appropriate model.
Outcome of the meta-analysis by three models after exclusion of the three European trials (Reference Verhagen, de Vet and Vermeer5) (2nd European, 3rd European, and European Coop). The studies are in increasing order of standard error and cumulative plots are shown. The weight contribution of each study to the pooled effect size is given as a percentage on the left. What can clearly be seen is that, because the lower precision studies are of higher than average quality, only the quality effects model does not allow their weights to be drowned by the inverse of their variance. The random effects model tries to do a similar adjustment but fails because of two problems: An artificially inflated variance and between study variability used in lieu of quality. (a) Fixed effects model: odds ratio (OR) 0.85 (0.71 – 1.02). (b) Random effects model: OR 0.84 (0.66 – 1.06). (c) Quality effects model: OR 0.76 (0.6 – 0.97).
