In recent years, health technology assessments (HTAs) have become a primary source of evidence for healthcare decision makers confronted with questions about the comparative effectiveness and safety of interventions. The usefulness of such reports, however, depends largely on the availability of primary research addressing a particular question of interest. To determine the best treatment option, decision makers and patients must be able to compare the net benefits of any competing interventions.
Head-to-head randomized controlled trials (RCTs) are considered the gold standard for evaluating the comparative efficacy, effectiveness, and safety of healthcare interventions. Many competing interventions, however, have never been compared directly in RCTs (Reference Ioannidis18). Therefore, researchers who conduct HTAs often have to decide whether or not to use statistical methods to compare competing interventions indirectly.
Traditionally, researchers have dealt with the lack of direct evidence by qualitatively comparing effect sizes of competing interventions. For example, researchers often examine whether confidence intervals of different treatment effects overlap. In a drug class review on proton pump inhibitors (PPIs), researchers pooled esophagitis healing rates for each drug at 4 and 8 weeks, using PPI arms of studies with different comparisons (Reference McDonagh25). Authors presented pooled point estimates with confidence intervals for each drug in a table and concluded, “Healing rates were similar and confidence intervals overlapped, indicating no significant differences between PPIs.” Such qualitative assessments, although very common, are very susceptible to biases and confounders. A recent review of the Database of Abstracts of Reviews of Effects (DARE) found that 9.5 percent of meta-analyses used indirect comparisons (Reference Glenny, Altman and Song13). Few studies, however, carried out a formal statistical analysis to conduct adjusted indirect comparisons.
Increasingly, methods researchers have been evaluating the validity and reliability of various statistical methods to conduct indirect comparisons (Reference Baker and Kramer2;Reference Bucher, Guyatt and Griffith5;Reference Caldwell, Ades and Higgins6;Reference Glenny, Altman and Song13;Reference Lumley23;Reference Song, Altman and Glenny39). A recent HTA, conducted by Glenny and colleagues (Reference Glenny, Altman and Song13) for the NHS R&D (National Health System Review and Dissemination) Health Technology Program, is the largest, most thorough attempt so far.
The objective of this study is to summarize the current state of the evidence on the pros and cons of indirect comparisons using study level data. We conducted a Medline search using “indirect comparisons,” “adjusted indirect comparisons,” and “network meta-analysis” as text words because no respective MeSH (Medical Subject Heading) terms were available. In addition, we hand-searched reference lists of relevant methods articles. We set forth to illustrate alternative approaches of indirect comparisons and highlight advantages and disadvantages of different methods to provide a summary for researchers who are considering indirect comparisons and readers who are critically appraising results of such methods. Table 1 summarizes the existing evidence, the following paragraphs discuss findings in more detail.
Summary of the evidence
WHAT ARE INDIRECT COMPARISONS?
A typical example of a body of evidence with insufficient information from head-to-head trials is depicted in Figure 1. The comparison of interest is the efficacy of two second-generation antidepressants for the treatment of major depressive disorder: sertraline versus venlafaxine. Only two small studies have compared these drugs directly (Reference Mehtonen, Sogaard and Roponen26;Reference Sir, D'Souza and Uguz38) and rendered conflicting results. One study reported statistically significantly higher response rates (defined as an at least 50 percent improvement of symptoms on the Hamilton Depression Rating Scale) for patients treated with venlafaxine than with sertraline (83 percent versus 68 percent; p=.05) (Reference Mehtonen, Sogaard and Roponen26), the other trial found no differences in response between treatments (71 percent versus 71 percent; p=.95) (Reference Sir, D'Souza and Uguz38). Both studies lasted 8 weeks and had no major methodological flaws. Based on the available evidence, therefore, it is impossible to assess reliably the comparative efficacy of these two agents.
Body of evidence for the comparison of sertraline versus venlafaxine.
In addition to the two direct head-to-head studies, however, seven placebo-controlled trials on sertraline (17;Reference Kocsis, Zisook and Davidson20;Reference Lydiard, Stahl and Hertzman24;Reference Olie, Gunn and Katz30–Reference Reimherr, Chouinard and Cohn32;Reference Schneider, Nelson and Clary35) and four on venlafaxine (Reference Cunningham8;Reference Khan, Fabre and Rudolph19;Reference Mendels, Johnston and Mattes27;Reference Thase41) provide an additional body of evidence that can be used. Likewise, eighteen trials used fluoxetine as a common control intervention (Reference Alves, Cachola and Brandao1;Reference Bennie, Mullin and Martindale3;Reference Boyer, Danion and Bisserbe4;Reference Costa e Silva7;Reference De Nayer, Geerts and Ruelens9–Reference Fava, Judge and Hoog12;Reference Kroenke, West and Swindle21;Reference Nemeroff and Thase28;Reference Newhouse, Krishnan and Doraiswamy29;Reference Rudolph and Feiger33;Reference Schatzberg and Roose34;Reference Sechter, Troy and Paternetti36;Reference Silverstone and Ravindran37;Reference Tylee, Beaumont and Bowden42–Reference Van Moffaert, Bartholome and Cosyns44). Because of the inconclusive and sparse head-to-head data, indirect comparisons of these additional data around a common comparator can be taken into consideration as an additional analytic tool to achieve an estimate of the difference in efficacy between sertraline and venlafaxine.
METHODS OF INDIRECT COMPARISONS
Articles in peer-reviewed literature have used two general approaches to conduct such an indirect comparison: (i) unadjusted indirect comparison and (ii) adjusted indirect comparison. Figures 2 and 3 depict the general principles of these two methods.
Unadjusted indirect comparisons.
Adjusted indirect comparisons.
Unadjusted Indirect Comparisons
Unadjusted indirect comparisons make comparisons across trial arms, treating data as if they had come from one large trial. As illustrated in Figure 2, in a first step an unadjusted indirect comparison would combine the summary statistics for study arms involving intervention A to create a weighted overall summary measure. The same would be done for intervention B. In a second step, analysts would then compare these two overall summary measures. Data from the comparative arms are not used.
The principal methodological problem with this approach is that it negates the randomized nature of individual trials. Combining data across arms creates two large cohorts that may or may not be comparable in prognostic (or other potentially confounding) factors. Glenny et al. (Reference Glenny, Altman and Song13) termed this approach “the naïve method.” When compared with direct estimates, this approach showed a high rate of discrepancies in unpredictable directions. In their simulation studies, this approach was unbiased but showed overly precise estimates as compared to other valid methods (adjusted indirect, logistic regression, and meta-regression). In addition, with such an approach, critics can legitimately question the exclusion of nonrandomized or observational studies. Therefore, unadjusted indirect comparisons should always be avoided.
Adjusted Indirect Comparisons
In adjusted indirect comparisons, analysts adjust the comparison of the interventions by the results of their direct comparison with a common control group (e.g., placebo or an active comparator), at least partially using the strength of the original RCTs (Reference Song, Glenny and Altman40). As illustrated in Figure 3, in a first step the effects of interventions A and B are compared with a common comparator (in our example P). The relative effects of A versus P are then compared with the relative effects of B versus P. For example, we can obtain an indirect estimate of the difference of benefits of sertraline versus venlafaxine by comparing the effect sizes of trials of sertraline versus placebo with those of trials of venlafaxine versus placebo. The underlying rationale is that, if the treatment effect of sertraline versus placebo is similar with that of venlafaxine versus placebo, then the comparative benefits of sertraline and venlafaxine will be similar as well. Of course, any common comparator can be used in place of placebo.
In adjusted indirect comparisons, data are not treated as if they had come from one large trial; rather, individual studies are maintained as the units of the analysis. As in a meta-analysis, investigators construct effect estimates of individual trials, weight them appropriately, and then combine them into a summary effect estimate. In adjusted indirect comparisons, then, they compare the summary effect estimates of the two sets of trials.
Conceptually, this follows a simple principle:
where θ is the estimate of effect (e.g., log odds ratio, log relative risk, or means).
Using our second-generation antidepressants example from above, the equation would read as follows:
![\begin{eqnarray*}
{\rm sertraline}\,{\rm versus}\,{\rm venlafaxine} &=& [{\rm sertraline}\,{\rm versus}\,{\rm placebo}]\\
&&-\,[{\rm venlafaxine}\,{\rm versus}\,{\rm placebo}]
\end{eqnarray*}](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20160921054824039-0946:S0266462308080240:S0266462308080240_eqnU2.gif?pub-status=live){kind=small}
Important for the validity of the result of an adjusted indirect comparison is its dependence on the consistency of the treatment effect across the two different sets of trials. Theoretically, treatments of the two sets of trials should be interchangeable without leading to a different conclusion. Differences in any factors that can modify the treatment effect between the two sets of trials will lead to biased results. For example, patients treated with sertraline must have similar prognostic factors, such as severity of disease, prior depressive episodes, and others, as patients treated with venlafaxine. To some extent, this requires that study protocols of trials included for indirect comparisons have to be similar.
In practice, however, trials can vary in numerous ways: population characteristics, precise interventions, cointerventions, length of follow-up, loss to follow-up, and study quality. Some of these factors can easily be assessed and controlled for in regression models; others require assumptions that may not be verifiable. It is impossible to clearly determine to what extent these differences might bias results of indirect comparisons. Because indirect comparisons essentially constitute an observational study, residual confounding can always be present. The Cochrane Collaboration handbook for authors cautions that indirect comparisons may suffer all biases that can occur in observational studies (Reference Higgins and Green16).
DIFFERENT APPROACHES FOR THE ANALYSIS OF ADJUSTED INDIRECT COMPARISONS
In their methods study, Glenny et al. (Reference Glenny, Altman and Song13) evaluated four statistical approaches: (i) logistic regression, (ii) random effects meta-regression, (iii) fixed effects adjusted indirect comparisons [Bucher], and (iv) modified (random effects) adjusted indirect comparisons [Bucher] using simulations. (Note: we will refer to the method proposed by Bucher et al. as adjusted indirect comparisons [Bucher] (Reference Bucher, Guyatt and Griffith5) to distinguish this statistical method from the term “adjusted indirect comparisons” that is used as an umbrella term, as outlined earlier and depicted in Figure 3. Results of these simulation models indicated that, under ideal circumstances (i.e., no differences in prognostic factors existed among included studies), all four methods resulted in unbiased estimates of direct effects (Reference Glenny, Altman and Song13). For the simulated scenario, fixed effects methods (adjusted indirect comparisons [Bucher] and logistic regression) underestimated the true variability in the measure of association. In practice, the correctness of their standard errors would be impossible to test.
Although all four statistical approaches led to valid results, they differ in several characteristics that might guide the choice of a method. Logistic regression is limited to dichotomous outcomes and odds ratios as the measure of association. By contrast, meta-regression and adjusted indirect comparisons [Bucher] can be extended to risk ratios, risk differences, and continuous outcomes. Meta-regression and adjusted indirect comparisons [Bucher], however, are limited to trials with two treatment arms. Glenny and colleagues (Reference Glenny, Altman and Song13) support the use of random effects for either of these approaches. Analysts can readily use logistic regression, meta-regression, and adjusted indirect comparisons [Bucher] with standard statistical software such as SAS or Stata (see Glenny et al. Appendix 7). An advantage of logistic and meta-regression models is the possibility to adjust for potentially confounding factors, especially trial-level covariates such as length of follow-up, cointerventions, dosages, and others. Figure 4 summarizes considerations for different approaches graphically.
Algorithm for the choice of a statistical model for indirect comparisons.
A disadvantage of all four approaches is that only trials with one common comparator can be included. Often, however, poor internal validity and heterogeneity severely limit the number of eligible trials.
More complex methods, such as network meta-analysis (Reference Lumley23) or Bayesian methods (Reference Caldwell, Ades and Higgins6;Reference Lu and Ades22), were not evaluated by Glenny and colleagues (Reference Glenny, Altman and Song13). Such approaches can handle less restrictive scenarios, but their use requires advanced statistical packages and expert statistical assistance. Network meta-analysis (Reference Lumley23), for example, allows analysts to use direct evidence and accommodates multiple common comparators. With network meta-analysis, researchers may be able to use a greater overall number of trials, but depending on the effect of interest and complexity of “networks,” the increase in power is not clear (Reference Lumley23). Network meta-analysis attempts to estimate the inconsistency (“incoherence”) for treatments effects with respect to evidence in the network.
HOW MANY STUDIES ARE NEEDED FOR INDIRECT COMPARISONS?
No guidance exists on how many studies are necessary to conduct indirect comparisons. If only few studies or only studies with low event rates are available, adjusted indirect comparisons can still render correct point estimates, but the precision of the estimates will be low. Essentially, they will lead to indeterminate results with wide confidence intervals. Confidence intervals of such findings often encompass clinically significant and nonsignificant differences, leading to indeterminate conclusions. Four times as many equally sized studies are necessary for an adjusted indirect approach to have the same power as a direct comparison (Reference Glenny, Altman and Song13). As stated above, analysts can address this issue in part by using more complex statistical approaches, such as network meta-analysis, that are not limited to one common comparator. However, these more complex methods require more assumptions, which are difficult, if not impossible, to verify, especially with a small number of studies, small sample sizes, or low number of events (i.e., small degrees of freedom).
CAN RESULTS FROM INDIRECT COMPARISONS AND HEAD-TO-HEAD TRIALS BE COMBINED?
Bucher and colleagues suggest that indirect comparisons should be conducted only when direct evidence is entirely missing (Reference Bucher, Guyatt and Griffith5). The Cochrane Collaboration handbook recommends that, in situations when both direct evidence and indirect evidence are available, the two approaches should be considered separately (Reference Higgins and Green16). They also recommend that, unless methodological issues compromise the head-to-head trials, direct evidence should take precedence as a basis for forming conclusions.
Glenny et al. (Reference Glenny, Altman and Song13) and Song et al. (Reference Song, Altman and Glenny39) take a more pragmatic approach, pointing out that sometimes direct randomized evidence is limited to few, often small studies with variable internal validity. Therefore, better, or at least more but equivalent information, or indeed information of higher quality might be available from indirect comparisons. Glenny and colleagues (Reference Glenny, Altman and Song13) recommend that analysts assess the combinability of studies by evaluating the similarity of study participants and interventions. They argue that statistical differences in direct and indirect estimates should not be the primary factor on which to make a decision whether to combine results. As in any RCTs, methodological issues (e.g., lack of allocation concealment, high differential loss to follow-up, and others) can compromise head-to-head RCTs and lead to biased results. Furthermore, head-to-head trials are often not powered to assess clinically relevant differences with enough statistical power to establish statistical significance (Reference Gotzsche14;Reference Greene, Concato and Feinstein15). Particularly, head-to-head trials that were conducted primarily for marketing reasons often have small sample sizes. In such trials, even with sound randomization and allocation concealment, there can be substantial differences of prognostic factors between groups at baseline, resulting in selection bias.
Therefore, basing the decision on whether or not to combine direct and indirect estimates on the similarity or dissimilarity of effect estimates appears to be less valid than a careful examination of the similarity of study participants and interventions. If populations seem to be similar, then analysts can combine results from direct and indirect approaches, regardless of differences in effect estimates. Nevertheless, such an approach is based on theoretical considerations rather than on empirical evidence. Ultimately, deciding whether to combine direct and indirect estimates will be a matter of informed judgment. Statistically, results from direct and indirect comparisons can be easily combined using a weighted combination (inverse variance weighted average).
HOW SHOULD ANALYSTS HANDLE A SITUATION IN WHICH RESULTS FROM DIRECT AND INDIRECT COMPARISONS CONFLICT?
Glenny et al. (Reference Glenny, Altman and Song13) identified twenty-eight systematic reviews (with forty-four comparisons) in DARE in which enough data were available to conduct both direct and indirect comparisons. Overall, they observed a moderate agreement between direct and indirect comparisons (weighted kappa: 0.53). In three of forty-four comparisons, the differences between the direct and adjusted indirect comparisons reached statistical significance. Song et al. (Reference Song, Glenny and Altman40) compared results derived from direct and indirect comparisons based on “real world” systematic reviews. They reported considerable discrepancies between the two methods, but these could have been a result of the very small number of trials that were used in each systematic review (three to seven total trials).
No clear guidance yet exists on how to proceed when direct and indirect evidence conflicts. As mentioned above, evaluating the similarity of study participants and interventions appears to be the most constructive approach. Dissimilarities in study populations or interventions may explain conflicting results. If no dissimilarities exist, then results could be combined. However, presenting both combined and individual results may be the best and most informative approach.
To gain additional evidence on the comparative efficacy of sertraline versus venlafaxine, we conducted a meta-regression of placebo controlled studies. Results indicate that sertraline and venlafaxine have a similar antidepressive effect. The relative risk of achieving response is 0.88 (95 percent confidence interval, 0.72 to 1.07), which is consistent with the head-to-head trial (Reference Sir, D'Souza and Uguz38) that indicated no difference in efficacy. Given the insufficient and conflicting data from head-to-head trials, these findings provide additional evidence to draw conclusions about the comparative benefits of these two antidepressants.
CONCLUSIONS
In the absence of sufficient head-to-head evidence, adjusted indirect comparisons can be viewed as an additional analytical tool to determine the comparative efficacy and effectiveness of competing interventions. Well-conducted methodological studies provide good evidence that adjusted indirect comparisons, when well done, can lead to results similar to those with direct comparisons. The internal validity of several statistical methods to conduct indirect comparisons, therefore, has been established. These methods studies, however, have been conducted under ideal circumstances. Researchers who consider indirect comparisons need to keep the limitations of indirect comparisons under “real-world” conditions in mind. First, given the limited information in many publications, the validity of indirect comparisons often has to be based on an unverifiable assumption. That assumption, essentially, requires that the effects of compared interventions be consistent across the different settings, populations, cointerventions, and other potentially confounding factors in the two sets of trials. Second, indirect comparisons are low in power, frequently leading to indeterminate results with wide confidence intervals. Inferences that can be drawn from such findings may be limited. Third, no consensus exists on how to interpret indirect results that differ substantially from direct evidence or on how to weigh findings of indirect comparisons against those results from nonrandomized direct evidence.
Many additional questions remain. At present, no consensus exists as to when conducting indirect comparisons or combining results from direct and indirect comparisons is appropriate. Furthermore, methods research has mainly focused on dichotomous outcomes, namely, to odds ratios. Future research needs to evaluate other binary outcome measures (e.g., risk ratios), continuous data, and time-to-event data. In addition, more complex statistical methods, such as network meta-analysis or Bayesian approaches that may increase power and precision by using multiple common comparators, have not been thoroughly validated. Future research also has to determine how indirect comparisons contrast with nonrandomized direct evidence in terms of accuracy of findings.
CONTACT INFORMATION
Gerald Gartlehner, MD, MPH (gerald.gartlehner@donau-uni.ac.at), Associate Professor, Department of Evidence-based Medicine and Clinical Epidemiology, Danube University, Karl Dorrek Straße 30, Krems, Austria, 3500
Charity G. Moore, PhD (moorecg@upmc.edu), Visiting Associate Professor, Department of Medicine, University of Pittsburgh, 200 Meyran Avenue, Suite 300, Pittsburgh, Pennsylvania 15238
