1. Introduction

Randomized controlled trials (RCTs)—trials where patients are randomly assigned to a treatment and a control group, while controlling for possible confounders—are currently the gold standard within evidence-based medicine (Worrall Reference Worrall2007). Usually they are conducted as sequential trials allowing for monitoring for early signs of effectiveness or harm.

In sequential trials, data are typically monitored as they accumulate. That is, we have interim looks at the data, and we may decide to stop the trial before the planned sample size is reached. By terminating a trial when overwhelming evidence for the effectiveness or harmfulness of a new drug is available, we can bound the prohibitive costs of a medical trial and protect in-trial patients against receiving inferior treatments. Thus, monitoring contributes to meeting ethical and epistemic requirements that clinical investigators are confronted with.

However, the early termination of sequential trials raises an important ethical concern: Is it mandatory to stop a trial as soon as the new treatment shows convincing signs of superiority? Or should the trial be continued in order to achieve a result that would convince the wide medical community of the superiority of the new treatment? On the one hand, the health of actual patients must not be jeopardized by administering an inferior treatment; on the other hand, establishing sound and univocal scientific conclusions will facilitate an effective cure of future patients.

The issue is complicated by the fact that evidence from trials stopped early is often met with skepticism in the medical literature: “RCTs stopped early for benefit … show implausibly large treatment effects. … Clinicians should view the results of such trials with skepticism” (Montori et al. Reference Montori2005, 2203). This standpoint is affirmed by the recent STOPIT-2 metastudy where Bassler et al. (Reference Bassler2010, 1187) blame truncated RCTs for “appreciable overestimates of effect.”

While we cannot adjudicate the far-reaching question about the ethical legitimacy of monitoring, we side with Worrall (Reference Worrall2008, 418) that “no informed view of the ethical issues … can be adopted without first taking an informed view of the evidential-epistemological ones.” In particular, we think that the skeptical attitude about trials stopped early for benefit stems from a fallacious statistical interpretation of such trials. These misinterpretations are, to our mind, mainly caused by a lack of awareness about issues in statistical methodology that also troubles other disciplines, such as economics and psychology. Indeed, the two grand schools of statistical inference—Bayesian and frequentist inference—are in outright conflict about how to plan and to evaluate a sequential trial.

Our essay takes the following route. First, we expose the arguments for and against the presence of bias in early stopped trials and explain why this problem is related to principled questions in statistical methodology (sec. 2). Subsequently, we argue that the real problem is the use of unconditional error assessments in sequential trials, rather than the often-invoked divide between Bayesians and frequentists (sec. 3). Then we show that conditional frequentist tests reconcile the need for valid postexperimental appraisal of the evidence with the realities of the current regulatory framework in medicine and, in particular, with the implied preference for frequentist analysis (sec. 4). Finally, we wrap up our results and sketch how a superior methodological framework can improve the design and practice of sequential trials and eventually lead to better decisions (sec. 5).

2. The Assessment of Truncated Trials

The practice of stopping RCTs early for benefit has been subject to severe epistemological criticism. Skepticism surrounds the results of these trials, due to the fact that they show implausibly large treatment effects, relative to what the medical community would be inclined to expect. In a review of 134 trials stopped early for benefit, Montori et al. (Reference Montori2005) point to an inverse correlation between sample size and treatment effect: the smaller the sample size achieved by the trial at the moment of stopping, the larger the estimate it provided for the effect. These findings are supported by a more recent study by Bassler et al. (Reference Bassler2010), where truncated trials report significantly higher effects than trials that were not stopped early.

Some prominent cases seem to corroborate this skepticism. Mueller et al. (Reference Mueller2007) report a case of two leukemia treatments where interim analyses suggested a high relative risk reduction (53% and 45%) in a particular chemotherapy regimen. However, that assessment had to be reversed after completion of the trial. In the medical community, such cases fuel mistrust toward anticipated claims of benefit and nourish the fear of promoting a treatment that is actually less efficacious. Therefore, stopping a trial early might lead to a result that the medical community does not trust, canceling the epistemic and ethical benefits that monitoring possesses in the long run.

However, not all methodologists share this pessimistic view on trials stopped early. Goodman, Berry, and Wittes (Reference Goodman, Berry and Wittes2010) observe that pronounced effect size differences between truncated and completed trials are actually predictable: highly efficacious treatments will naturally be more prone to early termination for benefit. Hence, the observed difference in estimated effect size is precisely what we should expect. Comparing truncated to completed trials amounts, as highlighted by Berry, Carlin, and Connor (Reference Berry, Carlin and Connor2010), to selecting the trials to be compared on the basis of their outcome.

In this context, prior knowledge or empirically based prior expectations are highly relevant for sound decision making. Unfortunately, at present they enter the final decisions only in a methodologically unsatisfactory ad hoc way. This observation suggests that systematic use of Bayesian inference may address the problem. A Bayesian represents subjective uncertainty by means of a prior probability distribution over the values of the quantity of interest (e.g., relative risk reduction). By means of Bayes’s theorem, this distribution is updated to a posterior probability distribution that synthesizes the observed evidence with the background knowledge.

In the Bayesian framework, implausibly large observed effects can be balanced by prior expectations and lead to a more conservative conclusion than in standard frequentist methodology. In particular, it can be explained that truncated trials provide, ceteris paribus, less confidence than trials with a comparable effect size that were completed (Goodman Reference Goodman2007). The smaller the actual sample, the more will the posterior distribution resemble the prior distribution, for a given effect size. So it appears that the worries of Montori et al. (Reference Montori2005) and Bassler et al. (Reference Bassler2010)—overestimation of treatment effect in truncated RCTs—could be alleviated by switching the statistical framework.

Despite the advantages just outlined, there are some serious counterarguments to the viability of Bayesianism in clinical trials. First of all, the specification of a prior probability function (and a decision model) is problematic in a number of ways (see Moyé Reference Moyé2008). Second, in Bayesian statistics, experimental design is apparently irrelevant for the postexperimental conclusions. This is unacceptable to regulatory bodies that are keen to promote proper design of medical trials as a means to ensure the validity of trial results (e.g., Food and Drug Administration 2010).

Even though some of these worries are regulatory rather than epistemological, they are certainly legitimate. Indeed, we believe that solving the interpretational problems with truncated trials does not require one to pass from the frequentist to the Bayesian paradigm. As we will argue in the upcoming sections, it is more fruitful to turn to a different distinction: namely, to replace unconditional by conditional procedures.

3. Problems with Unconditional Inference in Sequential Medical Trials

Sequential medical trials usually control the reliability of a testing procedure from a preexperimental point of view, by means of type I and type II error rates. These error probabilities are extremely important for proper experimental design, and they get a lot of attention from a regulatory point of view. Moreover, Mayo and Kruse (Reference Mayo, Kruse, Corfield and Williamson2001) have argued, among others, that if the sampling plan is violated, error probabilities cannot be properly controlled and are actually inflated far beyond acceptable.

However, adherence to a proper sequential sampling plan is not sufficient to secure a reliable result. Arguably, what is most disturbing to the medical community is the fact that, according to current procedures, a truncated trial has prima facie the same reliability as a trial carried to the planned end. This is because Neyman and Pearson’s type I and II error rates are unconditional quantities; that is, they are insensitive to whether the data are just at the significance boundary or far beyond it.

In line with this observation, we contend that the unconditional nature of Neyman-Pearson hypothesis tests is the culprit for their epistemological shortcomings. To motivate this claim, we walk the reader through an example by Cox (Reference Cox1958) and Royall (Reference Royall1997, 74–75). Suppose that we test against with known and that the toss of a fair coin decides whether we draw N = 1 or N = 100 independently and identically distributed observations. It seems natural to apply the most powerful test at the 5% level in either case. However, the probabilistic mixture of the two most powerful tests at the 5% level is not the most powerful test in the overall experiment. We can do better if we reject for in the case of N = 1, while rejecting if in the case of N = 100. Both procedures are tests at the 5% level, but the second, “gerrymandered” test has a greater power (69%) than the mixture of unconditional tests (63%).

One may be inclined to dismiss the second test because not all of its components are tests at the 5% level. In the N = 1 case, the nominal significance level of the test is 10%. However, from an unconditionalist (preexperimental) viewpoint, only the overall error rates should count. Here, the superior power features speak for the second, gerrymandered test. This feature of the prevalent Neyman-Pearson methodology reveals the tension between the preexperimental design of unconditional procedures and the need to efficiently learn from the actual data. Unconditional error rates and confidence intervals do not address that second goal: “Now if the object of the analysis is to make statements by a rule with certain specified long-run properties, the unconditional test … is in order. … If, however, our objective is to say what we can learn from the data we have, the unconditional test is surely no good” (Cox Reference Cox1958, 360). The example can, of course, be easily generalized. It undermines the view that unconditional, preexperimental error probabilities can qualify the goodness of an inference.

Therefore, practitioners who rely on unconditional procedures have to find informative and reliable postdata assessments of the evidence. Often, they report the observed p-value to quantify the conclusiveness of the rejection of the null. However, p-values really combine the worst of all worlds. Since comprehensive and devastating criticisms of using p-values in scientific experiments have been delivered elsewhere (Royall Reference Royall1997; Goodman Reference Goodman1999); we only mention their most fundamental failures: they neither possess a valid frequency interpretation nor do they provide a useful measure of confidence in the null hypothesis.

Moving to confidence intervals is often suggested as a way of circumventing the p-value problem (e.g., Cumming and Finch Reference Cumming and Finch2005). However, a 95% confidence interval merely specifies the set of parameter values that are consistent with the observation at the 95% level. This does not mean that we should have 95% confidence that the confidence interval includes the parameter value. In fact, the degree of confidence is just an average coverage rate over intervals from repeated random samples; it is not the coverage probability of the one particular interval that the investigator happens to get. Therefore, some confidence intervals may include the entire sample space (see Seidenfeld Reference Seidenfeld1981), raising the question of whether the entire notion is a misnomer.

These problems of unconditional inference can be overcome by conditioning on the relevant chunks of information. In the next section, we will see how conditional inference may resolve the methodological confusion about interpreting truncated RCTs without abandoning the framework of frequentist statistics.

4. Conditional Frequentist Inference

Conditional inference tries to improve upon unconditional procedures by quantifying the degree of confidence that we can have in our conclusions as a function of the observed evidence. More precisely, conditional inference builds on the strength of the observed evidence. As we will show in this section, it can be justified from both the Bayesian and the frequentist perspective. The idea comes up for the first time in Cox’s (Reference Cox1958) seminal paper and has been developed later by Kiefer (Reference Kiefer1977) and Berger (Reference Berger2003), together with various coauthors.

The main idea can be motivated by a very simple example (Kiefer Reference Kiefer1977; Berger Reference Berger2003). Two observations and are taken with probability law

If we now construct a confidence interval for , then the interval defined by

has an unconditional coverage of 75%. Yet this does not seem to be a sensible conclusion regarding the confidence that the data warrant with respect to the true value of . Dependent on whether we observe or , we are entitled to a statement with (a posteriori) confidence 50% and 100%, respectively. The unconditional coverage of 75% neglects that, after learning the strength of the evidence (i.e., the value of ), we are in a much better position to assess the confidence which the data grant about our inference. Thus, conditioning on the value of improves the accuracy of our conclusions (see Cox Reference Cox1958, 361–63).

It is also noteworthy that the probability distribution of does not depend on the value of . That is, is an ancillary statistic with regard to . In particular, conditioning on the value of is quite different from Bayesian conditionalization: where Bayesians change their subjective probability distributions by conditioning on the entire data, conditioning on the value of just helps to better appreciate the (frequentist) interpretation of the data.

If this idea is applied to hypothesis testing, which is the major issue in medical trials, unconditional error rates are replaced by conditional error probabilities. In the following, we will outline the basic idea of conditional tests, following Berger, Brown, and Wolpert (Reference Berger, Brown and Wolpert1994).

Consider, for the purpose of mathematical convenience, the case of testing a point null hypothesis against the simple alternative in some probability model . Define and as the probability densities of data under the hypotheses and , and let the Bayes factor be the ratio of the probability density functions. Now, let and be the cumulative distribution functions corresponding to the Bayes Factor.

We now divide into a partition defined by

The different represent, intuitively, different observed strengths of evidence. This can also be made precise mathematically: under the assumption , which is satisfied for many distributions used in practice, the have the same probability density under and , for all values of s. In other words, their distribution is independent of which hypothesis is true (Berger et al. Reference Berger, Brown and Wolpert1994, 1789–90).

This ancillarity property is shared with the statistic in the above toy example. Therefore, is excellently suited for the purpose of conditioning: to take the observed strength of the evidence into account without already telling us something about the parameter of interest. Thus, conditioning exploits a crucial strength of Bayesian paradigm—to identify a sensible measure of evidence—without assigning a subjective probability to competing hypotheses.

The conditional error probability can now be calculated by conditioning on the particular set in which the observed data fall. In particular, we can define a conditional frequentist test by

and for observed , we report conditional error probabilities

where the latter equalities have been proven by Berger et al. (Reference Berger, Brown and Wolpert1994, theorem 1). Clearly, by using the conditional instead of the unconditional error probabilities, we gain a much better appreciation of the chance of a wrong decision, given the particular data that we have observed. The higher the Bayes factor, the more confident we can be about an acceptance of the null and vice versa. In particular, the classical, unconditional test just detects whether the data are within or outside the rejection region (and leaves the rest to the notorious p-values) whereas the conditional test allows for a fine-grained, properly frequentist discrimination among trials with significant outcomes.

We turn now to briefly discussing a couple of objections that could be made from within the frequentist perspective. First, it could be argued that makes it far too easy to reject the null () whereas in medicine, evidence has to be really strong before we are convinced of the efficacy of a new treatment and approve of the drug. To this we simply respond that has been selected because of its simplicity, but we can easily change the rejection region according to contextual requirements. To obtain a sensible conditional test, we will often have to use nonancillary conditioning statistics and to include a no-decision region (Berger, Boukai, and Wang Reference Berger, Boukai and Wang1997, 145–47). However, these features align well with the caution toward premature conclusions that prevails in the medical community and do not pose any problem for the practitioner.

Second, there may be worries about the scope of the above procedure, which we have only explained for the easiest possible case of hypothesis testing. However, Berger et al. (Reference Berger, Boukai and Wang1997) have extended conditional tests to simple versus composite testing problems and, in particular, to the two-sided null hypothesis testing problems that frequently occur in RCTs.

Third, the use of the Bayes factor may indicate that the conditional test is actually a Bayesian test in frequentist cloths. Indeed, for impartial priors the posteriors

just correspond to the conditional error probabilities for rejecting and accepting , respectively. However, possesses a frequentist interpretation, too, since it identifies the most powerful frequentist test in the simple versus simple testing problem.Footnote ¹

Thus, Bayesians and frequentists can conduct the same (conditional) test and obtain the same numerical conclusions. For the medical practitioner, philosophical questions about the interpretation of probability are clearly secondary as long as there is methodological agreement on procedures and postexperimental data assessment (see Berger Reference Berger2003). In this sense, conditional inference is a genuine reconciliation of Bayesian and frequentist methodology and a real asset for practitioners.

We would like to conclude this section by means of an application of conditional inference to sequential medical trials. The example involves a trial for adjuvant therapy in resectable hepatocellular carcinoma (Lau et al. Reference Lau1999). The trial was stopped early based on interim findings, but additional data were available after the decision to stop was made. Pocock and White (Reference Pocock and White1999) describe the situation in detail:

Such shrinkage of the estimated benefit between the interim and the final analysis is precisely what fuels clinicians’ worries about “stopping on a random high” and adds to their skepticism about truncated trials. In this situation conditional error rates can provide real guidance. We set up an alternative hypothesis according to Lau et al.’s (Reference Lau1999) expectations that “131I-lipiodol would reduce the rate of recurrence [postulated to be 50%] by 50% and double the disease-free survival rate” (Reference Lau1999, 798).

Using this value in calculation of the Bayes factor yields a conditional type I error rate of at the interim analysis, instead of the unconditional error rate of .Footnote ² Moreover, we can dismiss the apparently strong unconditional p-value of , which is just indicative of an unexpectedly high performance. By contrast, the conditional error reflects the greater statistical uncertainty associated with the small sample when the decision to stop the trial was made. At the end of the trial, the conditional test still rejects the null, but the probability of error is now higher: the calculation based on yields a 14% probability of error, which is in line with the reservations of the clinicians involved.

We now briefly wrap up the advantages of conditional over unconditional inference. First, the assessment of the error probability depends on the observed data and is thus way more informative than in the unconditional framework. This alleviates the interpretational problem mentioned in section 2, since conditional error allows medical readers to assess the confidence in the outcome based on the observed data. Clearly, medical investigators should be more concerned with the actual probability of drawing the wrong inference than with the absolute (unconditional) error rate of the testing procedure, also because clinicians have to make ethical decisions for their actual patients (see Nardini Reference Nardini2013).

As a further point, the error probabilities (3) and (4) are independent of the stopping rule, that is the sampling plan determining when the trial is terminated. In an RCT, the stopping rule can never be fully specified, since one cannot cover in advance all eventualities that might happen during a sequential trial. Independence from the stopping rule entails that interpretation of the results and assessment of error are possible even if the stopping rule was misspecified or could not be adhered to due to unforeseen circumstances. This is a substantial practical asset (see Sprenger Reference Sprenger2009).

This should not be misunderstood as the claim that predata analysis and experimental design are superfluous. Unfortunately, Berger et al. (Reference Berger, Brown and Wolpert1994, 1803) make a claim in that direction, but given the strong emphasis on careful design by methodologists and regulatory bodies (see Moyé Reference Moyé2008; Food and Drug Administration 2010), this is unlikely to increase the acceptance of the conditional approach among medical practitioners. We would like to stress that no such claim is required for making a case for the superiority of the conditional frequentist approach. Moreover, since conditional tests can be conducted from both a Bayesian and a frequentist perspective, practitioners do not have to decide for either camp.

Finally, there are interesting implications for the philosophy of statistics: if the “error statisticians” (e.g., Mayo Reference Mayo1996) are right that learning from error is indeed a cornerstone of inductive inference, then a move to conditional inference may protect their framework against the objections that we have mentioned in section 3. In particular, there is no need to tie an error-statistical methodology to unconditional inference. However, further developing this line of thought goes beyond the scope of this article.

5. Conclusions

In this article, we have analyzed the impact of statistical methodology on a substantive ethical and societal question, namely, data monitoring in sequential medical trials. In the medical literature, trials stopped early for benefit are often charged with being biased toward implausibly large treatment effects (e.g., Bassler et al. Reference Bassler2010).

We think that this worry is based upon a misinterpretation of sequential trials that is in turn due to shortcomings of standard frequentist procedures. It has been argued (e.g., Goodman Reference Goodman2007) that a Bayesian perspective overcomes this problem: if a trial is stopped early because of an implausibly large effect, blending its result with a (conservative) prior probability distribution naturally mitigates the conclusion. However, as a matter of research tradition and regulatory requirements—in particular, concerns about individual biases in generating prior distributions—the Bayesian framework does not provide an easy way out.

In this essay, we contend that the real issue is not the contrast between Bayesian and frequentist methodology. Rather, we are concerned about the shortcomings of unconditional inference. We have elaborated that while unconditional error probabilities may be helpful in the design of an experiment, they do not tell us what we have actually learned from the data. We have therefore defended proper conditioning—calculating error probabilities conditional on the strength of the observed evidence—as a way of curing the deficits of unconditional frequentist inference. This approach has a natural application to sequential testing and both a valid Bayesian and a valid frequentist interpretation.

As we have demonstrated in a brief example, this approach holds considerable promise for the interpretation of early stopped trials in medicine. The possibility of postdata assessments of the probability of an erroneous conclusion represents an invaluable asset for the practitioner and the decision maker. The results of a medical trial tell much more than the simple acceptance or rejection of a scientific hypothesis: they indicate where evidence is strong and where it is inconclusive, indicating the need for further research. Conditional inference, we believe, can improve the methodology of clinical trials because it allows us to take this additional information into account. In conclusion, a clearer view on issues in statistical methodology can help to better appreciate data from sequential medical trials and lead to more efficient and ethically superior decisions in medical research.