3.2.1 A conceptual motivation: Minimally sufficient covariates

As a conceptual motivation for this argument, suppose first that observed covariates that are “minimally sufficient”—essentially, they contain all and only the possibly observable information about potential outcomes (Dawid Reference Dawid1979). In this case, as we show in the Supplementary Material, treatment assignment depends on the covariates if and only if they depend on potential outcomes (Section 3.1 of the Supplementary Material and Theorem A.1 in the Supplementary Material). Then, we may validly reject as-if random based on the non-independence of treatment assignment and covariates—as in standard balance tests.

This idea is useful because it underscores the importance of choosing jointly informative covariates—and then prioritizing the most informative subset for tests. Sufficiency only guarantees the “if” direction of Theorem A.1 in the Supplementary Material. Thus, it controls false negatives: if covariates are sufficient, then when treatment is not assigned independently of potential outcomes, we should expect a well-powered balance test to fail. For the “only if” direction, we need minimum sufficiency—that is, the and only part of its definition. This controls false positives: if covariates are minimally sufficient, a failed balance test implies a failure of as-if random. This is a key motivation for the prognosis-weighted tests we discuss in Section 4. By projecting potential outcomes onto covariates before running tests, we effectively discard uninformative covariates—as in the creation of a minimally sufficient from a merely sufficient covariate set.

In applications, unfortunately, covariates usually cannot be expected to contain such complete information about the values of potential outcomes. Moreover, sufficiency is difficult or impossible to validate. To be sure, a version of this condition could occur in some settings. For example, the pre-treatment value of the response variable may sometimes equal the post-treatment value in the absence of an intervention. Then this covariate’s correlation with potential outcomes under control is 1, which also implies sufficiency. Yet, even with a lagged dependent variable, temporal trends could imply differences in values of the outcome variable in the pre- and post-treatment periods, absent an intervention. These trends could be heterogeneous for different units, which implies a correlation less than 1. The direct practical implications of the minimal sufficiency theorem are thus limited.

3.2.2 The importance of prognosis

The more prognostic covariates are, however, the closer they may get to approximating the ideal situation of complete informativeness, that is, sufficiency. The logic suggests that the power and specificity of tests will improve as covariates become more predictive of potential outcomes.

Moreover, even when a set of measured covariates is not sufficient, we may be able to improve the performance of tests by prioritizing the individual covariates that are most closely associated with potential outcomes. We discuss simulation evidence that supports this claim in Section 4.2.

This argument parallels in some ways standard arguments about the role of prognosis in achieving control over confounding variables in observational studies (VanderWeele and Shpitser Reference VanderWeele and Shpitser2011, Reference VanderWeele and Shpitser2013): the more predictive of outcomes the measured covariates are, the less likely it is that unobserved factors produce violations of as-if random. The difference is that here we are focusing on covariate characteristics that allow us effectively to test as-if random, rather than control for violations of it. Testing and estimation can be complementary tasks. However, as we suggested in Section 3.1, with balance testing we should be attentive to both false negatives and false positives: we may fail to reject as-if random because spurious covariates are balanced, but we may also falsely reject it because irrelevant covariates are imbalanced.

The challenge is therefore to collect covariate data that allow us to test identification conditions convincingly. Unfortunately, the predictiveness of covariates for outcomes has been ignored in applications of balance testing in experiments, natural experiments and discontinuity designs. As we emphasized in Section 2, researchers do not typically report measures of covariate prognosis, nor do they discuss efforts to maximize the informativeness of covariates used in balance tests. Thus, it is usually difficult to assess the extent to which these pathologies of standard balance tests apply.

In sum, we argue that researchers should attempt to gather the most informative set of covariates possible, and then prioritize the most prognostic ones in their tests. The more prognostic are covariates, the more information they give us about the likely balance of potential outcomes, and the more useful are covariate balance tests. Measures such as the prognosis $R^2$ (Figures 1 and 2) can be viewed as a continuous operationalization of informativeness. Such measures can help researchers assess the adequacy of information about potential outcomes contained in a set of measured covariates. We return to this point in connection with our simulations (Section 4.2), where we assess how error rates in tests vary as a function of levels of prognosis observed in the empirical studies in Figures 1 and 2.

4.1.1 Prognosis-weighted tests of continuity in regression-discontinuity designs

First, we extend the fitted value approach to allow for prognosis-weighted tests of continuity in RD designs. Here, the key identification condition is as follows.

Continuity implies that the limits of these functions are the same approaching from above and below the threshold. This motivates the standard approach of testing for the equality of intercepts of two regressions, one above and the other below the threshold value of the running variable.

In applications, however, researchers typically test for the continuity not of potential outcomes but of covariates. Thus, they regress each pre-treatment covariate separately on the forcing variable, above and below the RD threshold, and conduct a test for equality of the intercepts at the threshold.

Unfortunately, such tests may not be informative about the continuity of potential outcomes themselves. Just as with tests of as-if random, researchers are subject to false negatives and false positives due to irrelevant covariates (Section 3.2). Covariates may be continuous at the threshold and yet potential outcomes may not be; or vice versa. The standard approach similarly raises problems of indeterminacy and multiple testing (De la Cuesta and Imai Reference De la Cuesta and Imai2016).

Fortunately, we can readily form a prognosis-weighted test statistic that is appropriate for testing Assumption 2. Following our previous approach of using only the prognostic part of the covariates, we first regress the outcome variable on covariates on the control group side of the RD threshold. Then, we fit regressions—not of covariates, as in standard practice, but of fitted potential outcomes—on the running variable, on each side of the threshold.

The test statistic is then a prognosis-weighted difference of intercepts from regressions above and below the threshold determining treatment assignment. Conceptually, it is as if we ran separate regressions of each covariate on the running variable above and below the assignment threshold, as in standard covariate-by-covariate tests—but then combined the intercepts from the separate regressions using prognosis weighting. As with tests of as-if random, the prognosis-weighted difference of intercepts allows a test of continuity based on the covariates that are most informative about potential outcomes. We present the tests formally in Section 5 of the Supplementary Material.

We encourage prognosis-weighted assessment of continuity as the primary test of identification for RD designs. Continuity is the sine qua non for valid analysis: if there are discontinuous jumps in the potential outcome regression functions at the point of discontinuity, it will be difficult to distinguish this discontinuity from true treatment effects (Calonico, Cattaneo, and Titiunik Reference Calonico, Cattaneo and Titiunik2014; De la Cuesta and Imai Reference De la Cuesta and Imai2016).

We note, however, that reliance on continuity for identification can also raise substantial estimation challenges, which can undercut the reliability of conclusions. As Stommes, Aronow, and Sävje (Reference Stommes, Aronow and Sävje2023, Reference Benjamini and Hochberg3) rightly note, “the statistical challenges under the RD design are better compared with those we typically face in other types of observational studies.” This is in part because reliance on continuity requires estimation of intercepts in regions where data may be sparse, limiting power; and it can also require complicated bias corrections and adjustments to standard errors (Cattaneo, Idrobo, and Titiunik Reference Cattaneo, Idrobo and Titiunik2020).

Conversely, in some applications, it may also be the case that the stronger condition of as-if random (Assumption 1) holds in a bandwidth around the assignment threshold. As-if random implies the weaker assumption of continuity. Essentially, as-if random implies that the slopes of the regression functions relating potential outcomes to the running covariate are flat (Cattaneo, Frandsen, and Titiunik Reference Cattaneo, Frandsen and Titiunik2015; Dunning Reference Dunning2012, 128–133 and 158–160; Sekhon and Titiunik Reference Sekhon and Titiunik2017). In this case, effect analysis may be substantially simpler and conclusions may be more reliable. Hypothesis testing can also more readily rely on randomization inference, a key advantage (Cattaneo et al. Reference Cattaneo, Frandsen and Titiunik2015). We thus encourage researchers to report secondary prognosis-weighted tests of as-if random, where applicable. We return to this point in Section 5, where we discuss RD studies in our sample where as-if random plausibly holds.

4.1.2 Prognosis weighting based on flexible regressions and nonlinear methods

Second, we explore a range of prognosis-weighted tests based on flexible regressions and nonlinear methods, including machine-learning techniques.

The fitted-value approach leads naturally to these extensions, since different methods can be used to fit covariate-adjusted potential outcomes in the control group and then extrapolate predicted values to the treatment group. The motivation is that by allowing better prediction of potential outcomes—when, for instance, outcomes are nonlinearly related to covariates—flexible methods may improve the power or specificity of tests. The performance of tests based on machine learning and other flexible methods appears not to have been assessed in the literature on prognostic scores, however (as noted by Stuart et al. Reference Stuart, Lee and Leacy2013 and Leacy and Stuart Reference Leacy and Stuart2014).

We implement these methods—including linear regression with expanded polynomial bases and covariate interactions, regularization with lasso, and machine learning methods, including random forests and gradient boosted trees—as options in our R software (Section 4.5 of the Supplementary Material). We use simulations to compare the performance of the different methods, as discussed in Section 4.2.

4.1.3 Bootstrapped hypothesis tests and equivalence testing

Finally, we develop statistical hypothesis tests appropriate for these varied forms of prognosis weighting. We discuss a bootstrapping approach that allows naturally for the statistical dependence of the prognosis weights and control group means and that can be readily adapted for clustered or blocked designs, as well as linear and flexible nonlinear fitting procedures (Section 4.4 of the Supplementary Material).

This bootstrap can be used in connection with the different approaches to fitting covariate-adjusted potential outcomes, including the flexible regressions and machine learning methods. In addition, researchers may use prognosis-weighted equivalence tests, as a replacement for or complement to traditional hypothesis testing, to assess identification conditions. We discuss the value and limitations of equivalence tests in Section 6 of the Supplementary Material.

Overall, then, across a range of different kinds of designs, prognosis weighting allows covariate balance tests in which rejections of null hypotheses stem from differences of covariate distributions across the treatment and control groups—as in standard approaches—but in which the differences are weighted by measures of prognosis. Since tests are based on a single prognosis-weighted test statistic and thus one p-value, the approach also avoids the problems of indeterminacy and multiple comparisons that beset standard covariate-by-covariate tests.

4.2.1 Prognosis-weighted versus unweighted tests

In one set of simulations, we compare the performance of unweighted to prognosis-weighted tests, while varying the informativeness of observed covariates about potential outcomes (Sections 7.1–7.4 of the Supplementary Material).

Specifically, we compare the prognosis-weighted test that uses linear regression to fit the potential outcomes to two unweighted tests: (i) the sum of standardized covariate differences of means (call this statistic $\delta _{UW}$ , see Section 4.3.1 of the Supplementary Material) and (ii) Hotelling’s $T^2$ , another common multivariate test statistic. The latter differ from covariate-by-covariate tests in that they are based on omnibus statistics but are similar to standard approaches in that they treat all covariates “equally.”

These simulations allow us to study how joint prognosis affects the power and specificity of covariate balance tests and how prioritizing informative covariates, through prognosis weighting, affects performance. Thus, we study rejection rates of the tests when as-if random holds and when it is false, varying covariate prognosis (Sections 7.1–7.4 of the Supplementary Material). When as-if random is false, the rejection rate measures statistical power of the test; when it is true, the rate measures false positives (or Type I error, inversely related to specificity). We consider settings in which observed covariates are sufficient and those in which they are not. In the latter case, we vary prognosis so that covariates are completely uninformative about potential outcomes or only partially so.

Results: Prognosis-weighted tests avoid both false negatives and false positives.

The results from the first set of simulations illustrate that by projecting out irrelevant covariates, prognosis weighting can reduce both false negatives and false positives. In contrast, unweighted multivariate tests that do not use information on covariate prognosis sacrifice power and/or specificity. The extent to which prognosis weighting improves performance depends on the prognosis of the covariates: as it increases, prognosis-weighted tests become both more powerful and more specific.

Three further conclusions are important to note. First, when covariates are sufficient but not minimally so, unweighted tests tend to reject as-if random when it is true or fail to reject it when it is false, due to the balance or imbalance of spurious covariates (Figure A1 and Table A3 in the Supplementary Material). In contrast, prognosis-weighted tests control Type I error at standard levels, failing to reject as-if random when it is true; yet they also reject as-if random when it is false, increasingly so as the prognosis of imbalanced covariates grows. Thus, compared to unweighted tests, prognosis weighting balances power and specificity: it better detects true failures of as-if random while simultaneously limiting spurious rejections (Figure A2 in the Supplementary Material).

Second and by contrast, when covariates are insufficient and fully non-prognostic—that is, composed only of noise—weighted and unweighted tests alike are prone to substantial error. When the spurious covariates are balanced in expectation, but as-if random is false, the false negative rate for both kinds of tests approaches 1 (top-left panel of Figure A3 in the Supplementary Material).

Third and finally, however, even when covariates are not sufficient, the power of the weighted—but not the unweighted—tests grows as prognosis increases (right panels of Figure A3 and Table A4 in the Supplementary Material). The performance of the prognosis-weighted tests improves as the joint informativeness of measured covariates grows because it prioritizes those individual covariates that are most informative.

Thus, the simulations illustrate the usefulness of prognosis weighting but also offer an important caveat, consistent with discussion in Section 3: the quality of balance tests—including prognosis-weighted ones—depends on the overall joint prognosis of measured covariates. Nonetheless, even when covariates are not jointly sufficient, the prognosis-weighted test achieves power on the order of 70%–80% in these simulations when the prognosis $R^2$ lies between 0.1 and 0.2 (see Figures A3 and A4 and Table A4 in the Supplementary Material). Indeed, prognosis-weighted tests can attain 80% power at low levels of expected imbalance with prognosis $R^2$ as low as 0.125 (Figure A4 in the Supplementary Material). Since these thresholds certainly depend on the data-generating structure, to be conservative, researchers might require a prognosis $R^2$ of 0.2 to defend their tests as meaningfully prognostic.

The results therefore underscore that diagnosing and reporting covariate prognosis is critical—and so is incorporating information about the relative prognosis of different covariates into tests.

4.2.2 Varieties of prognosis weighting: Linear versus machine learning methods

In a second set of simulations, we compare different types of prognosis-weighted tests. Here, the prognosis weights are fit using both linear and flexible non-linear methods, under different degrees and types of covariate prognosis and imbalance. We modify the data-generating processes so that potential outcomes are nonlinear functions of the covariates (Section 7.5 of the Supplementary Material).

We first consider simulations with polynomials of the covariates in the process for potential outcomes (Section 7.5.1 of the Supplementary Material). Next, we consider simulations with interaction terms in the outcome process (Section 7.5.2 of the Supplementary Material). Finally, we evaluate prognosis-weighting tests with two “difficult,” highly nonlinear relationships between covariates and potential outcomes (Section 7.5.3 of the Supplementary Material). Thus, we use (a) a “tree” specification that creates regime-dependent relationships based on the sign of an interaction term. This specification allows assessment of the methods’ performance when the functional form switches discretely based on the interaction term, creating fundamentally different covariate-outcome relationships across regions of the space. We also assess results using (b) a “sine” specification that incorporates high-frequency nonlinearities. This formulation challenges linear prognosis models with oscillatory components that standard polynomial approximations may not, a priori , seem to capture well.

In these final simulations, we compare the performance of (1) unweighted tests; (2) prognosis-weighted tests based on simple linear regressions; (3) prognosis-weighted tests based on regressions with expanded polynomial bases or interactions; and (4) prognosis-weighted tests based on two machine learning methods—(i) tuned random forests and (ii) tuned gradient boosted trees—as well as lasso with expanded bases (e.g., polynomial and covariate interactions). We also assess the performance of the best-fitting method (the one that produces the best fit to control potential outcomes) that is automatically selected by the software in each run of the simulations. We manipulate the target $R^2$ of the prognosis regressions to assess how test performance varies as covariate informativeness changes.

Results: (Saturated) linear models perform well.

The results of the second set of simulations suggest several useful insights and conclusions.

1. 1. Prognosis-weighted tests based on “saturated” linear regressions—that is, those with expanded polynomial bases or covariate interactions—can sometimes offer improvements in power over simple linear methods, as can flexible nonlinear methods (Figures A5 and A7–A10 in the Supplementary Material).

2. 2. In many simulations, however, the differences are minor. The extent of the improvement depends not just on nonlinearities in the relationship between covariates and potential outcomes but also on the nature of imbalances. When “main” terms are balanced in expectation but nonlinear (e.g., polynomial or interaction) terms are imbalanced, methods that allow nonlinear fits can offer improved power. Yet, when there is also expected imbalance on main terms, the expected performance of the tests is often indistinguishable (Figures A7–A10 in the Supplementary Material).

3. 3. In these simulations, automatic selection of the method that produces the best fit of potential outcomes given covariates in the control group need not lead to the most powerful test. Even with the complex, “difficult” data-generating processes, the expanded linear model with polynomial bases and covariate interactions has the greatest power (Figures A11 and A12 in the Supplementary Material).

Overall, the best performer is often the test based on the expanded linear model with polynomials and interactions: it controls Type I error at similar levels as other prognosis-weighted tests when as-if random is true, but it rejects as-if random with the highest probability when it is false. We therefore recommend simple tests with weights based on linear regression, particular with expanded bases where possible, due to their i) estimation stability and sometimes greater power and ii) the ready interpretability of weights in terms of the relative prognosis of the different covariates.

Our simulations also draw attention to the importance of patterns of linear versus nonlinear prognostic imbalances, which to our knowledge has not received attention in work on covariate balance testing. Standard approaches typically test only for main differences (e.g., differences of means), though some researchers do consider differences in distributions using, for example, K-S tests. Analysts should consider the substantive domain under consideration and be attentive to the possibility of nonlinear imbalances on prognostic variables. We return to further recommendations in Section 6.

5.1.1 Prognostic imbalance

In one set of studies, we find balance on noise covariates but greater imbalance on prognostic covariates (see Figure 2, Table 1 and Section 1 of the Supplementary Material). In such contexts, an analysis that does not consider covariate prognosis may support the existence of a natural experiment or valid discontinuity design. Yet, prognosis weighting may suggest greater cause for concern.

Consider the excellent study by Samii (Reference Samii2013), who assesses the consequences for ethnic tolerance of serving in an integrated military in Burundi in the aftermath of a brutal, ethnically charged civil war. Using for identification a discontinuity design based on a military retirement age, the article shows that serving in an ethnically integrated military decreased prejudicial behavior, though not necessarily the salience of ethnicity. To assess identification conditions, Samii (Reference Samii2013, 569–570) uses balance tests with pre-treatment covariates that have “strong potential to confound were they to exhibit discontinuities near the cutoff.” These include noncommissioned officer status; years in the military; years of prewar education; wartime death rates per military unit; and family wartime mortality. Informally, Samii describes why each of these pre-treatment covariate might be linked to ethnic tolerance (the outcome variable). The prognosis $R^2$ is $0.244$ (Table 1), indicating that measured covariates are jointly fairly predictive of the outcome. However, as with other standard balance tests in the literature, there is no formal measurement of the informativeness of individual covariates.

In fact, it turns out that prognosis varies substantially across the individual covariates included in the balance tests. In the standardized prognosis regression of prejudice on covariates, the coefficient is just 0.007 on years of prewar education, while the standardized coefficient on years in the military is $-0.507$ (see our Figure 2 and Section 1 of the Supplementary Material). Per Figure 2, the latter, prognostic covariate is also the only one for which as-if random would be rejected in a covariate-by-covariate difference of means test (p-value 0.028). And because the most prognostic covariate is imbalanced, while less prognostic covariates are statistically balanced, the prognostic-weighted omnibus test rejects as-if random (p-value 0.028), while an unweighted test does not (Table 1). Fortunately, a prognosis-weighted test of continuity does not reject (p-value 0.298).

Thomas (Reference Thomas2018) and Blattman (Reference Blattman2009) provide similar examples of imbalance on prognostic variables (Figure 2). As shown in Table 1, prognosis-weighted tests therefore call into question as-if random in Blattman (p-value 0.030) and continuity in Thomas (p-value 0.000). In the case of Blattman’s study of the effects of child soldiering in Uganda, the author spotted the imbalance on the prognostic age variable (see Figure 2) and controlled for this single covariate in treatment effect regressions. Blattman (Reference Blattman2009, 232) argued that in the Ugandan Lord’s Resistance Army, “abduction parties were under instruction to release only young children and older adults” but other indiscriminately kidnapped “adolescent and young adult males,” leading to the imbalance on age across abducted (treatment group) and non-abducted (control group) youths.

Formal diagnosis of prognosis—and graphical assessments like those in Figure 2—can generally aid the identification of such prognostic, imbalanced covariates. In some studies, like Blattman’s, qualitative understanding of the treatment assignment process may support an argument that treatment assignment is independent of potential outcomes—but only conditional on an imbalanced, prognostic covariate. In other studies, identification of such covariates may call into question the assumption of a valid natural experiment or discontinuity design.

5.1.2 Prognostic balance

In another set of studies, we find the opposite situation: noise covariates are imbalanced, but there is statistical balance on prognostic variables. In these cases, a naive unweighted test may imply a failed design, but a prognosis-weighted test instead supports key identification conditions.

Novaes (Reference Novaes2018), for example, uses a close-election design to assess whether barely winning an election causes mayors (who often play the role of partisan brokers) to switch parties at lower rates than near losers.Footnote ¹³ The author conducts balance tests on 27 covariates (see our Figure 2 and Section 1 of the Supplementary Material). Our analysis suggests statistical imbalance on several of these variables, but these covariates are all essentially non-prognostic. In contrast, more informative variables are tightly balanced. The p-values from prognosis-weighted tests therefore do not reject as-if random or continuity.

Kim (Reference Kim2019) is another example of a study with substantial imbalance on non-prognostic variables. The author exploits a discontinuity design based on a population threshold that assigns direct democracy to municipalities in Sweden to study effects on the political inclusion of newly enfranchised women. The author tests for balance on nine covariates. Four are negligibly prognostic, with standardized coefficients in the regression of women’s turnout (the primary outcome) on covariates that are near zero; two of these noise variables are significantly imbalanced. By contrast, lagged turnout is highly prognostic, with a coefficient of 0.590 yet also statistically balanced (p-value 0.386). Finally, three variables—the pre-treatment tax base, share of agriculture in the economy and land area—are both moderately prognostic and imbalanced (with respective prognosis coefficients and p-values of 0.157 and 0.000; $-0.227$ and 0.028; and $-0.156$ and 0.003). By combining information on imbalance and prognosis across covariates, an informativeness-weighted test helps to sort out these contrasting signals: an unweighted test of as-if random rejects (p-value < 0.000) while prognostic-weighted tests reject neither as-if random nor continuity. This is also a highly prognostic set of covariates overall (prognosis $R^2$ of 0.541, per Table 1).

Boas and Hidalgo (Reference Boas and Hidalgo2011) provide a final example in which there are imbalances in some placebo tests—but only with non-prognostic covariates. In this study, the authors study the effect of political incumbency on control of the media, using a close-elections design.Footnote ¹⁴ To assess as-if random and continuity of potential outcomes, Boas and Hidalgo (Reference Boas and Hidalgo2011, 876) present covariate-by-covariate tests for 18 variables. Difference-of-means tests reject the null for three (our Figure 2): an indicator for the state of Minas Gerais (“uf_mg”), the vote share of president Lula of the Worker’s Party in 1998 and the log of the electorate size (“votes”). However, these three variables have small standardized prognosis coefficients ( $\! -0.045$ , $-0.001$ and $-0.038$ , respectively) in the regression of politician’s control of local media on covariates (see Section 1 of the Supplementary Material). Other covariates, including the most prognostic covariates, such as latitude (prognosis coefficient $-0.167$ ) or time since application for a radio license (prognosis coefficient 0.323), are balanced. Boas and Hidalgo (Reference Boas and Hidalgo2011: 875) argue that the pattern of imbalance across the 18 covariates “is approximately what one would expect if incumbency status had been randomly assigned.” However, rules of thumb for the number of covariates that should be imbalanced in expectation do not readily apply, given dependence across covariates.

Pooling information across the different covariates to construct a single omnibus, prognosis-weighted test allows a more conclusive test than covariate-by-covariate comparisons. Thus, with the Boas and Hidalgo (Reference Boas and Hidalgo2011) data, prognosis-weighted tests reject neither as-if random nor continuity (our Table 1). The tests properly discount imbalances on non-prognostic variables.

5.1.3 Mixed prognostic balance and imbalance

In a final set of studies, there is imbalance on some prognostic and noise covariates, and balance on others. In these cases, the prognosis-weighted test is crucial for sorting out the relative weight of the balance and imbalance among prognostic covariates and for providing a single test statistic that can summarize the evidence for or against the identification conditions.

Fouirnaies and Hall (Reference Fouirnaies and Hall2014) provide one example. Per Figure 2, a large number of both prognostic and noise variables are statistically balanced, as indicated by the gray shading. However, several prognostic covariates are also statistically imbalanced. While unweighted tests of as-if random and continuity reject the nulls, prognosis-weighted tests do not.

In sum, in standard covariate-by-covariate tests, analysis of different variables would lead to different conclusions about identification conditions. And there is no way in typical practice to assess how meaningful is each test, because measures of prognosis are not provided. Prognosis-weighted tests instead combine information across covariates and properly base conclusions on the most informative of the variables.

5.2.1 The three problems of balance testing in close elections

Consider now, however, how the three problems of covariate balance testing that we have discussed impact the conclusions that can be drawn from this controversy:

(1) First, covariate prognosis is not measured or reported.

Caughey and Sekhon (Reference Caughey and Sekhon2011) and Eggers et al. (Reference Eggers, Fowler, Hainmueller, Hall, James and Snyder2015) emphasize the importance of lagged party incumbency, and variables correlated with it, as key covariates for balance tests. However, they do not empirically assess each covariate’s prognosis. Nor do they report measures of the overall associations of covariates used in balance tests with outcomes.

This is important because, as we show next, covariate prognosis in fact varies substantially both within and across these studies—and not always in the way one might expect a priori . This variation has important implications for interpreting the strength of the balance tests.

(2) Second, individual covariates are not equally informative—yet this variation is not incorporated in tests.

As shown in Figure 2, both the (a) imbalance and (b) prognosis of individual covariates varies substantially in the Caughey and Sekhon (Reference Caughey and Sekhon2011) data on the U.S. House. Several covariates are imbalanced, but many are not. And while covariates as a whole appear jointly informative, with a prognosis $R^2$ of 0.487 (Figure 1), many individual covariates are not prognostic—with standardized regression coefficients near zero.

Unfortunately, there is no formal procedure that takes into account the unequal informativeness of different covariates.Footnote ¹⁷ And covariate-by-covariate tests offer no ready way to reconcile the contrasting results: some tests reject and others do not, so one cannot readily infer the overall strength of the evidence for or against as-if random.

Prognosis-weighted omnibus tests address these problems. As shown in Table 1, the prognosis-weighted test rejects as-if random in Caughey and Sekhon’s (Reference Caughey and Sekhon2011) data, but an unweighted test does not. This is likely because, as Figure 2 suggests, there are large imbalances in several prognostic variables. Consistent with De la Cuesta and Imai (Reference De la Cuesta and Imai2016), the prognosis-weighted test does not reject the weaker assumption of continuity, even though the test is based on the most prognostic covariates. In each case, the omnibus statistic provides a rejection rule based on formal accounting for the varied prognosis of different covariates, while also addressing multiple testing concerns.

We note also that conclusions using Caughey and Sekhon’s data are sensitive to the treatment of missing data. Our software implementation, by basing the test of as-if random on the vector product of prognosis coefficients and differences of means as in Equation (6) in Section 4.2 of the Supplementary Material, allows us to use the full set of data available for each difference of means, as in covariate-by-covariate tests. However, listwise deletion leads to meaningfully different results, in particular an insignificant test statistic for as-if random. See discussion of the treatment of missing data in Section 8.2 of the Supplementary Material.

(3) Third and perhaps most importantly, weak covariate prognosis in existing cross-national tests does not allow for informative tests of identification conditions.

Building on the importance of lagged party incumbency in the U.S. House, Eggers et al. (Reference Eggers, Fowler, Hainmueller, Hall, James and Snyder2015) in fact test for balance only on this covariate.Footnote ¹⁸ This approach is entirely understandable as well as practical: lagged party incumbency is readily available across elections and countries, whereas the availability of other pre-treatment covariates may vary by context.

Yet, the prognostic value of lagged party incumbency in fact varies across countries and types of elections—and in close elections, it is not in fact prognostic on average. Thus, the correlation between the vote share of the incumbent party at time $t - 1$ and time t is $0.79$ across all countries and election types but varies from a low of $0.09$ in Brazilian mayoral elections to a high of $0.91$ in the German Bundestag (full data set); in close elections (defined by a bandwidth of 0.5, that is, the margin between the winning and runner-up party is less than 1 percentage point), it varies from a high of $0.32$ in New Zealand’s post-war parliament to a low of $-0.16$ is the Canadian House of Commons (1867–1911) (see Tables A1 and A2 in the Supplementary Material).

Most concerningly, the average prognosis is essentially zero across all close elections (Figure 1 and Table A2 in the Supplementary Material).Footnote ¹⁹ For reasons we discussed in Sections 3 and 4, this weak covariate prognosis implies that the cross-national balance tests are uninformative about the balance of potential outcomes in close elections.

5.2.2 Close elections: Methodological and substantive conclusions

Measurement of prognosis and implementation of prognosis weighting helps to synthesize and explain contrasting previous results in the study of close elections.

Yet, weak prognosis in a cross-national dataset of close elections runs the risk of misleading general conclusions. With the Eggers et al. (Reference Eggers, Fowler, Hainmueller, Hall, James and Snyder2015) data, prognosis-weighted tests reject neither as-if random nor continuity (Table 1). Yet, as we have shown, balance tests using irrelevant, non-prognostic covariates cannot validly support or falsify key identification conditions. In particular, they are prone to false negatives.

The failure to measure and account for prognosis—common to all the studies—therefore considerably weakens the conclusions that can be drawn. A finding of statistical balance on a single non-prognostic covariate, as in Eggers et al. (Reference Eggers, Fowler, Hainmueller, Hall, James and Snyder2015), cannot compellingly support the general as-if randomness of close elections cross-nationally.

More generally, the results underscore the critical importance of measuring prognosis formally and incorporating it into analyses. Prognosis is an empirical question. A priori , it appears natural that lagged party incumbency would be highly correlated with future incumbency. In fact, the correlation is negligible across different countries and types of elections.Footnote ²⁰

These findings imply that the methodological debate about close elections is far from settled. We do not view our results as yet confirming or contradicting the identification conditions in general. They instead suggest the need to leverage a richer set of prognostic covariates for cross-national tests.