Terminology

Throughout the recommendations and the accompanying report, four terms are used to describe the “other variables” that are neither independent nor dependent measures, but are the subject of the directive described above: pre-treatment measures, covariates, demographics, and control variables. Whether or not the authors of the Standards Report had precise meanings in mind for each of these, both the report and our discussion of it will benefit from making these definitions explicit.

For purposes of our discussion, the term “pre-treatment measure” is the most self-evident, and we will assume it refers to any measure in the data set that is assessed before the treatment occurs and thus could not have been affected by the treatment. The term “covariate,” on the other hand, is typically used for that subset of pre-treatment measures that are incorporated in the statistical model used to test experimental hypotheses. It is not clear from the report whether “covariate” is meant in this manner or is meant as a synonym for “pre-treatment measure.” This confusion exists throughout political science (see, e.g., Arceneaux and Kolodny Reference Arceneaux and Kolodny2009: 760).

Importantly, this distinction becomes blurred if the model is not pre-specified; as in the Standards Committee's recommendations, we endorse the pre-specification of models and will use the term “covariate” only for measures that researchers had planned to include in the model in advance. As outlined in many sources (e.g., Franklin Reference Franklin1991), the purpose of a covariate is to predict variance in the dependent variable that is clearly not attributable to the treatment. For this reason a covariate must be a pretreatment variable, although not all pretreatment variables must be included as covariates. Covariates need to be selected in advance based on what one knows about the major predictors of the dependent variable in the experiment. Their purpose is to increase the efficiency of the analysis model by eliminating nuisance variance.

The term “demographic” is usually reserved for that subset of pre-treatment measures which describe characteristics of the sort found on census data: age, education, race, income, gender and the like. If they are to be used in the analysis of an experiment, they should be pre-treatment measures as well. However, there is no reason to include such measures as covariates unless one has reason to believe they are strong predictors of the dependent variable. For most outcomes in political science experiments, demographics are only weakly predictive at best. The purpose of a covariate is to increase the power of the experiment by reducing variance. As argued in Mutz and Pemantle (Reference Mutz and Pemantle2011), the gain from adjusting by a weak predictor of the dependent variable does not overcome the cost in transparency and robustness. Adding an extremely weak predictor can even reduce power due to the loss of a degree of freedom.

There are other legitimate reasons to include a measure as a covariate. One is a suspected interaction. At times, demographics are included as hypothesized moderators of the treatment effect. For example, if one has reason to believe that the treatment will be greater among the poorly educated, then the moderator and its interaction with treatment are included in the analysis model.

The fourth term, “control variable,” is borrowed from the observational data analysis paradigm. Control variables are variables that are included in statistical models in order to eliminate potentially spurious relationships between the independent and dependent variables that might otherwise be thought to be causal. Given that potentially spurious relationships between the independent and dependent variables are eliminated by randomization in experimental designs, we find the frequent use of this term out of place in experimental research.

Observational studies often use demographics as standard control variables. However, there is a tendency for political scientists to use the terms demographics and control variables interchangeably, regardless of whether demographics are a likely source of confounding or spurious association. The term “control variable” is rampant in published experiments in political science (for just a few examples, see Hutchings et al. Reference Hutchings, Valentino, Philpot and White2004; Ladd Reference Ladd2010; Michelbach et al. Reference Michelbach, Scott, Matland and Bornstein2003: 29; Valentino et al. Reference Valentino, Hutchings and White2002), even when there is no evidence of differential attrition or any need to “control for” other variables.

Notably, covariates serve a very different purpose from control variables and should be selected based on different criteria. Covariates are included in the statistical model for an experiment because of their anticipated relationship with the dependent variable, to increase model efficiency. Control variables are included in observational analyses because of their anticipated relationship with the independent variables, to prevent spurious relationships between the independent and dependent variables. Choosing covariates solely due to their correlation with the independent variable is problematic in experiments, as we discuss at greater length below.

Because these four terms are used more or less interchangeably in the Standards Report, the recommendation is unclear as to which and how many variables the committee would like to see broken down by condition in a table. Is it the ones included in the original (pre-specified) model? This makes little sense because those variables are already included in the model. At other times it appears they are concerned specifically with those variables not included in the model, such as demographics or other available pre-treatment measures which the experimenter had no reason to include. But if these variables are not central to the outcome of interest, it is unclear why balance on those variables is important.

As discussed further below, and as illustrated by many examples from political science journals, the recommendation in favor of displaying all pretreatment means and standard errors by experimental condition is more likely to promote confusion than clarity. Indeed the number of pretreatment variables used in balance tests has reached numbers as high as fifteen or more, and many more pre-treatment measures are often available including variables such as household internet access, party identification, age, education, race, gender, response option order, household size, household income, marital status, urbanicity, home ownership, employment status, if the respondent was head of the household, children in household and region, to cite one example (see, e.g., Malhotra and Popp Reference Malhotra and Popp2012). As in this example, it is unclear why a particular set of pretreatment means is selected for comparison and how and why one should compare them.

The Role of Randomization

Fisher (Reference Fisher1935) introduced randomization as a way to make treatment and control groups stochastically equal, meaning they are equal on average. If a researcher wants to make experimental groups as equal as possible on a specific set of dimensions, he or she would not use simple random assignment. Random assignment produces random deviations of relative size inversely proportional to the square root of the sample size, whereas a matched block design produces almost no deviation at all. In other words, randomization is not meant as a mindless way to implement blocking on known variables. The benefit of randomization is that it distributes all unknown quantities, as well as the known quantities, in a (stochastically) equal manner. This is where random assignment derives its title as the “gold standard” for causal inference: because unknown factors as well as known ones are stochastically equalized, possible confounding is ruled out by design.

The flip side of the bargain is that confounding is ruled out only stochastically. The precise inference that can be drawn is that the observed data must be caused by the treatment unless an event occurred which has probability less than p, where p is usually equal to 0.05. Historically, this is where the confusion begins to seep in: what is the nature of the “exceptional” event of probability less than 0.05, where a possible Type I error occurs?

The strong (but mistaken) intuition of many researchers is that one should be able to examine the data and see whether the randomization was unlucky. If it were possible to do this, analyses of experimental data would look very different: the rooting out of unlucky draws would be built into the analysis, in a manner specified in advance, and accompanied by precise confidence statements. There are, in fact, rejection sampling schemes that accomplish this. The downside of rejection sampling schemes is that one cannot treat the randomization that one chooses to keep as if it were the first and only one; instead, complex statistical adjustments must be made (Morgan and Rubin Reference Morgan and Rubin2012). Notably, what is accomplished by doing so is a reduction in variance and a consequent increase in the statistical power of the experiment, not a reduction in the probability of a Type I error. The important point here is that balance is not necessary for valid inference in experiments. As Senn (Reference Senn2013: 1442) explains, “It is not necessary for groups to be balanced. In fact, the probability calculation applied to a clinical trial automatically makes an allowance for the fact that groups will almost certainly be unbalanced, and if one knew that they were balanced, then the calculation that is usually performed would not be correct” (emphasis in original).

With experimental data, judicious choice of covariates can greatly increase the power of the analysis, but this is a separate issue from confidence in the result. If one wants more confidence, he or she should use a smaller p-value. If a researcher uses a p-value of 0.05, then he or she will have to put up with a one in twenty chance that the result is mistaken. No amount of balance testing or digging into the data will eliminate or lower this uncertainty.

The Role of Covariates

Once a covariate is included in an analysis, the estimate of the treatment effect will be adjusted for this variable. Thus, there are as many potential estimates of treatment effects as there are sets of covariates that could be selected from among all available pre-treatment measures. Normatively, the model (including the precise set of covariates) is selected on the basis of theoretical considerations before the analysis is run, in which case there is one actual estimate of treatment effect. If the model is not pre-specified, the confidence statement surrounding the estimate is invalidated. For this reason, the second point under Section E in the Report––which asks researchers to be explicit about pre-specification of the model––is essential.

The most important observation to make about the many potential estimates of treatment effects is that the probability of an error is equally likely with any of the potential estimates. This is not to say that it does not matter which model is chosen. A better choice will reduce variance, increase efficiency, and lead to smaller confidence intervals. But it will not reduce the chance of Type I error. Likewise, the inclusion of other variables in the model will not increase robustness. Instead, the inclusion of covariates requires meeting additional assumptions that are not otherwise required. In particular, the relationship between the dependent variable and each of the covariates must be linear, the regression coefficient for each covariate should be the same within each treatment condition, and the treatments cannot affect the covariates, which is why they must be assessed pretreatment.

Including a large number of covariates in an analysis simply because they are demographics, or because they are available in the pretest is clearly inadvisable. With experimental data, “Rudimentary data analysis replaces scores of regressions, freeing the researcher from the scientific and moral hazards of data mining” (Green and Gerber Reference Green, Gerber, Milner and Katznelson2002: 810–11). But the problem goes beyond the risks of data mining. Many experimental studies suggest that findings are more “robust” if they survive models that include additional covariates (e.g., Harbridge et al. Reference Harbridge, Malhotra and Harrison2014: 333; Sances Reference Sances2012: 9). In reality, adding covariates simply because they are available reduces the robustness of the model (introducing an assumption of independent linear effects that do not interact with treatments), reduces transparency, and is unlikely to add any power.

What Purpose can Randomization Checks Serve?

It is crucial to any experiment that its random assignment be correctly accomplished. How might one detect errors in this regard? The first line of defense is a sufficiently detailed description of the randomization mechanism. Was it the RAND() function in Excel, a physical device such as a die, spinner, jar of balls, deck of cards, was it a printed random number table, or some other device? Was it pre-generated or generated as needed? If it was a blocked design, how was the blocking implemented? The randomization process is mentioned in Section C, and while we endorse this recommendation, it does not go far enough. The brief text in this recommendation and its sub-paragraph on hierarchical sampling do not cover enough bases to effectively prevent randomization errors. Because randomization is a process rather than an outcome, we think a more thorough description of the process is in order as recommended in Section 8a of the CONSORT (2010) checklist (Moher et al. Reference Moher, Hopewell, Schulz, Montori, Gøtzsche, Devereaux, Elbourne, Egger and Altman2010; Schulz et al. Reference Schulz, Altman and Moher2010).

The Report somewhat mischaracterizes our argument in saying we agree “that formal tests or their rough ocular equivalents may be useful to detect errors in the implementation of randomization.” The important points are (1) that such tests are not necessary in order to detect randomization problems; and (2) that they are not, in and of themselves, sufficient evidence of a randomization problem. Due to the rarity of randomization failure, we believe that the impulse to check for balance is probably spurred by something other than skepticism over the functionality of the random assignment process.

The terms “balance test” and “randomization check” are typically used interchangeably to indicate a table of the distribution of pre-treatment measures across treatment groups, often along with a statistical statement concerning the likelihood of the extremity of the distribution having been produced by chance alone. Such a statistic can be reported for each variable or a joint test can be reported as a single omnibus statistic for the joint distribution of all test variables. If one tests for differences in each variable individually, a large number of such tests obviously increases the chance of finding significance. If one uses a joint test, it will take into account the number of variables, but it will still matter a great deal which particular variables are chosen for inclusion in the omnibus test. A standard example of including such a check reads, “Randomization check shows that demographics and political predispositions do not jointly predict treatment assignment (X ² _[24] = 18.48, p = 0.779)” (Arceneaux Reference Arceneaux2012: 275).

The report does not provide guidance as to what these balance variables should be, except to refer to them as “pretreatment” or “demographic” variables. In examples such as the one above, the exact variables are not mentioned. Given the many different outcome variables that are examined in political science experiments, it is unclear why demographics, in particular, are deemed particularly important when other variables may be more pertinent to the outcome under study.

To reiterate, the Standards Committee calls for tables of unspecified pre-treatment measures across treatment groups “to help detect errors” in randomization. Most importantly, it is not clear how such tables accomplish this task. The distribution of pre-treatment measures across conditions provides evidence of errors only if a faulty randomization device was used; in other words, we are testing the null hypothesis, which is the assumption that the randomization mechanism worked. If we reject the null hypothesis and conclude that the randomization device was faulty, then the study can no longer be considered an experiment nor be published as one. In practice, however, when imbalance is identified, this is seldom the course of action that is taken as we describe further below.

Other Uses of Balance Testing

The Standards Report (Gerber et al. Reference Gerber, Arceneaux, Boudreau, Dowling, Hillygus, Palfrey, Biggers and Hendry2014: 92) suggests that there are additional reasons to require balance tests.

It is worth considering these additional rationales individually. Mistakes in coding variables do indeed occur with regularity, but why should they be more likely to occur with randomization variables than with the coding of other variables? Failure to account for blocking is already addressed elsewhere in the requirements where authors are required to describe whether and how their sample was blocked, as well as how they accomplished the random assignment process. Likewise, the description already must include mention of the unit of analysis that was randomized, so if the authors then analyze the data at a different unit of analysis, this will be evident.

The one scenario in which balance testing does make sense is when experimental studies take place over time, thus raising the possibility of differential sample attrition due to treatment. Sample attrition does not indicate a broken randomization mechanism, and it is already covered in the CONSORT diagram. Assuming a control condition is present, it sets an expectation for acceptable attrition levels. And if there is differential attrition across experimental conditions, then it makes perfect sense to conduct balance tests on pretreatment variables among post-test participants. If the post-attrition distribution of pre-treatment measures across treatment groups is distinguishable from the random pre-treatment distribution, then the experiment is clearly confounded.

For various reasons, we believe that error detection and differential attrition are not the primary reasons that balance testing is popular. Instead, as described above, we believe part of its appeal stems from researchers’ strong intuition that they can unearth the unlucky draw. Further, the Report of the Standards Committee explicitly says that error detection is not the only reason for doing randomization checks. As stated on page 5 of the standards document,

There are several problems with this statement, which we address in order of appearance. First, the phrase “statistically significant or not” is meaningless in the context of adjustment. The only thing one can test statistically is the null hypothesis, which is the assumption that the randomization mechanism worked. If one is estimating treatment effects, then one is already assuming that the mechanism worked, so there is no question of significance. This point has been made many times in the literature in political science (Imai et al. Reference Imai, King and Stuart2008) as well as in other disciplines (Boers Reference Boers2011; Senn Reference Senn1994).

Further, it is important to think through the consequences of this requirement for reviewers as well as authors. This statement implies that it is acceptable and even appropriate for a reviewer to (either subjectively or based on a prescribed test) perceive an imbalance in the table, assert that it is a variable that might be related to the dependent variable, and therefore insist that something be done to address the situation.

Regardless of whether there is a statistical test, what happens next? Is it incumbent upon the author to somehow “prove” that randomization was done appropriately? How can this possibly be accomplished? And if we conclude from an author's inability to produce such evidence that randomization was not done correctly, then what? If balance tests/tables are truly being used to ascertain whether random assignment was done correctly, then the only logical response to concluding that it was not done correctly is to throw out the study altogether, or possibly analyze it as purely observational data.

Random assignment was either done correctly or it was not; there is no middle ground. This does not appear to be widely understood. As an experimental study in the American Journal of Political Science explained, “To test the robustness of our randomization scheme, we tested for any differences among the other observables on which we did not block. . . .” (Butler and Broockman, Reference Butler and Broockman2011: 467). Results can certainly be more or less robust, but random assignment is either done correctly or it is not; there are no varying degrees of randomization.

Fixing a Broken Mechanism?

The assumption that one can “fix” a broken random assignment by the virtue of adding a covariate is commonplace throughout our top journals. For example, in a Public Opinion Quarterly article we are told that, “Partisanship is included in the analysis because of imbalances in the distribution of this variable across the conditions.” (Hutchings et al. Reference Hutchings, Valentino, Philpot and White2004: 521). Likewise, an article in the American Journal of Political Science assures us that “Every relevant variable is randomly distributed across conditions with the exception of education in Study 1. When we included education in our basic models, the results were substantially the same as those we report in the text” (Berinsky and Mendelberg Reference Berinsky and Mendelberg2005: 862).

There is no logic to including a “control” variable to correct for lack of true random assignment on just one or a few characteristics, a point that does not seem to be widely understood by political scientists. For example, Barabas and colleagues (Reference Barabas, Pollock and Wachtel2011: 21) assert that “we observed non-random treatment assignment (i.e., p < 0.10 differences between the treatment and control groups on partisanship, age, education, and race) which necessitates the use of statistical controls later in the paper.” Of course, by “non-random,” the authors probably did not mean that their randomization mechanism was faulty; therefore, they continue to treat the study as an experiment, not as an observational study resulting from a failed randomization mechanism.

Adding a variable to the statistical model for an experimental analysis because it failed a randomization check is an inferior model choice (see Imai et al. Reference Imai, King and Stuart2008; Mutz and Pemantle Reference Mutz and Pemantle2011). It is a misnomer to say that it “controls” for the lack of balance and there is no defensible reason to accept this as a “fix” for a broken random assignment mechanism, if that is indeed what we are looking for by providing such tables.

We suspect that instead of a failure to randomize, what many authors and reviewers actually have in mind is the unlucky chance that experimental conditions are unbalanced on some variable of potential interest. Of course, if it is a strong predictor of the dependent variable, a pre-treatment measure of that variable should have been used for blocking purposes or as a planned covariate in the model to increase model efficiency regardless of balance; this is the only appropriate purpose of covariates.

But more importantly, using a third variable to try to “correct” a model for imbalance ignores the fact that the alpha value used to test experimental hypotheses already takes into account that cells will be uneven on some characteristics due to chance. The committee report states that “we. . .do not counsel any particular modeling response to the table of covariate means that we ask the researcher to provide.” However, given that the only example provided of what one might do with this information is to adjust the treatment effects by including covariates, this seems somewhat misleading. As they elaborate, “Our guiding principle is to provide the reader and the reviewer the information they need to evaluate what the researcher has done and to update their beliefs about the treatment effects accordingly.” But exactly how should the reviewer or reader “update” his or her beliefs about the effects of treatment based on such a table?

If such a table truly serves as evidence (or lack thereof) that proper random assignment was accomplished, then such tables will greatly affect a study's chances of publication. By requiring such information, an editor automatically suggests to readers and authors that it is both informative and relevant because it is worth valuable journal space. If it is to be required, it seems incumbent upon the editors to inform authors as to how they will interpret such information. Will they conclude that random assignment was done incorrectly on this basis and thus automatically reject it from an experimental journal? Will they compel authors to provide evidence that random assignment was done correctly, and if so, what would be considered compelling evidence?

And are they required to present evidence of balance even on demographic variables that bear no relation to the outcome variables simply because they are widely used as control variables in observational analyses or on all pre-treatment measures because they happen to be in the study? We maintain that such practices have no scientific or statistical basis and serve only to promote further methodological confusion.

The report does not distinguish between pre-treatment measures available to the researcher but not chosen for inclusion in the model, and those chosen in advance for inclusion in the model. If, as is common with survey data, there are dozens of available pre-treatment measures, then is balance supposed to be reported for all of them? If so, why? As Thye (Reference Thye2007: 70) has noted, “Not all the factors that make experimental groups different from control groups are relevant to the dependent variable; therefore, not all factors must necessarily be equated. Many differences simply do not matter.” To advocate such a practice is to encourage mindless statistical models, which should not be promoted by any journal. It encourages a misunderstanding of what randomization does and does not accomplish. It also promotes further confusion in the field as to the distinction between experimental and observational analysis.

To reiterate, pre-treatment variables known from previous research to be highly predictive of the outcome should always be included in the model as covariates. To fail to do so is to reduce power so that only the strongest effects will be seen. It should not take a failed balance test to reveal such a variable, and the fact that a balance test fails for a particular variable makes it no more likely that this variable is in fact related to the dependent variable.

Finally, the question of whether a variable is adequately “controlled for” is a non sequitur in experimental research. Control variables exist for good reasons in observational studies (potential spuriousness), but treating a covariate as a control variable in the experimental setting makes no sense. Nonetheless, this practice is currently widespread. Including a variable in the statistical model because it has been found to be out of balance is also precisely the wrong reason to include a variable and should not increase our confidence in findings.

Taken at face value, the Standards Report promotes randomization checks strictly as a way of “evaluating the integrity of the randomization process” (Gerber et al., Reference Gerber, Arceneaux, Boudreau, Dowling, Hillygus, Palfrey, Biggers and Hendry2014: 92). They suggest that imbalances due to chance are distinguishable from imbalances due to faulty random assignment mechanisms. But if a fear of faulty mechanisms is the real reason for doing them, then the typical response (adding new variables to the model) is completely inadequate; if a randomization mechanism fails, the researcher needs to start over from scratch.

To summarize, failed balance tests cast doubt on experimental results; as a result, one seldom if ever finds a published experimental study with a “failed” randomization; instead they are routinely dubbed “successful” (Malhotra and Popp Reference Malhotra and Popp2012: 39) and even “highly successful” (Butler and Broockman Reference Butler and Broockman2011: 467). Moreover, if an author admits a “failed” balance test, it is strictly on one or two “unbalanced” variables that are, as a result of the balance test, included as covariates. This practice does not fix the problem if the randomization mechanism was, indeed, broken.

The real harm in this practice is the possibility of a Type II error when a skeptical referee or editor causes a correct finding to be suppressed or uses it as a reason to alter the statistical model to include more covariates in order to suggest that they have “adjusted” for a bad or unlucky randomization. This practice implies that the reader's ad hoc estimates of treatment effects and confidence might be superior to the researcher's stated estimates and confidence. As mentioned above, changing the model voids confidence statements.

At times, this kind of misunderstanding of randomization is made explicit, even within our top journals. For example, as an article in the Journal of Politics explains,

A test of mean differences across five demographic variables is not what gave this study internal validity; proper use of random assignment did. Moreover, controlling for these variables in a regression equation or using them as covariates would not have fixed a failed randomization, nor would it have increased the power of the study, unless those variables were chosen in advance for the known strength of their relationships with the dependent variable rather than for their relationships with the independent variable, as is suggested above.

Many researchers do not appear to understand that the alpha value used in statistical tests already incorporates the probability of the unlucky draw. As Hyde (Reference Hyde2010: 517) suggests in another experimental study,

Here again, a randomization check is being used to try to uncover the unlucky draw in order to increase confidence in the findings as opposed to presenting “misleading results.” This is a well-intentioned impulse, but one should not update his or her confidence in the findings on this basis.

Using balance tests on a subset of observed variables as a way of establishing group equivalence promotes further confusion because of the current popularity of matching techniques in analyzing observational data. If, for example, a researcher matches treated and untreated subjects on five demographic characteristics, there is a tendency to see this as equivalent to an experiment in which a balance test has been performed on these same five variables. What is lost here is an understanding of the fundamental importance of random assignment. Matching techniques, no matter how complex, cannot accomplish the same strength of causal inference as a true experiment. Only random assignment collectively equates subjects on observed and unobserved characteristics.

The lure of the unlucky draw, however, goes far beyond this. There is a strong urge to believe that one can test for occurrences of the exceptional event: that not only does Type I error have a visible signature but also that we can sense it, and therefore should look at balance tests even though we are not able to prescribe an acceptable response to what we see. This may be what is responsible for the heightened concern about errors in randomization.

Sub-Optimal Statistical Models

Randomization checks notwithstanding, a more serious and widespread problem in political science experiments is confusion surrounding analyzing experimental versus observational data. By the Standards Committee's own count, 75% of the experimental studies published in political science do not show the unadulterated effects of treatments on outcomes (Gerber et al. Reference Gerber, Arceneaux, Boudreau, Dowling, Hillygus, Palfrey, Biggers and Hendry2014: 88). In other words, 75% of experimental results never show the reader the dependent variable means by experimental condition or a regression including only treatment effects; instead, they present multiple regression equations in which effects of treatment are already adjusted by many other “control” variables, or they present predicted means as a function of a multivariate regression equations including other variables (e.g., Hutchings et al. Reference Hutchings, Valentino, Philpot and White2004: 521–2).

For example, in one analysis of experimental results, in addition to dummy variables representing six different experimental treatments, the author includes in his experimental regression analysis nine different “control variables” including if the respondent follows politics “most of the time,” if he/she is a college graduate, age, female, minority, employed, internet connection speed, conservative ideology and liberal ideology. The rationale for this particular set of variables when predicting the dependent variable––nightly news exposure––is unclear (see Prior Reference Prior2009). Likewise, in another experiment, in addition to dummy variables for experimental treatment effects, the author includes 13 additional predictors of the outcome, none of which significantly predicts the dependent variable, and the reader is instructed that these variables are included as “control variables” (Ladd Reference Ladd2010: 39).

What is particularly unfortunate about this practice is that reviewers and authors often seem to be under the impression that an experimental finding is more robust if it survives the inclusion of a large number of “control variables” when nothing could be further from the truth. Instead of encouraging this practice, reviewers and editors should look at such large models with suspicion and demand justifications for the particular statistical model that is used. Findings can be coaxed over the line of statistical significance by virtue of what is included or excluded.

We are not suggesting that social scientists are dishonest when such variables are included in a model. In fact, many authors find themselves compelled to include them in an analysis specifically because of a reviewer or editor's request. Even when they are aware that their finding is more valid without excessive and unjustified variables in the model, they comply in order to achieve publication. Adding variables after the fact invalidates the reporting of confidence levels. Moreover, the proper reporting of confidence is not an idle exercise; in fact, some suggest that it has large scale consequences (Ioannidis Reference Ioannidis2005).

In some cases, these additional variables in models testing experimental effects even include items assessed after the treatment. For example, in an article in the American Journal of Political Science, a study of income distribution norms and distributive justice promotes the inclusion of a variable assessed post-treatment as a means of strengthening confidence in the experimental findings.

In short, what is most common within political science is for researchers to analyze experimental data as if it were observational data, often including “control variables” inappropriately. If there is no reason to think variables will increase efficiency in the estimation of treatment effects, and no reason to think that they are even correlated with the outcome, they should not be in the model, regardless of what they are called. Which other variables are or are not included is unsystematic and typically unjustified with some models including one set, and another model within the same paper including a different set, thus opening the floodgates for all kinds of foraging for results through their inclusion and exclusion.

We are not the first to note the logical problems inherent in randomization checks. Psychologist Robert Abelson (Reference Abelson1995: 76) dubbed the practice of testing for differences between experimental groups a “silly significance test”: “Because the null hypothesis here is that the samples were randomly drawn from the same population, it is true by definition, and needs no data.” Senn (Reference Senn1994: 1716) calls the practice of performing randomization tests “philosophically unsound, of no practical value, and potentially misleading.” In the context of political science, Imai and colleagues (Reference Imai, King and Stuart2008: 482) echo this sentiment, suggesting that any other purpose [than to test the randomization mechanism] for conducting such a test is “fallacious.”

The field of political science is populated with researchers primarily trained in observational modes of research. For those trained exclusively in observational methods, the source of confusion is obvious. If one treats experimental data as if it were observational, then of course one would be worried about “controlling for” variables, and about imbalance in any variable not used as a covariate. We believe the Standards Committee should take a stand on whether they believe “control” variables are sensible in experimental studies and/or whether they are an acceptable fix for the broken random assignment mechanisms that balance tests are supposedly designed to root out.

So, how can researchers be certain that randomization was done properly? The CONSORT guidelines already provide guidance as to the kinds of details that can help reviewers and readers judge the randomization process (see Moher et al. Reference Moher, Hopewell, Schulz, Montori, Gøtzsche, Devereaux, Elbourne, Egger and Altman2010; Schulz et al. Reference Schulz, Altman and Moher2010). Notably, because randomization is a process rather than an outcome, what is more useful than tables of means is a description of that process. Political scientists should test randomization mechanisms in advance of studies if there are concerns, and promote transparency by describing the process of randomization for each study.

When debating the utility of randomization checks, one argument we have heard a number of times is “Why not just do both and let the reader decide?” In other words, why not present both the original analysis and one adjusted by including the covariates that fail a specified balance test? Assuming the randomization mechanism is not faulty, there remain several good reasons not to do this. We elaborate on four such reasons.

1. 1. Incorrectness. Significance statements and size comparisons for the estimated treatment effect will be wrong. To see why, consider the process by which the adjusted estimate is computed. After the random assignment to experimental conditions, a set of covariates exhibiting imbalance is added to the model. An estimated treatment effect is computed by regressing onto the treatment variable and this larger set of covariates. Confidence intervals and p-values for such an estimator do not coincide with confidence intervals and p-values for a model in which the same covariates are chosen before the units are assigned to conditions (see Permutt Reference Permutt1990).

2. 2. Intractability. Computing correct confidence statements for a model in which covariate selection is not fixed in advance has, to our knowledge, never been undertaken. An idealized example is worked out in Permutt (Reference Permutt1990). Whether or not such a computation is feasible, it is certainly not included in any standard statistics package. We can therefore be fairly certain that the correct computation was not carried out.

3. 3. Inefficiency. Even if confidence was computed correctly for the adjusted estimate, the new estimator would not be an improvement over the old one. Any selection of covariates, whether chosen in advance or based on imbalance due to random assignment, leads to an estimator. The better estimator is the one with the least variance. For any pre-treatment measure, Z, one might choose to include Z in the model, exclude it, or include it only if it is unbalanced. The last of these is never the best choice. One always does better by deciding up front whether to include Z as a covariate. The mathematical proof supporting this is discussed in greater detail in Mutz and Pemantle (Reference Mutz and Pemantle2011).

4. 4. Irrelevance. One might argue that presenting both estimators and allowing the reader to choose is best because it reports everything that would originally have been reported, plus one more piece of data which the reader is free to ignore. Reporting a second conclusion, however, casts doubt on the first conclusion; it does not merely add information. It leads to “the wrong impression that we need balance, which is one of the many myths of randomization” (Statisticalmisses.nl 2013). Recommendation E2 of the Standards for Experimental Research calls for an analysis to be specified prior to the experiment, and deviations from this come at a cost in credibility. Furthermore, if given a choice between two models, many would automatically choose the model with more covariates based on a (faulty) belief that such models are more robust. The researcher's job is to present the best data analysis, not to present them all and allow the reader to choose.