Political science has increasingly embraced the experimental method to establish causal relationships suggested by theories and observational studies—from experiments’ traditional sub-disciplinary home, political psychology, to international relations. The most frequently voiced concern with experiments, qualms about their external validity in terms of sample, has been well documented and addressed (Coppock, Leeper, and Mullinix Reference Coppock, Leeper and Mullinix2018; Krupnikov and Levine Reference Krupnikov and Levine2014; Krupnikov, Nam, and Style Reference Krupnikov, Nam, Style, Druckman and Green2021; Lupton Reference Lupton2019; McDermott Reference McDermott, Druckman, Greene, Kuklinski and Lupia2011; Mutz Reference Mutz, Druckman and Green2021).Footnote 1 Experiments uniquely provide scholars with the internal validity necessary to confidently identify causal effects, and issues with specific experiments tend to arise through errors of application by individual scholars rather than through any broad problems with the methodology.
Yet while experiments allow scholars to identify causal effects, how and why treatment effects occur remain questions that challenge experimentalists. The issue is identifying which of many possible mechanisms transmit an effect, and weaknesses exist in current experimental techniques in parsing these mechanisms. But doing so is key to designing real-world policy interventions. For example, if citizens are reluctant to support an African American Supreme Court justice or presidential candidate, it matters whether they do so because of the color of the nominee’s skin or because African Americans are assumed to be liberal (McDermott Reference McDermott1998; Sen Reference Sen2017). The two mechanisms suggest different solutions to address the problem. Without knowledge of causal anatomies, it is possible to predict outcomes but not to prescribe policy changes.Footnote 2
This special issue has two aims. First, I highlight recently developed methodological tools that confront challenges involved in identifying causal mechanisms. Scholars often use composite treatments that allow for an efficient design but leave them unable to specify which mechanism is responsible for effects. Hainmueller, Hopkins, and Yamamoto (Reference Hainmueller, Hopkins and Yamamoto2014) offered a solution that avoids this trade-off with conjoint analysis. This work was extended by Jenke et al. (Reference Jenke, Bansak, Hainmueller and Hangartner2021), who validated attribute importance as the mechanism behind average marginal component effects (AMCEs) in conjoint analysis using eye-tracking. Eye-tracking allows for the measurement of information accrual and processing, which gives insight into the cognitive models employed behind a choice. Another weakness in testing causal mechanisms is that average causal mediation effects rely on strong assumptions that are often known to be false. Bansak (Reference Bansak2020) presented causal mediation estimands that trace the causal anatomies of multiple treatments without the assumption of no unobserved confounding of the mediator–outcome relationship. Acharya, Blackwell, and Sen (Reference Acharya, Blackwell and Sen2018) also relaxed this assumption in their approach to mediation analysis, which experimentally manipulates the mediator rather than observing it.
Second, I include papers that underscore rarely noted issues with experiments, some of which have solutions and others that remain unsolved. Miratrix et al. (Reference Miratrix, Sekhon, Theodoridis and Campos2018) and Dafoe, Zhang, and Caughey (Reference Dafoe, Zhang and Caughey2018) focused on weighting and the information equivalence (IE) assumption, respectively, and provided needed recommendations for best practices. Gaines, Kuklinski, and Quirk (Reference Gaines, Kuklinski and Quirk2007) concentrated on four issues: the frequency with which subjects are treated in an experiment versus in the real world and the endurance of treatment effects, the possibility of mutual causation, the lack of control groups in many studies, and spillover effects. These issues are straightforward yet vital to experimental validity. Last, Franco, Malhotra, and Simonovits (Reference Franco, Malhotra and Simonovits2015) brought needed attention to the underreporting of experimental results. My goal in highlighting these concerns is not to question past work but instead to increase the future scholarly returns of experiments.
1 Innovations
Experimentalists have traditionally faced the issue that the often-composite nature of experiments’ treatment conditions leaves scholars uncertain as to the specific component responsible for an experimental effect. Treatments are often multidimensional, consisting of the alteration of many aspects of a prompt rather than a single element, and those that are one-dimensional suffer from confounding effects of other, correlated components. Hainmueller et al. (Reference Hainmueller, Hopkins and Yamamoto2014) have offered a solution with conjoint analysis. Conjoint analysis permits the identification of component-specific effects while allowing scholars to test causal hypotheses about multidimensional preferences efficiently, in a single experiment. Conjoint designs are suited for use in a wide swath of substantive areas including voting, public opinion, climate change, and international relations (Bansak et al. Reference Bansak, Hainmueller, Hopkins, Yamamoto, Druckman and Green2021).
Conjoint experiments and most experimental studies consist of asking respondents to state their choices, leaving the cognitive processes behind these stated choices unspecified. Jenke et al. (Reference Jenke, Bansak, Hainmueller and Hangartner2021) leveraged eye-tracking to provide data on the information-gathering processes of respondents who were completing a conjoint experiment. Previous papers had established AMCEs as indicating the impact of attributes’ effects. However, the assumed mechanism behind the causal effect—attribute importance—had not been validated. Eye-tracking allowed the authors to validate the interpretation of AMCEs as the relative importance of components in the decision. They also measured the change in information processing that occurs as conjoint tables become increasingly complex, showing that while the AMCEs remained consistent, respondents transitioned from one processing strategy to another in a manner that is consistent with bounded rationality. Broadly speaking, eye-tracking can be used to inform scholars as to the mechanisms behind causal effects if the mechanisms are consistent with one information processing strategy and not another. With the advent of webcam eye-tracking technologies (Semmelmann and Weigelt Reference Semmelmann and Weigelt2018; Xu et al. Reference Xu, Ehinger, Zhang, Finkelstein, Kulkarni and Xiao2015; Yang and Krajbich Reference Yang and Krajbich2020), this method’s accessibility is rapidly improving and its application in experimental political science should increase.
Another improvement was made by Bansak (Reference Bansak2020). He presented comparative causal mediation (CCM) estimands that allow for the estimation of the effects of multiple treatments (and a single mediator) and comparisons of their magnitudes. The advantage of the approach over others is twofold. CCM estimands allow the assumption of no mediator–outcome confounders to be relaxed. This is always a concern because even if some of the possible confounders are measured, some may be missed or are unable to be measured. Audience costs provide a good example. Possible mediator–outcome confounders include age, left–right ideological leanings, beliefs on policy issues, and personal connections to the military. Many datasets utilized by international relations scholars do not contain variables for all of these confounders. Bansak’s approach also permits unit-specific parameters rather than assuming constant effects. The assumption of constant effects across all people is an extremely rigid assumption that is known to be false in most areas of study. With these advantages, CCM estimands should be used whenever designs feature multiple treatments.
Acharya et al. (Reference Acharya, Blackwell and Sen2018) likewise avoided the assumption of no mediator–outcome confounders in their strategy to analyze causal mechanisms. Imai et al.’s (Reference Imai, Keele, Tingley and Yamamoto2011) foundational approach provided a framework to identify indirect effects, but their approach requires this assumption. Acharya et al.’s strategy is to manipulate the information environment of a survey experiment such that respondents selectively receive information about the mediator of interest rather than observing it. The difference between the overall average treatment effect and the treatment effect when the mediator is set at a particular value is identified as the role of the mechanism in the treatment effect, either through indirect effects or interactions. Their approach allows scholars to tease apart causal mechanisms without the restrictive assumptions previously employed.
2 Underexamined Procedures and Issues
In addition to new methodologies, scholars have provided useful recommendations regarding underexamined procedures involved in experiments. One such area is weighting. Scholars publishing survey experiments rarely state their weighting procedures and even more rarely provide a justification for weighting or not weighting (Franco et al. Reference Franco, Malhotra, Simonovits and Zigerell2017), despite this being precisely what the Standards Committee of the Experimental Research Section of the American Political Science Association suggests (Gerber et al. Reference Gerber2014). Since characteristics that determine selection into survey samples may moderate treatments, it is problematic to treat nonprobability samples as if their sample average treatment effects (SATEs) mimic population average treatment effects (PATEs). Miratrix et al. (Reference Miratrix, Sekhon, Theodoridis and Campos2018) analyzed this problem and provided recommendations for scholars. They recommended that the SATEs should be calculated without weights. Then, the PATEs should be calculated with weights and the two estimates compared. A difference in the estimates should prompt consideration of the quality of the weights and covariates that could explain treatment effect heterogeneity. Regardless of the difference, their analyses suggest that for all survey experiments run on datasets that provide weights to researchers, both the SATEs and the PATEs should be reported.
Dafoe et al. (Reference Dafoe, Zhang and Caughey2018) provided the first in-depth analysis of the IE assumption, which is another understudied area of experimental methodology. This is a crucial assumption in regards to the legitimacy of causal claims in experiments. If the IE assumption does not hold, then causal effects cannot be reliably attributed to the variable of interest and may operate through an unintended causal channel. The authors reviewed three solutions to IE violations—abstract encouragement, covariate control, and embedded natural experiments—and evaluated their effectiveness at reducing them using placebo tests. Across several studies, they found that embedded natural experiments best promote IE. In addition to bringing attention to this underlying assumption of experiments and stressing the importance of considering alternative causal channels through which an effect might flow, they specify a useful recommendation of steps that should be taken to minimize IE violations.
Gaines et al. (Reference Gaines, Kuklinski and Quirk2007) provided a list of rarely noted issues with experiments. These include the mismatch of the number of real-world and experimental treatment incidences over time, the causal complexity that can lie behind the assumption of a one-way causal relationship, experiments that do not include a control group, and accidental spillover effects based on the placement of experiments within a survey. Despite the 14 years since the paper’s publication, many of these challenges continue to be a concern in the literature, and they are important. For instance, the duration of experimental effects determines the implications of experiments for policies, political behavior, and candidate success in the real world. Some papers have measured how long their effects last and discussed the ensuing implications (see, e.g., Chong and Druckman Reference Chong and Druckman2010; Gerber et al. Reference Gerber, Gimpel, Green and Shaw2011; Kalla and Broockman Reference Kalla and Broockman2018; Levendusky Reference Levendusky2013), but the number is far fewer than best practices would hope for. This paper deserves to be read by all experimentalists, and its suggestions warrant common application.
Another undernoted issue is underreporting of experimental results, which was shown by Franco et al. (Reference Franco, Malhotra and Simonovits2015) to be occurring at an alarming rate in political science. The authors analyzed experimental studies that won a Time-Sharing Experiments in the Social Sciences award between 2002 and 2012 and made their way to publication. Based on the publicly available survey questionnaires (which were posted before data collection), they found that 30% of papers did not report all treatment conditions that were part of the experiment. In addition, 60% of papers utilized fewer outcome variables than were listed in the questionnaire. This is problematic because if scholars are selectively reporting only results that confirm their hypotheses, then positive results will be overrepresented and lead to bias in the literature.
A solution to this problem has as of yet not been suggested in political science, beyond the rather optimistic hope of the authors that political science will change its culture and norms. Even if preregistration becomes standard, reviewers will not necessarily compare the reported results to the preregistration; doing so would significantly increase the time needed to review a piece. In other fields where preregistration is required in order to publish, such as medicine, it has been found to be ineffective at preventing scholars from excluding negative results from their published studies (Chan and Altman Reference Chan and Altman2005; Chan et al. Reference Chan, Hróbjartsson, Haahr, Gøtzsche and Altman2004). One potential solution is a requirement by journals that authors of preregistered work explicitly state that they have reported all preregistered measures, conditions, and hypotheses.Footnote 3 This would change underreporting from an act of omission to active dishonesty about one’s work and thus act as a more effective deterrent.
3 Concluding Remarks
Our goal as political scientists is to make discoveries about human behavior in real-world political contexts. To do this, it is necessary to explain outcomes—to establish both causal relationships and the mechanisms behind them—not merely predict outcomes. Predictive models that are not interpretable or do not identify a causal mechanism are of limited utility. Experiments provide the most rigorous way for social scientists to identify causation, and, thanks to recent innovations in experimental methods, we have made inroads in creating feasible ways to analyze causal mechanisms.
The following areas of research would be helpful for future work to focus on. First, we need to continue to create ways to identify which causal mechanisms are at play in a causal effect without introducing overly restrictive assumptions. The papers included in this special issue are a start, but more work needs to be done. Second, research should strive to incorporate the recommendations of Miratrix et al. (Reference Miratrix, Sekhon, Theodoridis and Campos2018), Dafoe et al. (Reference Dafoe, Zhang and Caughey2018), and Gaines et al. (Reference Gaines, Kuklinski and Quirk2007). Third, solutions need to be found for the problematic status quo regarding underreporting of experimental results. As an experimentalist, I view these challenges with excitement, as their solutions will allow us to identify causation and make policy recommendations with increasing confidence.
About the Author
Libby Jenke is an Assistant Professor of Political Science at the University of Houston. She is an experimental methodologist with expertise in eye-tracking and mouse-tracking. Her papers are available at http://www.libbyjenke.com.
