Exploring the Discrepancy

What accounts for the large difference in treatment effects between field and survey experiments? One possibility is publication bias. Null results may be less likely to be published than significant results, particularly in a survey setting. A second possibility is social desirability bias, which may cause respondents to under-report socially undesirable behavior. Related is hypothetical bias, in which costs are more abstract in hypothetical environments. Survey and field experiments may also not mirror each other and/or real-world voting decisions. Potential ways in which the survey setting may differ from the field are: treatment salience and noncompliance, differences in outcome choices, and costliness/decision complexity. Weak treatments and noncompliance may decrease treatment effect sizes in field experiments. Design decisions may change the choice sets available to respondents. Finally, surveys may not capture the complexity and costliness of real-world voting decisions. It is possible that more complex factorial designs—such as conjoint experiments—may more successfully approximate real-world settings. However, common methods of analysis of conjoint experiments may not capture all theoretical quantities of interest.

Publication Bias and P-Hacking

Publication bias and p-hacking can lead to overestimated effects in meta-analysis (Carter et al. Reference Carter, Schönbrodt, Gervais and Hilgard2019; Duval and Tweedie Reference Duval and Tweedie2000; Sterne, Egger, and Smith Reference Sterne, Egger and Smith2001; van Aert, Wicherts, and van Assen Reference Van Aert, Wicherts and van Assen2019). While I have identified heterogeneity stemming from the type of experiment performed as a potential source of overestimation, this may reflect that null results are less likely to be published than studies with large and significant negative treatment effects. I therefore now turn to the possibility of publication bias and/or p-hacking. To formally test for publication bias, I use the p-curve, examination of funnel plot asymmetry, trim and fill, and PET-PEESEFootnote ⁶ methods.Footnote ⁷

Of the eight field experimental papers located, only five are published. By contrast, only one of the 15 survey experimental papers remains unpublished, and this is a recent draft. This may reflect that the null results from field experiments are less likely to be published than their survey counterparts with large and highly significant negative treatment effects. While recognizing that the sample size of studies is small, OLS and logistic regression do not indicate that reported p-value is a significant predictor of publication status, although the directionality of coefficients is consistent with lower p-values being more likely to be published (see Online Appendix Table A.11). However, this simple analysis is complicated by the fact that the p-value associated with the average treatment effect across all subjects may not be the primary p-value of interest in the paper.

To more formally test for publication bias, I first use the p-curve (Simonsohn, Nelson, and Simmons Reference Simonsohn, Nelson and Simmons2014a, Reference Simonsohn, Nelson and Simmons2014b; Simonsohn, Simmons, and Nelson Reference Simonsohn, Simmons and Nelson2015). The p-curve is based on the premise that only “significant” results are typically published, and it depicts the distribution of statistically significant p-values for a set of published studies. The shape of the p-curve is indicative of whether or not the results of a set of studies are derived from true effects, or from publication bias. If p-values are clustered around 0.05 (i.e., the p-curve is left skewed), this may be evidence of p-hacking, indicating that studies with p-values just below 0.05 are selectively reported. If the p-curve is right skewed and there are more low p-values (0.01), this is evidence of true effects. All significant survey experimental results included in the meta-analysis are significant at the 1% level, implying that publication bias likely does not explain the large negative treatment effects in survey experiments.Footnote ⁸ For field experiments, there is not a large enough number of published experiments to make the p-curve viable.Footnote ⁹ Only six studies are published, and of these only four are significant at at least the 5% level.

Next, I test for publication bias by examining funnel plot asymmetry. A funnel plot depicts the outcomes from each study on the x-axis and their corresponding standard errors on the y-axis. The chart is overlaid with an inverted triangular confidence interval region (i.e., the funnel), which should contain 95% of the studies if there is no bias or between study heterogeneity. If studies with insignificant results remain unpublished the funnel plot may be asymmetric. Both visual inspection and regression tests of funnel plot asymmetry reveal an asymmetric funnel plot when the survey and field experiments are grouped together (see Online Appendix Figure A.7 and Table A.12). However, this asymmetry disappears when accounting for heterogeneity by type of experiment, either with the inclusion of a survey experiment moderator (dummy) variable or by analyzing field and survey experiments separately (see Online Appendix Table A.12 and Figures A.9–A.11). Trim and fill analysis overestimates effect sizes and hypothesizes that three studies are missing due to publication bias when analyzing all studies together (see Online Appendix Figure A.8 and Table A.13). However, when trim and fill is used on survey experiments or field experiments as separate subgroups, estimates remain unchanged from random effects meta-analysis and no studies are hypothesized to be missing. Similarly, PET-PEESE estimates remain virtually unchanged when survey and field experiments are analyzed as separate subgroups.Footnote ^10, Footnote ¹¹

In sum, while publication bias cannot be ruled out completely—particularly with such a small sample size of field experiments—there is no smoking gun. This implies that differences in experimental design likely account for the difference in the magnitude of treatment effects in field versus survey experiments, rather than publication bias.

Social Desirability Bias and Hypothetical Bias

A second possible explanation is social desirability or sensitivity bias, in which survey respondents under-report socially undesirable behavior. A respondent may think a particular response will be perceived unfavorably by society as whole, by the researcher(s), or both, and they underreport such behavior. In the case of corruption, respondents are likely to perceive corruption as harmful to society, the economy, and their own personal well-being. They may therefore be more likely to choose the socially desirable option (no corruption), particularly when observed by a researcher or afraid of response disclosure.Footnote ¹² However, a researcher is not the only social referent to whom a respondent may wish to give a socially desirable response. Respondents also may not wish to admit to themselves that they would vote for a corrupt candidate. Voting against corruption in the abstract may therefore reflect the respondents’ actual preferences.

However, sensitivity bias is unlikely to account entirely for the difference in magnitude of treatment effects. A recent meta-analysis finds that sensitivity biases are typically smaller than 10 percentage points and that respondents under-report vote buying by 8 percentage points on average (Blair, Coppock, and Moor Reference Blair, Coppock and Moor2018). As vote buying is often considered a form of corruption, the amount of sensitivity bias present in corruption survey experiments may be similar.

A related but distinct source of bias is hypothetical bias. Hypothetical bias is often found in stated preference surveys in environmental economics, in which respondents report a willingness to pay that is larger than what they will actually pay using their own money because the costs are purely hypothetical (Loomis Reference Loomis2011). For corruption experiments, this would manifest as respondents reporting a willingness to punish corruption larger than in reality as the costs in terms of trade-offs are purely hypothetical. There are few costs to selecting the socially desirable option in a hypothetical survey experiment. By contrast, the cost of changing one’s actual vote (as in field experiments) may be higher. Voters might have pre-existing favorable opinions of real candidates, discount corruption information, or have strong material or ideological incentives to stick with their candidate. As the informational treatment will only have an effect on supporters of the corrupt candidate who must change their vote—opponents have already decided not to vote for the candidate—these costs are particularly high. Where anticorruption norms are particularly strong—as in Brazil as highlighted by Boas, Hidalgo, and Melo (Reference Boas, Daniel Hidalgo and Melo2019)—the magnitude of hypothetical bias may be particularly large.

How might we overcome social desirability bias and hypothetical bias in survey experiments? For social desirability bias, one option is the use of list experiments. None of the survey experiments included here are list experiments. More complex factorial designs such as conjoint experiments have also been shown to reduce social desirability bias (Hainmueller, Hopkins, and Yamamoto Reference Hainmueller, Hopkins and Yamamoto2014; Horiuchi, Markovich, and Yamamoto Reference Horiuchi, Markovich and Yamamoto2018). For hypothetical bias, an option is to eschew hypothetical candidates in favor of real candidates. In fact, the only corruption survey experiment to date to use real candidates found a null effect on vote choice (De Figueiredo, Hidalgo, and Kasahara Reference De Figueiredo, Hidalgo and Kasahara2011), and McDonald (Reference McDonald2019) elicits smaller effects in survey experiments using the names of real politicians versus a hypothetical politician. Of course, for corruption experiments this limits researchers to having actual information regarding the corrupt actions of candidates for ethical reasons.

Do Field and Survey Experiments Mirror Real-World Voting Decisions?

Even if subjects (voters), treatments (information), and outcome (vote choice) are similar, contextual differences between survey and field experiments may also offer fundamentally different choice sets to voters. These discrepancies between survey and field experimental designs, as well as those between the designs of different survey experiments, may alter respondents’ potential outcomes and thus capture different estimands. Some possible contextual differences are discussed below.

Treatment Strength, Noncompliance, and Declining Salience

Informational treatments may be weaker in field experiments in part because of their method of delivery. Survey treatments tend to be clear and authoritative, and often provide information on the challenger (clean or corrupt). By contrast, many of the informational treatments used in past information and accountability field experiments—fliers and text messages—provide relatively weak one-time treatments that may even contain information subjects are already aware of. If the goal is to estimate real world effects, interventions should attempt to match those conducted in the real world (e.g., by campaigns, media, etc.). In fact, the natural experiment conducted by Ferraz and Finan (Reference Ferraz and Finan2008)—which takes advantage of random municipal corruption audits conducted by the Brazilian government—may provide evidence of the effectiveness of stronger treatments. The results of the audits were disseminated naturally by newspapers and political campaigns, and their study provides the largest estimated treatment effect amongst real-world experiments. While not measuring specific vote choice, past experiments using face-to-face canvassing contact have also demonstrated relatively large effects on voter turnout (Green and Gerber Reference Green and Gerber2019; Kalla and Broockman Reference Kalla and Broockman2018), but these methods have not been used in any information and accountability field experiments to date.

Treatment effects in field experiments (fliers, newspapers, etc.) may also be weaker in part because they can be missed by segments of the treatment group. More formally, survey experiments do not have noncompliance by design; therefore, the average treatment effect (ATE) is equal to the intent-to-treat (ITT) effect,Footnote ¹³ whereas field experiments present ITT estimates because they are unable to identify which individuals in the treatment area actually received and internalized the informational treatment. Ideally, we would calculate the complier average causal effect (CACE)—the average treatment effect among the subset of respondents who comply with treatment—in field experiments, but we are unfortunately unable to observe compliance in any of the corruption experiments conducted to date.

A theoretical demonstration shows how noncompliance can drastically alter the ITT. The ITT is defined as ITT = CACE × πC, where πC indicates the proportion of compliers in the treatment group. When πC = 1, ITT = CACE = ATE. If the ITT = -0.0033—as random effects meta-analysis estimates in field experiments—but only 10% of treated individuals “complied” with the treatment by reading the flier sent to them, this implies that the CACE is , or approximately -3 percentage points. In other words, while the effect of receiving a flier is roughly -0.3 percentage points, the effect of reading the flier is -3 percentage points. As the ITT = CACE × πC, any noncompliance necessarily reduces the size of the ITT. However, for the CACE to be equal in both survey and field experiments, the proportion of treatments that would need to remain undelivered in field experiments would have to be approximately 99% (i.e., 99% of subjects in the treatment group did not receive treatment or were already aware of the corruption information), implying that noncompliance likely does not tell the whole story.

Finally, treatments may be less salient at the time of vote choice in a field setting. Survey treatments are directly presented to respondents who are forced to immediately make a vote choice. Kalla and Broockman (Reference Kalla and Broockman2018) note that this mechanism manifests in campaign contact field experiments, where contact long before election day followed by immediate measurement of outcomes appears to persuade voters, whereas there is a null effect on vote choice on election day. Similarly, Sulitzeanu-Kenan, Dotan, and Yair (Forthcoming) show that increasing the salience of corruption can increase electoral sanctioning, even without providing any new corruption information. Weaker treatments or lower salience of corruption in field experiments will weaken the treatment effect even amongst compliers (i.e., the CACE), further reducing the ITT.

Weak treatments, noncompliance, and declining treatment salience over time therefore make it unclear whether the zero and null effects observed in field experiments stem from methodological choices or an actual lack of preference updating. Future field experiments should therefore consider using stronger treatments (e.g., canvassing), performing baseline surveys to measure subgroups amongst whom effects may be stronger, using placebo-controlled designs that allow for measurement of noncompliance, and performing repeated measurement of outcome variables over time to capture declining salience.

Outcome Choice

While vote choice is the outcome variable across all of the experiments investigated here, the choice set offered to voters is not necessarily always identical. Consider a voter’s choice between two candidates in a field experiment conducted during an election. A candidate is revealed to be corrupt to voters in a treatment group but not to voters in control. The treated voter can cast a ballot for corrupt candidate A, or candidate B, who may be clean or corrupt. The control voter can cast a ballot for candidate A or candidate B, and has no corruption information. Now consider a survey experiment with a vignette in which the randomized treatment is whether the corrupt actions of a politician are revealed or not. The treated voter can vote for the corrupt candidate A or not, but no challenger exists. Likewise, the control voter can vote for clean candidate A or not, but no challenger exists. Conjoint experiments overcome this difference, but the option to abstain still does not exist in the survey setting.Footnote ¹⁴ These differences in design offer fundamentally different choice sets to voters, altering respondents’ potential outcomes and thus capturing different estimands.

Complexity, Costliness, and Conjoint Experiments

Previous researchers have noted that even if voters generally find corruption distasteful, the quality of the information provided or positive candidate attributes and policies may outweigh the negative effects of corruption to voters, mitigating the effects of information provision on vote share.Footnote ¹⁵ These mitigating factors will naturally arise in a field setting, but may only be salient to respondents if specifically manipulated in a survey setting.

A number of survey experiments have therefore added factors other than corruption as mitigating variables, such as information quality, policy, economic benefit, and co-partisanship. Studies have randomized the quality of corruption informationFootnote ¹⁶ (Banerjee et al. Reference Banerjee, Green, McManus and Pande2014; Botero et al. Reference Botero, Cornejo, Gamboa, Pavao and Nickerson2015; Breitenstein Reference Breitenstein2019; Mares and Visconti Reference Mares and Visconti2019; Weitz-Shapiro and Winters Reference Weitz-Shapiro and Winters2017; Winters and Weitz-Shapiro Reference Winters and Weitz-Shapiro2020), finding that lower quality information produces smaller negative treatment effects (see Online Appendix Figure A.13). Policy stances in line with voter preferences have also been shown to mitigate the impact of corruption (Franchino and Zucchini Reference Franchino and Zucchini2015; Rundquist, Strom, and Peters Reference Rundquist, Strom and Peters1977). Evidence also suggests that respondents are more forgiving of corruption when it benefits them economically (Klašnja, Lupu, and Tucker Reference Klašnja, Lupu and Tucker2017; Winters and Weitz-Shapiro Reference Winters and Weitz-Shapiro2013). Evidence of co-partisanship as a limiting factor to corruption deterrence is mixed.Footnote ¹⁷ Boas, Hidalgo, and Melo (Reference Boas, Daniel Hidalgo and Melo2019) posit that abandoning dynastic candidates is particularly costly in Brazil. This evidence suggests that voters punish corruption less when it is costly to do so and that these costly factors differ by country.

The fact that moderating variables may dampen the salience of corruption to voters has clearly not been lost on previous researchers. However, in the field setting numerous moderating factors may be salient to the voter. While there is likely no way to capture the complexity of real-world decision making in a survey setting, conjoint experiments allow researchers to randomize many candidate characteristics simultaneously, and thus they have become a popular survey method for investigating the relative weights respondents give to different candidate attributes. In addition, conjoints force respondents to pick between two candidates, better emulating the choice required in an election. Finally, conjoints may minimize social desirability bias because they reduce the probability that the respondent is aware of the researcher’s primary experimental manipulation of interest (e.g., corruption).Footnote ¹⁸

Researchers often present the results of conjoint experiments as average marginal component effects (AMCEs), and they then compare the magnitude of these effect sizes. The AMCEs represent the unconditional marginal effect of an attribute (e.g., corruption) averaged over all possible values of the other attributes. This measurement is valuable, and crucially allows researchers to test multiple causal hypotheses and compare relative magnitudes of effects between treatments. However, this may or may not be a measure of substantive interest to the researcher, and it implies that the AMCE is dependent on the joint distribution of the other attributes in the experiment.Footnote ¹⁹ These attributes are usually uniformly randomized. However, in the real world, candidate attributes are not uniformly distributed, so external validity is questionable. When we have a primary treatment of interest, such as corruption, we want to see how a “typical candidate” is punished for corruption. However a typical candidate is not a uniformly randomized candidate, but rather a candidate designed to appeal to voters. The corruption AMCE is therefore valid in the context of the experiment—marginalizing over the distribution of all other attributes in the experiment—but would likely be much smaller for a realistic candidate.Footnote ²⁰ This implies that AMCEs have more external validity when the joint distribution of attributes matches the real world and the experiment contains the entire universe of possible attributes.Footnote ²¹

When researchers have strong theories about the conditions that shape voter decision-making, a more appropriate method may be to calculate average marginal effects to present predicted probabilities of voting for a candidate under these conditions;Footnote ²² for example, in a conjoint experiment including corruption information, the probability of voting for a candidate who is both corrupt and possesses other particular feature levels (e.g., party membership or policy positions), marginalizing across all other features in the experiment.Footnote ²³

To illustrate this point, I replicate the conjoint experiments conducted in Spain by Breitenstein (Reference Breitenstein2019) and in Italy by Franchino and Zucchini (Reference Franchino and Zucchini2015) and present both AMCEs and predicted probabilities. The Breitenstein (Reference Breitenstein2019) reanalysis is presented in the main text, while the reanalysis of Franchino and Zucchini (Reference Franchino and Zucchini2015) is in the appendix.Footnote ²⁴ Note that I group all corruption accusation levels into a single “corrupt” level in my replications. The Breitenstein (Reference Breitenstein2019) predicted probabilities are presented as a function of corruption, co-partisanship, political experience, and economic performance. The charts therefore show the probability of preferring a candidate who is always corrupt, but is a co-partisan or not, has low or high experience, and whose district experienced good or bad economic performance, marginalizing across all other features in the experiment. For Franchino and Zucchini (Reference Franchino and Zucchini2015), the predicted probabilities are presented as a function of corruption and two policy positions—tax policy and same sex marriage—separately for conservative and liberal respondents. The charts therefore show the probability of preferring a candidate who is corrupt, but has particular levels of tax and same sex marriage policy, marginalizing across all other features in the experiment. Note that Franchino and Zucchini (Reference Franchino and Zucchini2015) correctly conclude that their typical “respondent prefers a corrupt but socially and economically progressive candidate to a clean but conservative one,” and Breitenstein (Reference Breitenstein2019) presents certain predicted probabilities. While I therefore illustrate how predicted probabilities can be used to draw conclusions that may be masked by examination of AMCEs alone, the authors themselves do not make this mistake. I perform the same analysis including only cases where the challenger is clean in the appendix.

A casual interpretation of the traditional AMCE plots presented in Figure 3 and Online Appendix Figure A.17 suggests that it is very unlikely a corrupt candidate would be chosen by a respondent. By contrast, the predicted probabilities plots presented in Figure 4 and Online Appendix Figures A.18–A.19 show that even for corrupt candidates in the conjoint, the right candidate or policy platform presented to the right respondents can garner over 50% of the predicted hypothetical vote.Footnote ²⁵ Further, the attributes included in these conjoints surely do not represent all candidate attributes relevant to voters, and indeed they differ greatly across experiments. As in Agerberg (Reference Agerberg2020), the level of support for corrupt candidates also varies based on whether or not the challenger is clean (see Online Appendix Figures A.14, A.20, and A.21). In other words, respondents find it costly to abandon their preferences even if it forces them to select a corrupt candidate, and this costliness varies highly depending on contextual changes and choice of other attributes included in the experiments.

Figure 3.

Breitenstein (Reference Breitenstein2019) Conjoint: Average Marginal Component Effects

Figure 4.

Breitenstein (Reference Breitenstein2019) Conjoint: Can the Right Candidate Overcome Corruption?

Candidate or policy profiles that result in over 50% of voters selecting a corrupt candidate may not be outliers in real-world scenarios. Unlike in conjoint experiments, real-world candidates’ attributes and policy profiles are not selected randomly, but rather represent choices designed to appeal to voters. Voters may also be unsure whether the challenger is also corrupt or clean. It may therefore be preferable to analyze conjoint experiments as above, comparing outlier characteristics (e.g., corruption) with realistic candidate profiles that target specific voters, rather than fully randomized candidate profiles.

When the most theoretically relevant trade-offs are unclear, we may be able to illuminate voter decision making processes through the use of decision trees.Footnote ²⁶ The decision tree in Figure 5 was trained using all randomized variables in the Breitenstein (Reference Breitenstein2019) conjoint, and the tree was pruned to minimize cross-validated classification error rate. Figure 5 draws similar conclusions to the predicted probabilities chart shown in Figure 4 with respect to what factors matter most to voters. A similar figure depicting corrupt candidates facing clean challengers only can be found in Online Appendix Figure A.16.

Figure 5.

Breitenstein (Reference Breitenstein2019) Conjoint Decision Tree: Predicted Probabilities of Voting for Candidate