GEOGRAPHICAL DISTRIBUTION OF FIELD EXPERIMENTS IN POLITICAL SCIENCE

Three main sets of issues are explicitly associated with the location of field experiments: (1) the often-unacknowledged power relationships between experimenters and subjects of research; (2) relatedly, the North–South hierarchies embedded in the knowledge-production process that are exacerbated by the high costs associated with field experiments; and (3) site-selection bias and geographical clustering of field experimentation. The first issue usually emphasizes the fact that the randomization “in the tropics” conducted by Western researchers from wealthy institutions on poor subjects may not always meet full ethical standards (Cronin-Furman and Lake Reference Cronin-Furman and Lake2018; Herman et al. Reference Herman, Panin, Owlsley, Blair, Dyzenhaus, Wellman, Grossman, Opalo, Singh, Alarian, Pruett and Tan2022; McDermott and Hatemi Reference McDermott and Hatemi2020). A Nobel Prize–winning social scientist noted “…nearly all RCTs on the welfare system are done by better-heeled, better-educated, and paler people on lower-income, less-educated, and darker people” (Deaton Reference Deaton2020, 21).

When we examined the geographical distribution of the current wave of field experiments, we found that, indeed, the majority of researchers are located in the Global North, with the United States accounting for most of the experiments, as shown in the bottom-left panel of figure 1. However, the top-left panel of figure 1 shows that between 2014 and 2019, the concentration of the country locations of field research, computed as a Hirschmann–Herfindahl index, declined steadily. In the limit, a concentration index of 1 means that all experiments are taking place in one country, whereas a concentration index of 0 means that they are taking place in different countries. ³ The concentration of experiments originating in the developing world has been low and constant over time. The distribution of the data on published field experiments is even more skewed and relatively stable over time: among 16 countries of origin, 84.39% of 205 coauthors were affiliated with US institutions between 2014 and 2019. Less than 4% of published RCTs in political science originated from three emerging economies (i.e., Brazil, China, and Russia) and none from low-income countries. Therefore, scholars must grapple with the following tradeoff: as experimental evidence becomes more standard, the research of political scientists trained at and employed by North American institutions, although contributing significantly to knowledge, entails potential positionality biases.

Figure 1

Geographic Clustering of Field Experiments

The top-left/top-right panel plots calculated Hirschman–Herfindahl Indexes of concentration by the country of institutions/study location country of preregistered experiments. The bottom-left/bottom-right panel plots country shares of institutions/study location country of preregistered experiments.

It is notable, however, that despite the recent notoriety gained by the randomista movement in development, field experimentation has much deeper historical roots in advanced industrial democracies. The earliest field experiments in political science studied “get-out-the-vote” mailings during the 1924 US presidential election; followed by voter mobilization in Ann Arbor, Michigan, in the early 1950s (Gerber Reference Gerber, Druckman, Greene, Kuklinski and Lupia2011); and several US federal and local government program evaluations during the 1960s and 1970s (Ogden Reference Ogden2017). Assuaging the randomization “in-the-tropics” concern to some extent, field experiments in political science have developed a geographical bimodal distribution over time, with studies of voting behavior mainly focused on the United States. This was followed in a second wave by experiments on non-Western countries in the subfields of comparative politics and political economy of development. Empirically, what is striking in our data in terms of study location is the shift from the non-advanced industrialized countries into the United States and Europe during the period we studied (2014–2019). Figure 2 demonstrates the similarly increasing trend in “RCT domestication” (i.e., the researcher country also being the site of experimentation) in the top-left panel and the relative decline of the share of “North–South” experiments (i.e., an advanced industrialized country fielding an experiment in a developing country) in the top-right panel. “Domestication” refers to the country of study corresponding to the country of the lead researcher’s institution. ⁴

Figure 2

Geographic Positionality of Field Experiments

The top-left panel plots the share of preregistered experiments taking place in the same country as the preregistering institution. The top-right panel plots the share of experiments preregistered by an institution in an advanced industrialized democratic (AID) country taking place in a non-AID country. The bottom-left panel plots the share of preregistered experiments taking place outside of the preregistering country where English is an official language in the location. The bottom-right panel shows the shares of the top nine country locations between 2014 and 2019.

In parallel, and to the contrary, there is an opposite concern in terms of geographical coverage of experimentation from a knowledge-gain perspective—namely, that the geographical focus in political science traditionally has been Western-centric rather than global. This epistemic concern rests on a long-standing geographical imbalance in political science research. Wealthier democratic countries with English, Spanish, or French as their main spoken language were more likely to be studied (Wilson and Knutsen Reference Wilson and Knutsen2020). An emerging literature also demonstrates that field experiments are more likely to occur in certain contexts (i.e., political, geographical, and institutional) (Blair, Iyengar, and Shapiro Reference Blair, Iyengar and Shapiro2013; Corduneanu-Huci, Dorsch, and Maarek Reference Corduneanu-Huci, Dorsch and Maarek2021; Das Reference Das2020). Our data optimistically show that the sites of experimentation diversified to more than 100 countries between 2014 and 2019.

The share of nondomestic experiments taking place in a country where English is an official language—a potential source of site-selection bias—also showed a marked decrease during the period. Moreover, several low- and middle-income countries from Sub-Saharan Africa, East Asia, and the Middle East (e.g., Ghana, Liberia, Mali, Uganda, Lebanon, and Vietnam) were in the top-10 ranking of RCT geographical coverage as they were featured in multiple studies published in leading political science journals (Figure A5 in the Online Appendix). These countries previously were epistemically marginal in the leading political science literature. Liberia, for instance, was studied in only 14 of a total 27,689 articles appearing in top political science journals during more than a century. However, it became a top-10 site for field experiments, accounting for approximately 6% of all field RCTs published in the same journals in only five recent years.

Nevertheless, despite clear evidence of geographical diversification and epistemic gains in previously neglected contexts, the experimental sites remained highly concentrated, with only five countries accounting for approximately 50% of all preregistered field experiments. If anything, the geography of RCTs became more concentrated during the period that we studied. The cautious diversification of experimental sites according to geographical and topical evidence-gap maps, coupled with stricter ethical criteria for field experimentation (Phillips Reference Phillips2021), may address critiques of North–South epistemic power divides and may ensure that the topical interests of field experimenters more closely match local developmental priorities.

INSTITUTION AND TOPIC CLUSTERING AMONG EXPERIMENTAL RESEARCHERS

Mirroring geographical patterns, the skewed distribution of knowledge production is not unique to field experiments because it affects most academic and policy research. Nevertheless, field experimentation stands out in this respect because of the high costs and complex logistical requirements compared to other methods of inquiry. This increases the likelihood that the top producers are influential institutions and authors who have access to resources as well as policy and research networks. Therefore, the concern is that the ensuing institutional and geographical imbalances are even more pronounced for field experiments. A comprehensive bibliometric review of 25 development journals (2000–2019) revealed that, indeed, even when controlling for an overwhelming majority of articles on Africa written by non-African authors (87%), the share of experimental research on an African country by an African author is 2.5 times lower than the share of the equivalent body of observational research (Panin Reference Panin2020). Similar scientometric studies of other world regions also highlighted the relative diversity of observational studies (Cansun and Arik Reference Cansun and Arik2018; Codato, Madeira, and Bittencourt Reference Codato, Madeira and Bittencourt2020). We expanded this analysis by examining the affiliation of the lead primary investigators among all preregistered experiments in political science.

Figure 3 (left panel) plots the share of experiments by institutions that have political science departments ranking among the world’s top 10, according to the Shanghai world rankings (Shanghai Ranking 2019). We found that, indeed, the pool of experimenters was highly concentrated institutionally, with the top 10 academic institutions accounting for 20% to 40% of all experiments between 2014 and 2019. There was a clear downward trend during this period, however, which suggests a diffusion of the methodology that may be driven by its strength in causal identification, decreasing costs, and economies of scale. ⁵

Figure 3

Clustering Around Pioneering Institutions

Share of preregistered experiments that include authors from the top 10 universities in political science according to the Shanghai world rankings of universities, and their numbers on the right. Note that eight of these 10 institutions have endowments greater than $10 billion dollars USD, placing them among the top 15 endowed universities in the world. All of them except one have endowments greater than $4 billion dollars USD.

We also found a relatively high author concentration around influential researchers and associated networks. Figure 4 shows that the share of experiments with pioneering authors (i.e., at least 10 experiments in the left panel and at least five experiments in the right panel) was in the same range as the share from top institutions. In our supplementary data on published work in leading political science journals, 71% of all field experiments had at least one author from a top-20 institution and 40% of all authors were associated with a top-10 program.

Figure 4

The Impact of Pioneering Researchers

Share of preregistered experiments by political scientists with at least 10 (five) experiments on the left-hand (right-hand) side.

Influence clustering entails both costs and benefits. Preexisting experimental infrastructure lowers transaction costs and generates economies of scale in knowledge production. Conversely, barriers to entry for a wider pool of researchers increase the overall costs of innovation. The availability of grants and other funding schemes that would privilege researchers from broader institutional circles and more systematic inclusion and crediting of Global South contributors could diversify the institutional pool in field experimentation. During the past two decades, pioneering policy labs and research networks (i.e., Abdul Latif Jameel Poverty Action Lab, Center for Effective Global Action, Working Group in African Political Economy, and Evidence in Governance and Politics) made substantial investments in fostering collaborations between researchers from the Global North and the Global South. Knowledge transfers and local capacity building in experimentation generate significant positive externalities that may further address epistemic inequalities.

The experimental topics also entail positional implications. We text-mined experiment titles across our registries to generate several theoretically informed and occasionally overlapping categories of field experiments in political science: electoral learning, governance and accountability, minority representation, and postconflict recovery. Figure 5 (left panel) shows that governance as a conceptual-umbrella category accounted for approximately 50% of preregistered experiments. We also coded a residual “policy-learning” category that is highly heterogeneous. Although our coding scheme is imperfect, the high share of this category may reflect broader disciplinary debates regarding the role of experimentation in addressing underlying theoretical mechanisms in political science (Humphreys and Weinstein Reference Humphreys and Weinstein2009) versus adopting a more technical “mindset of plumbers” advocated in economics (Duflo Reference Duflo2017). ⁶

Figure 5

Field Experiments by Topic and Partners

The left-hand plot breaks down the share of preregistered experiments by text-mined key words. The right-hand side plots the share of preregistered experiments with public and private partners, respectively.

Monitoring topical trends is important because each topic raises its own unique set of ethical and research-design issues; therefore, a “one-size-fits-all” normative formula may not be optimal. For instance, policy evaluations are characterized by less-regulated ethical norms than pure research, and any steps toward codifying these norms would be crucial. In the case of postconflict or poverty-alleviation interventions, the experimental design itself may increase these ethical stakes given that a control group of vulnerable recipients does not receive the treatment (Evans Reference Evans2021). This is not the case in field experimentation on voter turnout or informational treatments. Moreover, social scientists in other disciplines increasingly have begun to conduct field experiments on similar topics. Fieldwork consolidation to prevent excessive location clustering and participant fatigue may benefit from further formal or informal cross-disciplinary coordination mechanisms.

EXPERIMENTAL PARTNERS

For experimentation on political topics, the three key actors with stakes in the experimental design and implementation are the subjects/citizens, the researchers, and the implementers (i.e., research firms, donors, political parties, nongovernmental organizations, government agencies, and others) (Haas et al. Reference Haas, Haenschen, Kumar, Panagopoulos, Peyton, Ravanilla and Sierra-Arévalo2022). Of the total articles with field experiments published between 2000 and 2017 in three top political science journals, 62 % entailed a partnership (Levine Reference Levine, Druckman and Green2021). Figure 5 (right panel) shows the shares and evolution of partnership relationships over time. Political scientists rely more on partnerships with non-state partners than with governments, and the share of experiments with private partners increased during the period of our sample. We suspect that this is because of both practicality and topic-related circumspection.

On a practical level, private or nongovernmental organizations (NGOs) often are preferred as main implementing partners because they are more flexible and able to randomize, whereas many governments—for legal or political reasons—face difficulties when designing control groups and eliminating recipients from treatment. Moreover, governments traditionally have been more reluctant to engage with academic researchers. In development, field experimentation initially focused on working with NGOs and adopting a gradualist and practical approach with respect to partnerships, “…not by knocking at the minister’s door but by marginal successes that create credibility for the movement, gains for policy makers” (Duflo quoted in Ogden Reference Ogden2017, 25). In general, design adjustments tailored to the constraints of various non-state partners ranging from NGOs to political campaigns have proven fruitful for sustained researcher–implementation partner collaboration in political science (Green, Calfano, and Aronow Reference Green, Calfano and Aronow2014).

Topic-wise, in some cases and contexts, the study of sensitive political phenomena—to a larger extent than in the case of economic development—also precludes direct partnerships with governments or political leaders, which may explain the pattern in our data. The search for and use of evidence by decision makers often entail instrumental considerations. Circumspect awareness of this aspect is important for the relationship with the implementation partner. For instance, for conditional-cash-transfer experiments taking place in Africa, the researchers, in collaboration with international NGOs, often chose not to work with local politicians because of clientelistic concerns (Ouma Reference Ouma2020).

Partner selection is paramount for both normative questions—ethical parameters and project legitimacy in the field (Haas et al. Reference Haas, Haenschen, Kumar, Panagopoulos, Peyton, Ravanilla and Sierra-Arévalo2022; Ouma Reference Ouma2020), as well as for scientific outcomes. From a normative standpoint, if the government is the main partner in a field experiment that relies on cluster randomization, its “right to rule” in certain policy areas (e.g., public schools and public health clinics) alleviates some ethical concerns regarding the lack of informed consent (Evans Reference Evans2021). Scientifically, the interaction and basic trust between the community and the partner organization are essential for both experimental compliance and findings. There is evidence that the type of partner—either government or NGO—involved in the implementation of identical field experiments conducted on the same site could lead to divergent treatment effects (Allcott Reference Allcott2015; Bold et al. Reference Bold, Kimenyi, Mwabu and Sandefur2018; Vivalt Reference Vivalt2020). For example, a study of educational reforms in Kenya by Bold et al. (Reference Bold, Kimenyi, Mwabu and Sandefur2018) found that the treatment significantly raised learning outcomes when implemented by an international NGO, whereas an identical intervention had no impact when implemented by the Kenyan government. The explicit acknowledgment of the political-economy stakes that partners have is a crucial positional aspect for both normative stances and scientific value.