Citizen participation in decision making has been widely lauded as a method for improving outcomes in democratic governance (Fung and Wright Reference Fung and Wright2001), environmental management (Koontz and Thomas Reference Koontz and Thomas2006; Reed Reference Reed2008), and international development (Mansuri and Rao Reference Mansuri and Rao2004; Mansuri and Rao Reference Mansuri and Rao2012). The World Bank alone has invested billions of dollars in the implementation of community-driven development, which emphasizes the participation of beneficiaries in decision making around development projects (Mansuri and Rao Reference Mansuri and Rao2004). Forms of direct democracy, such as participatory budgeting, have spread all over the world (Ganuza and Baiocchi Reference Ganuza and Baiocchi2012; Goldfrank Reference Goldfrank2012) from its origins in South America to nearly every global region, including East Asia (Hong and Cho Reference Hong and Cho2018), South-East Asia (Grillos Reference Grillos2017), Africa (Wampler and Touchton Reference Wampler and Touchton2017), Europe (Sintomer et al. Reference Sintomer, Herzberg and Röcke2008) and North America (Lerner and Secondo Reference Lerner and Secondo2012). Public participation is now formally encouraged in several national constitutions.
While there are compelling normative reasons to encourage more inclusive forms of decision making independent of results, it is important to recognize that participation also imposes costs on participants. These costs may be particularly burdensome to the poor, who are already constrained by both time and material resources. Many scholars point to positive benefits of participation relative to none, particularly with respect to the resolution of collective action problems, such as environmental resource management (Agrawal Reference Agrawal2005; Brooks et al. Reference Brooks2012; Ostrom Reference Ostrom1990). However, there are also instances where participation has failed to yield anticipated results on cooperation (Lubell Reference Lubell2004). Complicating matters, forms of participation vary greatly in practice, differing along several key dimensions (Fung Reference Fung2006). Some forms of participation are more costly and time-intensive than others, and so the particular design of participatory institutions should ideally be justified with demonstrated benefits of one form over another. A crucial open question in this line of literature is: Which particular forms of participation improve outcomes and through what mechanisms?
Political theorists extol the virtues of a particular form of citizen participation: deliberation (Dryzek Reference Dryzek1994; Habermas Reference Habermas1989; Habermas Reference Habermas1996; Manin Reference Manin1987). In addition to (and mutually reinforcing of) its normative benefits related to democratic principles of inclusion, deliberation, it is argued, serves at least two other core functions: an epistemic function by improving informational quality and leading to better decisions; and an ethical function by promoting mutual respect (Mansbridge et al. Reference Mansbridge, Parkinson and Mansbridge2012). The claim that deliberation may, in fact, make “better citizens” (Mansbridge Reference Mansbridge1999) is closely related to these epistemic and ethical functions, with the specific empirical implications being that deliberation may lead citizens to: (1) make better decisions for the collective; and (2) engage in more pro-social behavior.
Yet, empirical studies of deliberation have largely shied away from an explicit examination of outcomes, instead focusing on procedural factors (Landemore Reference Landemore2017). Scholars have called for the elaboration and testing of specific, falsifiable hypotheses that follow from deliberative theory (Mutz Reference Mutz2008), and the literature on deliberative democracy has recently begun to embrace the experimental method but has thus far provided limited exploration of effects on either decision quality or behavior change. Empirical work has established that deliberation results in shifts of opinion (Barabas Reference Barabas2004; Farrar et al. Reference Farrar, Fishkin, Green, Luskin and Paluck2010; Fishkin and Luskin Reference Fishkin and Luskin2005; Luskin, Fishkin, and Jowell Reference Luskin, Fishkin and Jowell2002) but has had very little to say about whether changes produced by deliberation are actually “for the better” (Neblo Reference Neblo2007).
When decision making concerns the allocation of resources to maximize some collectively valued outcome (as opposed to choosing between different possible goals), the outcome can be objectively assessed as being in the public interest or not. Such decision processes are quite common to the use of deliberation in the international development context, for example, in community-driven development and some forms of participatory budgeting. For example, a development agency may ask a community to decide not whether, but rather how, to invest in improving their water source. However, empirical work on deliberation has largely focused on developed-country contexts, despite the fact that its role in the design of participatory decision-making processes is of great relevance to international development practitioners (Heller and Rao Reference Heller and Rao2015).
This study provides a rigorous experimental test of the hypothesis that participation in deliberative decision making improves collective outcomes, and it distinguishes between decision quality and behavior change as drivers of those outcomes. Using random assignment to different decision-making processes in a controlled laboratory setting in a developing-country context, I assess whether participation in collective decision making leads to socially optimal decisions and/or behavior. I specifically isolate the use of deliberative argumentation, as defined by deliberative theorists, as opposed to another commonly used form of collective decision making: preference aggregation through majority-rule voting procedures.
I find strong experimental evidence in support of the epistemic benefits of deliberation: participation in deliberative decision making leads to better decision quality, that is, choices that are more in line with the socially optimal outcome for the group. Perhaps more importantly, these benefits are not achieved when participation involves only majority-rule voting. This suggests that participatory processes intending to incorporate local knowledge cannot do so effectively with simplistic forms of preference aggregation. Rather, the persuasion and exchange of knowledge achieved through more intensive deliberative institutions may be worth the additional cost.
Evidence is weaker in support of related behavior change (investment of effort on behalf of collective outcomes). However, I find that deliberation causes changes in preferences and that those who engage in deliberation are more likely to perceive the process as fair. While there is no support for a general effect on behavior change in my primary analysis, I offer some preliminary, suggestive evidence that behavior change may occur in certain cases, specifically, for those individuals whose preferences have been changed through the process, and I recommend additional research to explore this further.
Related Literature
Despite long-standing claims about the beneficial effects of participation, the empirical evidence in support of participatory decision making is inconsistent (Birnbaum Reference Birnbaum2016; Duit and Hall Reference Duit and Hall2014; Koontz and Thomas Reference Koontz and Thomas2006). Efforts to systematize the findings are complicated by the myriad forms that participatory institutions take in practice. There is a large and growing body of literature examining participatory institutions in developing countries (Grillos Reference Grillos2017; Mayka Reference Mayka2019; Rich, Mayka, and Montero Reference Rich, Mayka and Montero2019; Touchton, Sugiyama, and Wampler Reference Touchton, Sugiyama and Wampler2017; Touchton & Wampler Reference Touchton and Wampler2014; Wampler Reference Wampler2004). However, the literatures on participatory institutions and local collective action in developing countries have only recently begun to engage directly with political theory on deliberative democracy (Heller and Rao Reference Heller and Rao2015). Meanwhile, experimental work on deliberation has rarely been conducted in developing countries, despite concerns that lessons from experimental research in Western democracies may not generalize (Henrich, Heine, and Norenzayan Reference Henrich, Heine and Norenzayan2010).
One of the central elements of deliberation is that it involves “reason-based decision-making,” in which participants try to persuade each other of a course of action using reasons that appeal to others, such as fairness, group-mindedness, or logic regarding effectiveness (Fung and Wright Reference Fung and Wright2003; Gutman and Thompson Reference Gutman and Thompson2004; Neblo Reference Neblo2005; Thompson Reference Thompson2008). Deliberative discussion is believed to be more transformative than a mere aggregation of individual preferences (Chambers Reference Chambers2003; Elster Reference Elster, Elster and Hylland1986), and it has the potential to lead to better decision making and to more pro-social behavior.
There has been an “empirical turn” in deliberative democracy (Bächtiger et al. Reference Bächtiger2010; Carpini et al. Reference Carpini2004; Ryfe Reference Ryfe2005). The “science of deliberation” (Dryzek et al. Reference Dryzek2019) has now provided convincing evidence that people are willing (Esterling, Neblo and Lazer Reference Esterling, Neblo and Lazer2011; Neblo, Esterling, and Lazer Reference Neblo, Esterling and Lazer2018) and able (Gerber et al. Reference Gerber2018) to engage in quality deliberations, and that deliberative encounters increase political discussion beyond the formal event (Lazer et al. Reference Lazer2015). Scholars have demonstrated that deliberation can result in shifts of opinion (Barabas Reference Barabas2004; Farrar et al. Reference Farrar, Fishkin, Green, Luskin and Paluck2010; Fishkin and Luskin Reference Fishkin and Luskin2005; Luskin, Fishkin, and Jowell Reference Luskin, Fishkin and Jowell2002) and successfully avoid the risk of polarization (Esterling, Fung and Lee Reference Esterling, Fung and Lee2021). Observational data suggest that deliberation leads to decisions that are more consistent with an individual's underlying values (Niemeyer Reference Niemeyer2011) and more rational, in the sense of increased single-peakedness (List et al. Reference List2012).
However, we still have very little causal evidence about whether or not deliberative argumentation improves decision quality (Landemore Reference Landemore2017; Neblo Reference Neblo2007). This is due, in part, to a reluctance on the part of some deliberative democrats to embrace the existence of an objective truth in political disagreements (Rawls Reference Rawls, Copp, Hampton and Roeme1993). Others have counter-argued that work on deliberation must acknowledge and test its epistemic benefits for truth seeking (Cohen Reference Cohen2009; Estlund Reference Estlund1998; Landemore Reference Landemore2017).
Deliberation may improve decision quality through several pathways. First, it may result in more socially oriented decision making through constrained self-interest (Ackerman and Fishkin Reference Ackerman and Fishkin2002; Mansbridge et al. Reference Mansbridge2010). Social pressure may make it difficult to rely on purely self-regarding arguments during deliberation. Individuals may yield to the “forceless force” of the better argument (Habermas Reference Habermas1975; Habermas Reference Habermas1984), leading to group decisions that are more in line with the collective good even if competing individual preferences remain intact. Secondly, deliberation may improve decisions by correcting information asymmetries. Deliberative processes may allow individuals to gain new information that causes them to update pre-existing beliefs or gain new perspectives (Caluwaerts and Ugarriza Reference Caluwaerts and Ugarriza2012; Martí Reference Martí2006). This could lead them to value the decision outcomes differently, even if their underlying preferences have not changed. Finally, the deliberative process may change the decision criteria through which people translate preferences and beliefs into a decision, for example, by helping participants to overcome cognitive biases and acknowledge previously missed logical implications of existing knowledge (Hafer and Landa Reference Hafer and Landa2007; Landa Reference Landa2015).
There is also reason to expect that deliberation may lead to more pro-social behavior, such as investments of effort toward the achievement of collective outcomes. Work on procedural utility has confirmed that individuals value not only outcomes, but also the processes that lead to them (Frey, Benz, and Stutzer Reference Frey, Benz and Stutzer2004), and may value the same outcome more if they participate in creating it (Norton, Mochon, and Ariely Reference Norton, Mochon and Ariely2012). Therefore, they may also be more likely to invest in, maintain, or comply with those outcomes over the long run. The procedural justice literature argues that people are more willing to behave in compliance with a decision if they believe it was fairly determined (Thibaut & Walker Reference Thibaut and Walker1975; Lind and Tyler Reference Lind and Tyler1988; Tyler, Reference Tyler1990; Tyler and Blader, Reference Tyler and Blader2000)—even if their personal preference over the outcome has not changed. Recent work has shown that deliberation leads to an increase in self-stated willingness to pay for environmental public goods (Wang, Fishkin, and Luskin Reference Wang, Fishkin and Luskin2020), though behavioral measures of pro-social behavior are still hard to come by. While simpler forms of participation, such as majority-rule voting procedures, may be sufficient to activate these mechanisms, one might expect that more intensive forms of engagement, such as deliberative discussion, would be more effective at doing so. Furthermore, if self-interest is truly constrained during the decision-making process, as argued earlier, then this constraint could also be internalized as a norm and thus nudge participants toward more pro-social behavior in the future as well.
There are several major impediments to establishing these causal relationships with observational data. First, complex variation in program design makes it very difficult to isolate particular dimensions of participation (and related causal mechanisms) that could be driving observed effects. Secondly, there is self-selection in both the creation of participatory institutions by policymakers and engagement in participatory processes by citizens, which creates concerns over reverse causality. Finally, socially desirable decisions and behavior—while theoretically reasonable as hypothesized outcomes of participation and deliberation—are nearly impossible to define in many real-world deliberative situations without imposing the values of the researcher. This area of research could thus benefit from the controlled variation offered by experimental research (Falk and Heckman Reference Falk and Heckman2009).
Laboratory experiments have contributed greatly to the study of socially beneficial behavior in the form of cooperation for collective action (Ostrom Reference Ostrom2005; Poteete, Janssen, and Ostrom, Reference Poteete, Janssen and Ostrom2009). One of the most consistent findings in this body of literature is that face-to-face communication Q28 increases the likelihood of cooperative behaviors (Bornstein Reference Bornstein1992; Dawes, van de Kragt, and Orbell, Reference Dawes, van de Kragt, Orbell and Mansbridge1990; Ledyard Reference Ledyard, Kagel and Roth1995; Ostrom Reference Ostrom1998; Ostrom, Gardner, and Walker Reference Ostrom, Gardner and Walker1994; Sally Reference Sally1995). However, the effect of communication varies with contextual factors (Cardenas Reference Cardenas2004) and with the content of the communication (Lopez and Villamayor-Tomas Reference Lopez and Villamayor-Tomas2017), and researchers still lack a complete understanding of how exactly communication improves cooperation. Some prior experimental work has examined the effects of: differing democratic procedures on decision acceptance (Morrell Reference Morrell1999); group decision-making processes on cooperation (Grossman and Baldassarri Reference Grossman and Baldassarri2012; Hamman, Weber, and Woon Reference Hamman, Weber and Woon2011; Olken Reference Olken2010); deliberation on altruism (Sulkin and Simon Reference Sulkin and Simon2001); decision rule on gender inequality (Karpowitz, Mendelberg, and Shaker Reference Karpowitz, Mendelberg and Shaker2012); and participation on psychological ownership (Aga, Noorderhaven, and Vallejo Reference Aga, Noorderhaven and Vallejo2017). To the best of my knowledge, no prior studies have looked at both decision quality and behavior change in an experimental context that resembles common uses in developing-country contexts.
Research Methods
I take advantage of a controlled laboratory setting to assess differences between commonly used approaches to group decision making. This experiment involved a randomly assigned decision-making procedure, which, in turn, determined which of three effort tasks participants would complete. Individual performance on the effort task earned money toward a collectively shared outcome: a team fund that was then divided evenly across all team members, irrespective of individual performance. I differentiate between groups that rely on a simple majority-rule voting procedure and groups that use deliberative discussion resulting in a consensus-based decision (a more intensive and time-consuming form of decision making, but one perhaps more likely to influence outcomes).
An important advantage of laboratory research is that I can move beyond self-reported intentions to measure actual behavior within an incentivized behavioral game. In addition, the quality of decisions, which are difficult to specify in real-world situations without imposing the values of the researcher, can be more easily assessed in a controlled laboratory setting where the socially optimal decision is easily calculated.
Laboratory experiments have been criticized for lacking generalizability across cultures and contexts (Henrich, Heine, and Norenzayan Reference Henrich, Heine and Norenzayan2010; Levitt and List Reference Levitt and List2007). Others argue that these concerns are overstated, as many common laboratory findings are, indeed, replicable across cultures (Klein et al. Reference Klein2018), and that generalizability is a problem common to all research methods, not just lab experiments (Falk and Heckman Reference Falk and Heckman2009). Furthermore, the “realism” of an experimental context should be judged not by the context, but rather by how well the experimental design approximates the real-world experience it is meant to simulate (Falk and Heckman Reference Falk and Heckman2009).
Deliberative scholars have argued that experiments on deliberation can be considered valid to the extent that they involve representation of varied viewpoints on a perspective of public concern (Caluwaerts and Ugarriza Reference Caluwaerts and Ugarriza2012). My experiment meets these conditions by simulating decisions related to shared collective outcomes on which there is no pre-treatment consensus. It mimics the experience of groups of individuals coming together to decide how to allocate scarce resources (in this case, their time and effort) toward the achievement of collective outcomes. This closely resembles common forms of participatory decision making that are promoted in international development, such as community-driven development and some forms of participatory budgeting. It is also designed in a way to allow for distinguishing between decision quality and behavior change as drivers of improved outcomes. I further mitigate concerns by recruiting subjects from a relevant, developing-country context.
Study Setting
The experiment was conducted at Busara Behavioral Lab in Nairobi, Kenya. Nairobi was chosen as the site of this research because it is located in a country where participatory institutions are currently being designed, as well as because its people face scarcity of resources and would thus reasonably be concerned about the additional demands placed upon them by intensive forms of group decision making.
Kenya ratified a new constitution in 2010 by popular referendum. The new constitution includes requirements for citizen participation in government decision making, but the specific method of public engagement was left largely to the discretion of the newly formed county governments, many of which were still struggling to develop a public participation plan at the time key informant interviews with county officials were conducted in 2014 (Grillos Reference Grillos2018). The World Bank has provided support for the implementation of participatory budgeting, which typically takes place at the ward level and relies on consensus-based decision making (Wampler and Touchton Reference Wampler and Touchton2017).
Kenya is also a setting in which the allocation of scarce resources to achieve collectively valued outcomes is extremely salient. In Kenya, community fundraisers known as harambee are a common form of local collective action to provide local public goods and services (Miguel and Gugerty Reference Miguel and Gugerty2005; Wilson Reference Wilson1992). This degree of community responsibility is fairly common throughout the developing world, where governments are often minimally responsive to marginalized communities (see, for example, Habyarimana et al.'s [2009] discussion of local public good provision in neighboring Uganda).
The experimental protocol was conducted using z-Tree software (Fischbacher Reference Fischbacher2007), with the exception of the decision-making treatments. For the decision process itself, each team met separately in a smaller room outside of the computer lab, with facilitators following a protocol preprogrammed into Qualtrics. Busara staff (Kenyan citizens) implemented the experimental protocol in Swahili, the national language. All sessions occurred between July 27 and August 23, 2018.
Participants
Busara draws its research subjects primarily from the Kibera slum, a low-income population for whom local collective action is common. This experiment included 570 participants, spread across four treatments and one control group. A total of 63 per cent of the participants were female, 35 per cent had never been married, and 74 per cent reported having engaged in some sort of real-world collective action in their communities within the past month. The participants were, on average, 34 years old with two children and with ten years of education (the equivalent of having completed some secondary school). (For a full table of descriptive statistics, see Online Appendix A.) A larger percentage of participants were assigned to the control group in order to increase statistical power. Of the 570 total participants, 210 (~37 per cent) were assigned to the control group and the remainder were split evenly across the four treatment groups, with 90 participants in each of the four treatments. (For a discussion of statistical power calculations, see Online Appendix B.)
Experimental Design
Participants were first randomly assigned to a session, which was randomly assigned to a treatment group. On the day of the experiment, participants engaged in a collective decision-making process within teams of five to determine which of several real-effort tasks they would participate in to earn money toward a shared team fund. The form that this decision-making process took depended on the treatment group assignment. There were four overlapping treatment groups and a control group. Depending on the randomly assigned treatment group, the decision was made through either a private or public vote and using either a majority-rule voting procedure or through deliberative discussion requiring full consensus. In the control group, the decision was made via random assignment—in other words, the team members did not make any decision.
The experiment took place in several stages, as summarized in Figure 1 and in the following text. The full experimental protocol documents are included in the Online Appendix.
Steps in the experimental design.
Step 1: practice rounds and pre-treatment survey
Prior to treatment, I captured some information about the real-world activities of participants. Next, all participants were given an opportunity to briefly test out each of three effort tasks: the letter-counting task, the sliders task, and the Stroop task. All are adaptations of previously vetted real-effort tasks. In the letter-counting task, participants are shown a string of letters and numbers and asked to count the number of times a particular letter appears in the sequence (adapted from Rey-Biel, Sheremeta, and Uler Reference Rey-Biel, Sheremeta and Uler2011). In the sliders task, participants are given a target number between 0 and 100 and asked to move an on-screen slider to that number (adapted from Gill and Prowse Reference Gill and Prowse2012). In the Stroop task, an arrow appears on the screen and participants must tap the side of the screen that the arrow points to or the side of the screen that the arrow points from, depending on the color of the arrow (adapted from Stroop Reference Stroop1935).
The practice rounds were incentivized based on individual performance on the task, which I use as a measure of task-specific ability in the later analysis. Participants were told that they would later be given an opportunity to participate in one of these activities in order to earn money as a team. They were then asked to fill out a survey, which asked them to rate the three effort tasks according to enjoyment, difficulty, and overall preference.
Step 2: decision-making treatments
Individuals were randomly assigned to teams of five participants each. Each team then engaged in one of five decision processes (four treatments and a control) that would determine which of three effort tasks their team would work on. In the following, I describe the four treatment groups, which were determined by two overlapping treatment variations: (1) public versus private voting procedures; and (2) non-deliberative majority-rule voting versus deliberative consensus decision-making procedures.
In all sessions, teams met face to face in a separate room and had an opportunity to introduce themselves. Facilitators then explained how the team effort task would work (participants would earn points for a shared team fund through performance on one of three effort tasks). In the control group, the task was then selected using a random number generator in Qualtrics, the team was informed of the selection, and they returned to the computer lab to play. In the decision-making treatments, in contrast, the team was told that they would select the task and the decision rule was explained.
In the deliberative consensus groups, each participant was asked to state their preference and give the reason why they preferred that task. This was done specifically to ensure that some reason-based argumentation took place, an element that deliberative theorists emphasize as central to what defines “deliberation” as opposed to mere discussion. Then, team members were given time to discuss the options in order to arrive at a consensus decision. Once the team believed they had arrived at a decision, the consensus was then confirmed via a vote. If the vote revealed that there was not yet a full consensus, the team was asked to repeat the deliberation until they felt they could all agree on a single task.
In the non-deliberative majority-rule groups, in contrast, individuals were simply asked to vote for their preferred outcome. The facilitator directed the groups to vote immediately after explaining the team effort task and the choice between the three tasks, and the teams were not encouraged to discuss anything prior to the vote. In the case of a tie, the task with the fewest votes was removed as an option, and the team was asked to engage in a second round of voting to break the tie. The treatment groups also varied according to whether vote tallying took place through secret ballot or a public show-of-hands procedure.
Step 3: team effort task
Teams then had an opportunity to earn more compensation by performing the task selected in Step 2. Participants were invited to complete as many iterations of the activity as they could within 10 minutes, and they would earn 5 ShillingsFootnote 1 per point earned (which would go to a team pot). Before the task began, we also asked participants to guess how well their team would perform on the task—a measure of expectations regarding the contributions of others. At the end of the activity, the money earned by the team would go toward a collectively shared outcome, a team fund to be evenly split across its members. This portion of the experiment resembled a public good game, in which individual task performance earned money toward a collectively shared outcome (an aggregate team pot).
Step 4: post-treatment survey
After the activity was completed, participants were told how well they performed, how much money the team earned in total, and what their share of the winnings was. They were then asked to rate the enjoyment and difficulty of the task once again, and they were also asked questions to assess their perceptions regarding autonomy, fairness, agreement with the outcomes, willingness to work with the team again, and so on.
Analytic Methods
My main analyses use individual performance on the effort task (earnings toward the collectively valued outcome) as the primary dependent variable, with the decision-making treatments as the key explanatory variables. This main analysis takes the form of a linear model with robust standard errors clustered by team (the randomly assigned five-person team with which individuals shared their collective earnings). My main model also includes several pre-registered control variables that are known to influence pro-sociality.Footnote 2 After testing for differences between the coefficients of the public- and private-vote groups and finding no statistically significant differences between them, my final model collapses the treatments into only three groups: control, majority rule, and deliberation.Footnote 3
The variable task-specific ability serves primarily as a way to distinguish between effects through decision quality (choosing the task that the team was already collectively best at) and effects through behavior change (investing more effort toward collective outcomes, irrespective of ability). Task-specific ability is measured through the individually incentivized version of the effort task that was conducted during the pre-treatment survey. Since average ability on the tasks did not vary across treatment groups prior to the treatment, task-specific ability should only mediate outcomes through better task selection. This variable thus helps me to establish whether the effect has occurred through better decision quality, rather than through increased effort (behavior change).
As shown in the balance table in Online Appendix A, the individuals assigned to the various treatments were not statistically significantly different (α = 0.05) from those in the control group with respect to any of the pre-treatment demographic variables, nor with respect to pre-treatment preferences or ability on the tasks. The only differences that appear across groups emerge subsequent to the treatment (with respect to the decisions made as a team and the effort individuals exerted in the team task).
The experimental design and analytical approach described earlier was pre-registered through a pre-analysis plan (PAP) in the Evidence in Governance and Politics (EGAP) repository. The main analysis described in the PAP is that which is illustrated in Figure 2 and in Online Appendices C and D, which uses tasks completed as the main outcome variable, with treatment assignment as the key explanatory variables and ability as a key control to distinguish between behavior change and decision quality mechanisms. This was the primary motivation for the design of the study and can be considered confirmatory. All other analyses in this article should be viewed as more exploratory in nature. The PAP did make efforts to anticipate some of the exploratory analyses included here (including all those presented in Figure 4 and Online Appendix G), but they were clearly identified as exploratory within the language of the PAP itself. (For a thorough discussion of deviations between what appears in the PAP and what appears in the exploratory analyses presented in this article, which are viewed as conditional on the results of the main analysis, please see Online Appendix K.)
Treatment effects—decision quality versus behavior change.
To further explore the sub-mechanisms through which improved collective outcomes are achieved, I also analyze several (pre-registered) exploratory outcome variables, which serve as potential causal mediators. These additional variables include: (1) preference alignment; (2) preference change; (3) perceived fairness; and (4) acceptance of the team decision. In the pre- and post-treatment surveys, individuals were asked to name the task they would choose if given the option. The preference alignment variable indicates whether the initial individual task preference identified in the pre-treatment survey matches with the ultimate task selected by that individual's team. Preference change measures whether an individual who did not initially prefer their team's selection later changes their mind, selecting the team choice in the post-treatment survey. This suggests that they were persuaded to agree with their teammates about the task selection. In the post-treatment survey, participants were asked the following questions: “There were three tasks presented to you earlier, but only one was chosen for your team. How much did you agree with the final decision?”; and “How fair do you think the decision was to choose a task for your team?” Each question was scored using a five-point Likert scale. Responses to these questions are used to measure acceptance of decision and perceived fairness, respectively.
Results
Primary Analysis
Deliberation increases collective earnings through improved decision quality
Individual earnings on behalf of the team are highest in the deliberation treatment. On average, individuals in the deliberation treatment completed fifty tasks within the ten minutes allotted, which amounts to an additional five tasks (corresponding to 25 Shillings in additional earnings for their team) as compared with the control group. The majority-rule treatments, in contrast, complete only one additional task relative to the control group. Results from the main analysis suggest that deliberation results in better collective outcomes. The earnings for the deliberation teams are, on average, 11 per cent higher than the control group and 9 per cent higher than the majority-rule group. The difference between the coefficients for the deliberation treatment and majority-rule treatment is also statistically significant (p = 0.02).
This increase in earnings occurs through strategic decisions regarding task selection among the deliberation groups. When controlling for task-specific ability in the main model (see Online Appendix E), the effect on contributions disappears (though the coefficient is still positive). Figure 2 shows treatment effects on contributions (individual performance on the team effort task), ability (performance on the individually incentivized version of that same effort task), and effort (contributions, controlling for ability). No treatment effects for the majority-rule treatment are different from 0, meaning that outcomes were no different from those in the control group. Deliberation, on the other hand, has a statistically significant effect on contributions and task-specific ability but not on effort. This means that deliberation changed decision quality but not behavior.
“Ability” in these analyses is merely a proxy for decision quality. Individuals in the deliberation groups were not, by random chance, more skillful at any of the three tasks compared with people in other groups (see Online Appendix A). The effect of deliberation on ability can thus only have occurred through its epistemic function: it led to better decisions on task selection. The teams in the deliberation groups were more likely to strategically choose the socially optimal (most profitable) task for their particular team (that is, the task at which they were most skillful as a collective).
Figure 3 demonstrates this further. “Made best choice” is an additional (not pre-registered) team-level variable that measures whether a team made the best decision for its members. Here, “best decision” refers to the socially optimal decision—that for which they would have earned the most money based on their collective ability on each task. Being in one of the deliberation treatments is a statistically significant predictor of making the best team-level decision (see Online Appendix E). Regressions comparing only the two treatment groups (deliberation versus majority rule) reveal a similar pattern (see Online Appendix F). Deliberation outperforms majority rule to a statistically significant degree on total contributions, on ability (as a proxy for decision quality), and on my more direct measure of decision quality (whether the team chose the best task).
Decision quality (proportion of teams choosing most profitable task).
More than 65 per cent of deliberation treatment teams choose the best task for their team. As might be expected, when tasks were randomly assigned in the control group, around 30 per cent of teams happened to be assigned the task that they were best at. However, even in the majority-rule treatment groups, teams chose the task they were best at in fewer than 40 per cent of cases. Deliberation served the purpose of allowing teammates to share relevant information about ability and strategy, and therefore make better decisions than they would have through individual calculation.
Deliberation led teams to more frequently choose the most locally appropriate task for their team—the task that was most profitable for their own mix of abilities. Individuals in the deliberation treatment generate more contributions to the team fund, and this is mostly attributable to improved task selection (better decision making), as opposed to effort (behavior change). It is important to note here that this does not entirely rule out the possibility of an effect on behavior, but it suggests that such an effect, if it exists, is too small across the full sample to be statistically significant—specifically, less than 0.26 standard deviations (see Online Appendix B).
Exploratory Analyses
Deliberation increased preference change, perceived fairness, and acceptance of outcomes
Deliberation was also associated with higher preference alignment, preference change, perceived fairness, and agreement with outcomes, as compared with the control group. Figure 4 shows the treatment effects of both the majority-rule and deliberation treatments on each of these intermediate outcomes (for the regression output, see Online Appendix G).
Treatment effects—secondary outcomes.
As expected, in the control group, about one-third of individuals happened to be randomly assigned the task that was their initial preference. Both the majority-rule and deliberation treatments outperform the control on preference alignment. However, this effect is both slightly larger and more statistically significant in the majority-rule treatment than in the deliberation groups. In the deliberation treatment, about 48 per cent of individuals got their first choice of task, whereas in the majority-rule groups, just over 51 per cent of individuals got their way.
Despite more people in the majority-rule group getting the outcome they initially preferred, individuals in the majority-rule treatment actually exhibited lower levels of agreement with the decision that was made—even compared with the control group, where the task was assigned randomly. Individuals in the deliberation treatment, on the other hand, were more likely to say that they agreed with the decision that was made, and they were more likely to perceive the process as having been fair. The coefficients for both variables indicate an effect of about 0.3 points on a five-point Likert scale, or a shift of six percentage points relative to the control. In the majority-rule treatments, the answer to the fairness question was not significantly different from the control group. In the deliberation group, 85 per cent report that they “completely agree” with the decision, whereas in the majority-rule group, fewer than 60 per cent select this response. Similarly, with respect to fairness, 69 per cent of individuals in the deliberation group rate the process as “completely fair” compared with only 59 per cent of the majority-rule group.
I also find that those in the deliberation groups are more likely to change their self-reported individual preferences over the tasks after the treatment. Those who have experienced the deliberation treatment are significantly more likely to change their preference to match the choice taken by their team. In the deliberation group, 57 individuals (about 31 per cent) changed their preference in favor of the collective team decision. This proportion is actually quite striking when we consider that almost 50 per cent of them already preferred that task and thus could not possibly have shifted their preference in favor of it. In thirty of the thirty-six teams assigned to deliberation (83 per cent), at least one individual was persuaded to switch their preference to the task chosen by collective decision. In contrast, fewer than 20 per cent of individuals shifted their preference to the team choice in the control and majority-rule groups.
Suggestive Results
Preference change induced by deliberation may lead to increased effort
In a follow-up (not pre-registered) analysis to the results presented earlier, I also estimated the effect of each secondary variable on the final outcome (contributions), both with (see Online Appendix J) and without (see Online Appendix H) the crucial control for ability. (It should be recalled that the former proxies for decision quality and the latter proxies for effort.) Getting one's initial preference is strongly associated with greater contributions, while preference change is the only intermediate variable that is a predictor of effort. This result is statistically significant at α = 0.05, though not when using an alpha that has been adjusted for multiple hypothesis testing (see Online Appendix L). However, the coefficient on preference change is also larger in magnitude than the other intermediate variables, representing an increase of 3.2 tasks completed for individuals who have experienced a change in preferences. That reflects a 6.9 per cent increase in tasks completed relative to the average participant score. Those who were persuaded to prefer a task that they did not initially prefer performed better, on average, during the team task, even after accounting for task-specific ability. While this result cannot be considered conclusive with the present data, it suggests that deliberation may have an effect on behavior change, conditional on preference change.
Further exploring this correlation, I find that there is an indirect effect of deliberation on effort, mediated by preference change. In my main analysis presented earlier, there was no statistically significant effect of deliberation on behavior change (effort), only on performance through better collective decisions related to task ability. However, even though the direct effect of deliberation on effort could not be clearly established, there could still be a statistically significant indirect effect through an infrequent mediator (Baron and Kenny Reference Baron and Kenny1986; Zhao, Lynch, and Chen Reference Zhao, Lynch and Chen2010).
Causal mediation analysis (Hicks and Tingley Reference Hicks and Tingley2011) suggests that the average causal mediation effect of preference change on effort is 0.42 (with 95 per cent confidence interval: 0.03–1.00). This is statistically significant at conventional levels, even when the comparison group includes the majority-rule treatment. However, since the causal mediator is not randomly assigned, this portion of the analysis lacks the causal identification afforded by the experimental design. Causal mediation analysis requires much stronger assumptions than the primary experimental results that I presented earlier in the article. In addition, the statistical significance does not hold up to corrections for multiple hypothesis testing. Thus, I present this as a merely suggestive result, which points to a potential venue for future research. It suggests that deliberation may lead to changes in behavior, but only if and when it manages to change preferences. It raises a question as to whether individuals may put more effort into achieving collective outcomes if they have been persuaded by reason-based argumentation about the best way to achieve those outcomes.
I also used causal mediation analysis to look at the even stronger correlation between preference alignment and total contributions. Preference alignment has a statistically significant average causal mediation effect on contributions (ability) when the deliberation treatment is compared to the control group alone, but when the comparison group includes the majority-rule treatment, the statistical significance disappears. Thus, preference alignment does improve collective outcomes through somewhat increasing the match between individual ability and the task selected (relative to the control group), but it does not fully explain why the deliberation group outperforms the majority-rule treatment with respect to decision quality.
Discussion
How Does Deliberation Improve Decision Quality?
Some descriptive analyses can help us better understand the effect on decision quality. This effect is not driven by mere preference alignment. Individuals in the majority-rule treatment are more likely to get their initial preference, but they are less likely as a team to make the choice that would maximize collective contributions. Table 1 disaggregates teams in each treatment group by how well their initial individual preferences (based on the pre-treatment survey question) mapped onto the best (most profitable) choice for their team. Some teams already had a clear majority preference for the socially optimal choice (three or more individuals preferred it). In other teams, the best choice was tied for first place (with two individuals listing it as their preference, equal to a second task). Still other teams had a clear majority preference for a different task, that was not the best choice. For all three categories of initial preferences, the deliberation teams have a higher success rate in terms of arriving at the socially optimal choice through group decision making.
Decision success by initial preference category
Where the deliberation treatment especially outshines majority-rule voting procedures is where there is an initial preference for a suboptimal decision. Of 18 such teams in the majority-rule treatment, only four managed to arrive at an optimal solution.Footnote 4 Of 15 such teams in the deliberation treatment, more than half arrived at the socially optimal decision despite initially having a majority preference for a different task.
Since preference aggregation alone cannot explain the outcomes, improvements in decision quality could be driven by some combination of: (1) correction of information asymmetries; (2) a change to the decision criteria being applied; and (3) constrained self-interest.
Information asymmetries exist in this context because individuals have some information about their own ability with respect to each task but cannot know the ability of their teammates absent an explicit discussion with them. Participants are not very good at guessing where they stand relative to others in the room. After each practice round, individuals were asked to guess their rank on each task for a chance to earn a small cash bonus if they guessed correctly (± 1). Fewer than 30 per cent of participants earned that bonus for any given task. (With 15 people per session, a random guess should have yielded a 20 per cent chance of being correct.) Thus, deliberation could improve decisions by allowing for information about others' abilities to be shared. In fact, most of the reasons given during the reason-based argumentation related to how easy the task was.
However, sharing information about individual ability is not the full story, as what many participants perceive to be the “easiest” task is not the task that is, in fact, the most profitable. About 58 per cent of the participants list as their top preference the same task that they rate as being “easiest” in the pre-treatment survey. However, only about 50 per cent of participants rate as “easiest” the task that they actually performed best on (earned most money from) during the practice rounds.
It is possible that even those participants who had accurate information about their relative level of ability on each task were not basing their decision on that information prior to deliberation. They could, instead, have been persuaded through deliberation that this was the correct way to make the decision (altered decision criteria or correction of cognitive biases) without necessarily changing their underlying personal preference for a given task. While many arguments revolved around how easy the tasks were, not all of them did—some referred to how enjoyable or even “challenging” their preferred tasks were.
With regards to constrained self-interest and the so-called “forceless force of the better argument,” I asked our facilitators to observe the deliberative discussions and code them according to whether the arguments primarily appealed to the good of the group, mostly referred to personal preferences, or involved an equal mix of individual versus collective reason giving. Most groups used a mix of the two, but in the nine cases where deliberation teams overrode the initial majority preference to arrive at the socially optimal decision, more than half primarily used collectivist reasoning.
Does Deliberation Increase Pro-social Behavior Through Preference Change?
While the effects on behavior change in the general sample are not statistically significant, this should not be conflated with a null result. I do not have strong evidence in support of behavior change, but I am cautious in interpreting this as counter-evidence to that hypothesis. It is possible that the effect I find here is a true result but just too small for conventional statistical significance.
Some specific features of the study context may have limited my ability to detect effects with respect to effort—meaning that this effect could be larger or less heterogeneous in a different setting. First, although I offer alternative activities (in the form of an activity sheet with a short story and puzzles), there is arguably very little opportunity cost to an individual's participation in the effort task. Since they have already planned to spend this time in the lab setting, the additional cost of participating in the activity may be negligible to them. Only about 2 per cent of participants contribute nothing at all to the team effort task and the majority of these also failed to complete any tasks during the individually compensated rounds.
Second, the study participants are, on average, a highly cooperative sample. More than 70 per cent report having been involved in some form of real-world collective action (participating in a community project or fundraiser event) within the past month. In a post-treatment measure of complementary effort—a standard voluntary contribution mechanism public good game—more than 70 per cent of participants contribute at least half of their endowment. If the majority of participants are already prone to exert their maximum effort, then that limits my ability to observe meaningful variation on effort across the treatment groups. This would imply that ability, not willingness to contribute, is indeed the main constraint on achieving collective outcomes in this context. However, this may actually be true of many communities in developing countries, where social capital is high but successful collective action remains limited due to resource constraints. If so, the finding that deliberation led to better strategic decision making (which minimized the costs of contribution) may have direct relevance for policy makers. Future research should aim to replicate this experimental design to the extent possible in a field setting in multiple contexts that vary based on pre-study predisposition to cooperation.
Since we only (weakly) see effects on behavior through preference change, not through perceptions of fairness, the evidence seems to be more supportive of procedural utility as a potential agent for pro-social behavioral change rather than procedural justice (though procedural justice may, of course, still play an important role in legitimacy more generally). Further supporting the procedural utility hypothesis is the fact that those whose general preference shifted toward the task their team selected are also more likely to rate that task as the “most enjoyable” in the post-treatment survey. Of those who shifted preferences, 88.46 per cent of them also rate that team task as the most enjoyable. (For comparison, 70.8 per cent of them rated the task as “easiest” in the post-treatment survey.) Of those preference-shifting participants who rate the task as most enjoyable in the post-treatment survey, only 22.6 per cent of them already believed the task to be the most enjoyable in the pre-treatment survey.
Conclusion
The experimental results demonstrate that individuals who engage in deliberative discussion involving reason-based argumentation achieve better collective outcomes, whereas majority-rule voting did not outperform external random assignment. This occurs primarily through better decision making. The deliberative teams make better strategic decisions regarding the allocation of resources toward achieving collectively valued outcomes (in this case, the selection of the task used to earn money for the team). There is also a potential second pathway through which deliberation improves outcomes: an indirect effect through preference change that leads some individuals to invest more effort into achieving those socially optimal outcomes (that is, work harder to earn money for their team).
The effects of deliberation on better decision-making do not occur through preference aggregation alone, lending some evidence to the notion that deliberation is more transformative than other forms of collective choice. At a bare minimum, deliberation allows for the exchange of useful information that individuals do not have access to in isolation. Beyond that, individuals genuinely seem to be persuaded by the arguments of their peers and come to change their views about the experience of performing the tasks themselves.
Regarding the other pathway to improved collective outcomes, that is, behavior change, the main analysis did not provide direct support for increased effort as a result of deliberation. However, I provide some suggestive evidence that deliberation may have an indirect effect on behavior through preference change. If they have been persuaded through reason-based argumentation to pursue a certain course of action, individuals may then be motivated to put more effort into that action. This suggests that involving people in deliberative decisions that affect their lives, irrespective of the decisions that are ultimately made, may change people's relationship to the decision outcomes. However, more research is needed to confirm this suggestive result.
The main results presented here have major policy implications, corroborating the hunch of many a grass-roots development practitioner. The study outcomes are very supportive of the use of deliberative processes in decisions concerning the achievement of shared collective goals. Recent research suggests that government policy does have substantial potential to influence the quality of deliberation (Sanyal and Rao Reference Sanyal and Rao2018), and so an emphasis on reason-based argumentation could yield concrete improvements in outcomes. However, those planning to design participatory decision processes such as these should still carefully weigh both the costs and benefits of the process in their design. This experiment validates the existence of hypothesized effects, but it cannot speak to whether the magnitude of those effects would provide a benefit outweighing the opportunity cost of participants' time in a more realistic field setting.
One contribution of this work is to bring the comparative public policy literature on participatory decision making into closer dialogue with democratic theory, as well as experimental work from political psychology and behavioral economics. I demonstrate here that deliberative discussion leads to decision making that is more than the sum of its parts. Collective rationality is perhaps less bounded than that of any one individual. In addition, there is experimental evidence for the notion that preferences are transformed through the process of deliberation and weaker, suggestive evidence that those who experience this transformation may indeed become “better citizens,” in the sense that they engage in more pro-social behavior. Future work should further explore this more nuanced version of the behavior change hypothesis.
Supplementary Material
Online appendices are available at: https://doi.org/10.1017/S0007123421000375
Data Availability Statement
Replication data for this article can be found in Harvard Dataverse at: https://doi.org/10.7910/DVN/HT35YO
Acknowledgments
I thank the staff at Busara Center for Behavioral Economics for their support and feedback in preparing the experiment for implementation, in particular, Jennifer Adhiambo, Mary Chebet, Joseph Kimani, Kelvin Kihindas, and Chaning Jang. For input into the research design, analytical methods, and theoretical/conceptual focus, I acknowledge Molly Scudder, Bill Clark, Jay McCann, Mike Touchton, Brian Wampler, David Yu, and Richard Zeckhauser. For helpful comments on previous drafts of this article, I thank participants from conferences of the American Political Science Association (APSA), the Association for Public Policy Analysis & Management (APPAM), the Southern Political Science Association (SPSA), and the International Association for the Study of the Commons (IASC), as well as the New York University Center for Experimental Social Science's Conference on Experimental Political Science, the Workshop on the Ostrom Workshop, the Earth Systems Governance Conference, and the Political Science Research Workshop at Purdue University. I am particularly grateful for discussant comments from Brigitte Seim, Dmitri Landa, and Jon Woon, as well as three anonymous reviewers.
Financial Support
This experiment was implemented thanks to financial support from the Exploratory Research in the Social Sciences Grant from Purdue University's College of Liberal Arts, and early work on this research was supported by the Harvard Law School Project on Negotiations' Graduate Student Grants Program.
Competing Interests
None.
Ethical Standards
Research at Busara Center for Behavioral Economics is conducted with IRB approval from both Princeton University in the United States and Maseno University in Kenya. The Purdue University IRB opted to defer to these institutions in approving this research. Beyond IRB approval, this plan of research was discussed with local staff at Busara before finalizing the design in order to ensure the local appropriateness of participant activities, and payments were set to average local daily wages, in order to ensure that they were high enough to compensate participants fairly for their time but not high enough to be considered coercive. Generally speaking, this research involved low-risk activities such as answering non-sensitive questions about recent social/civic engagement, playing short computer games, and discussing these games with fellow participants.
