Introduction
Citizens in established democracies have high levels of support for democracy, but much lower levels of support for political parties and institutions (Dalton, Reference Dalton2004; Norris, Reference Norris2011). Indeed, many citizens experience a disparity between their ideal form of democracy and how democracy works in practice. As a fix for this widespread disillusionment with representative democracy, analysts have suggested that citizens may require more and deeper participation in the democratic decision-making process (Dryzek et al., Reference Dryzek, Bächtiger, Chambers, Cohen, Druckman, Felicetti, Fishkin, Farrell, Fung, Gutmann, Landemore, Mansbridge, Marien, Neblo, Niemeyer, Setälä, Slothuus, Suiter, Thompson and Warren2019; Elstub & Escobar, Reference Elstub and Escobar2019; Mansbridge, Reference Mansbridge, Elkin and Soltan1999; Smith, Reference Smith2009).
In response, democratic theorists have envisioned several new ways in which citizens can be involved in democratic decision-making. These new forms of citizen participation, often called democratic innovations, are “[p]rocesses or institutions that are new to a policy issue, policy role, or level of governance, and developed to reimagine and deepen the role of citizens in governance processes by increasing opportunities for participation, deliberation and influence” (Elstub & Escobar, Reference Elstub and Escobar2019, p. 14). Proponents of mini-publics focus on the exchange of reasoned arguments and promoting considered judgement amongst participating citizens (Gutmann & Thompson, Reference Gutmann and Thompson2004; Smith, Reference Smith2009). Proponents of participatory processes underline the educative benefits and behavioural effects of active participation for citizens (Barber, Reference Barber2003; Pateman, Reference Pateman2012). Proponents of referendums and citizens’ initiatives emphasise the importance of including all citizens in decision-making processes in order to come to more legitimate decisions (Altman, Reference Altman2010; Saward, Reference Saward2001). All these democratic innovations reconceptualise the role of citizens in democracy and assume a variety of positive effects on participating citizens. The question remains, however, to what extent these expected positive effects also materialise in practice.
A growing amount of research examines the effects of democratic innovations on participants (Jacquet & van der Does, Reference Jacquet and van der Does2020; Michels, Reference Michels2011). Yet, while there is a substantial number of quantitative studies on the effects of democratic innovations on participants, systematic reviews aggregating the results of individual studies are scarce. A few reviews do exist that bring together insights from qualitative and quantitative research (Boulianne, Reference Boulianne2019; Gastil, Reference Gastil2018; Michels, Reference Michels2011; Smith & Tolbert, Reference Smith and Tolbert2007; van der Does & Jacquet, Reference van der Does and Jacquet2021). However, while these studies offer valuable insights into the phenomenon and suggest that democratic innovations indeed have effects on participants, most studies either focus on one type of democratic innovation or focus on a specific sub-set of effects on citizens. Hence, some important lacunae still exist.
First, we lack a systematic and robust appraisal of the effects of different types of democratic innovations. Most reviews zoom in on one innovation in particular, be it mini-publics (Boulianne, Reference Boulianne2019; Gastil, Reference Gastil2018; van der Does & Jacquet, Reference van der Does and Jacquet2021), or forms of direct democracy (Smith & Tolbert, Reference Smith and Tolbert2007). From this perspective, the study by Michels (Reference Michels2011) gives us most insight in the effects of different democratic innovations.Footnote 1
Second, while theories on deliberative, participatory and direct democracy expect democratic innovations to affect participants in many different ways, shaping their attitudes, behaviour and capabilities, several existing review studies focus on a subset of effects. As such, Boulianne (Reference Boulianne2019) investigates the effects on political trust and efficacy, while Michels (Reference Michels2011) assesses the effects on participants’ policy attitudes and capabilities. The studies that do focus on a wider range of effects (Gastil, Reference Gastil2018; Smith & Tolbert, Reference Smith and Tolbert2007; van der Does & Jacquet, Reference van der Does and Jacquet2021), are – as noted above – limited by focusing on only one type of democratic innovation.
Third, none of the existing reviews statistically evaluates the effects of the different democratic innovations across studies using meta-analytical techniques.
The contributions of this paper are therefore twofold. First, the paper is the first to systematically study the effect of different types of democratic innovations on participants’ attitudes, behaviour and capabilities. In doing so, it provides an overview of the effects that have been studied for each type of democratic innovation, and thereby lays bare gaps in the empirical literature. Second, this study is the first to conduct a statistical meta-analysis, which allows us to estimate the robustness of the effects of democratic innovations on participants across studies. The meta-analysis includes 100 quantitative studies on the effects of democratic innovations on participating citizens between 1980 and 2020. To do so, we collected an exceptionally large dataset (Ntests = 1,621; Nstudies = 100; Ndatasets = 110), which allows for a systematic analysis of data from 37,974 respondents spread over 22 years and 16 countries.
Based on our analyses, we answer the following two research questions: (1) How thoroughly has each type of democratic innovation and each type of effect on participants been studied? And, most importantly, (2) to what extent (if at all) do democratic innovations have an effect on participants’ attitudes, behaviour and capabilities?
We find first, perhaps unsurprisingly, that the effects of mini-publics on participating citizens have been most extensively studied, while the effects of participation in participatory processes and referendums and citizens’ initiatives, relatively, have been studied less. Second, we find empirical support that participation in mini-publics changes citizens’ policy attitudes and has a positive effect on citizens’ political attitudes, knowledge, internal efficacy and reasoning skills. We find limited evidence regarding mini-publics’ effect on civic engagement. Our analysis of the limited number of available and suitable studies seems to suggest that participation in participatory processes appears to have a positive effect on participants’ political attitudes and knowledge and no effect on their internal efficacy, while referendums and citizens’ initiatives appear to increase citizens’ knowledge and internal efficacy. It needs to be stressed, though, that this by no means proves the (absence of) effects given the limited number of studies into participatory processes and referendums and citizens’ initiatives.
The next section describes the three types of democratic innovations and the types of effects on participants that this meta-analysis focuses on. In section 3, we subsequently outline our research design, data collection and the methods used for our meta-analysis. In section 4 we discuss the potential effects of participation in each type of democratic innovation and present our results for those innovations. Finally, in the conclusion we reflect on our findings and provide suggestions for future research to address important remaining research gaps regarding the effects of democratic innovations on participating citizens.
Potential effects of democratic innovations on attitudes, behaviour and capabilities
Over the past decades, different types of democratic innovations have been developed that build on theories of deliberative democracy, participatory democracy and direct democracy (Dryzek et al., Reference Dryzek, Bächtiger, Chambers, Cohen, Druckman, Felicetti, Fishkin, Farrell, Fung, Gutmann, Landemore, Mansbridge, Marien, Neblo, Niemeyer, Setälä, Slothuus, Suiter, Thompson and Warren2019; Elstub & Escobar, Reference Elstub and Escobar2019; Smith, Reference Smith2009). While many different forms of democratic innovations exist, in this meta-analysis we zoom in on (1) mini-publics, (2) participatory processes and (3) referendums and citizens’ initiatives. These three types of democratic innovations were identified as important types by the seminal works of Smith (Reference Smith2009) and Elstub and Escobar (Reference Elstub and Escobar2019), and are also the most common types of democratic innovations to have been empirically researched.Footnote 2 As each of these three is rooted in a different theory or model of democracy (O'Flynn, Reference O'Flynn, Elstub and Escobar2019), they each have their own strengths and weaknesses in terms of the democratic problems they are potentially able to address and the effects they have on their participants (cf. Warren, Reference Warren2017).Footnote 3 Additionally, the effects on participants are central in theorising on mini-publics, whereas theories behind participatory processes and referendums and citizens’ initiatives also put great emphasis on their policy impact and impact on society at large, which are beyond the scope of this meta-review. The empirical studies included in this meta-analysis focus on three overarching types of effects on participants, namely (1) attitude change, (2) behavioural change, and (3) capability change. Each of these types are further divided into subtypes. Note that each of the theories of deliberative, participatory and direct democracy expect democratic innovations to affect participants in different ways, placing stronger relative emphasis on effects on attitudes, behaviour or capabilities.Footnote 4 In section 4.2 we outline these expected effects for each democratic innovation (mini-publics, participatory processes and referendums and citizen's initiatives) in more detail.
Changes in attitudes are defined as ‘changes in how individuals perceive and evaluate the world’ (Jacquet & van der Does, Reference Jacquet and van der Does2020, p. 4). Attitude change consists of policy attitude change – often also referred to as opinion change –, political attitudes change –entailing attitudes towards the wider political system, such as trust and external efficacy –, and process attitude change –attitudes towards the democratic innovation itself, such as support for participatory processes. Behavioural changes are defined as changes in ‘the ways in which [citizens] engage (or not) in various activities’ (Jacquet & van der Does, Reference Jacquet and van der Does2020, p. 4). These changes are subdivided into political participation – or ‘activities intended to influence actual political outcomes’ (Ekman & Amnå, Reference Ekman and Amnå2012, p. 287) – and civic engagement – meaning all activities that could indirectly influence political outcomes but occur mainly out of interest for societal issues (Ekman & Amnå, Reference Ekman and Amnå2012). Capability change refers to changes in ‘the extent to which [citizens] are able to articulate their ideas and pursue their objectives’ (Jacquet & van der Does, Reference Jacquet and van der Does2020, p. 4). This category is subdivided into knowledge, internal efficacy – the feeling that one as an individual citizen can contribute to achieving social and political change (Campbell et al., Reference Campbell, Gurin and Miller1954, p. 187) – and reasoning skills.
Data and methods
In this section, we start by describing the data collection and selection process, as well as the coding process. We then go on to explain how the dependent and independent variables have been measured. Lastly, we describe how the meta-analysis is conducted, how we deal with publication bias, and what other robustness checks have been carried out.
Data collection, selection and coding
Data collection and selection
We have composed a dataset that includes 100 studies and 110 datasets which report 1,621 unique estimates. These data were collected through a systematic search as prescribed by the PRISMA 2020 statement (Page et al., Reference Page, McKenzie, Bossuyt, Boutron, Hoffmann, Mulrow, Shamseer, Tetzlaff, Akl, Brennan, Chou, Glanville, Grimshaw, Hróbjartsson, Lalu, Li, Loder, Mayo‐Wilson, McDonald and Moher2021).Footnote 5 Our data stem from three sources: (1) database searches, (2) citation searches and (3) review articles. These searches yielded 2,468 potentially relevant records. After deleting duplicate files (N = 70), 2,398 records were screened for inclusion in the meta-analysis based on seven predefined inclusion and exclusion criteria (see online Appendix C for a more elaborate explanation of our inclusion criteria).Footnote 6 The 1,879 records that, based on their abstracts, did not adhere to the inclusion criteria were excluded. Two more records were excluded that could not be retrieved, while another 417 records were excluded after reading the full study. This process resulted in a final dataset of 100 studies adhering to all seven criteria. The PRISMA flowchart in Figure 1 summarises the full data collection and selection procedure. An overview of the included articles can be found in online Appendix B, and a detailed account of our data collection and selection process can be found in online Appendix C.Footnote 7
Data collection and selection process. Note. Figure adapted from PRISMA flow diagram.
Coding process
The 100 studies in the final dataset were consequently coded for their independent and dependent variables, the features of the democratic innovation (see Elstub & Escobar, Reference Elstub and Escobar2019; Karpowitz & Mendelberg, Reference Karpowitz, Mendelberg, Lupia, Greene, Kuklinski and Druckman2011), whether they applied a pre-test, whether they included a control group, their statistical technique, their observed effects, and several other descriptive codes.Footnote 8 In order to ensure internal consistency throughout the coding process, the researchers defined mutually exclusive and precise definitions based on the literature for each code. In case a question in coding arose, the researchers jointly refined the coding scheme.Footnote 9 After the first coding round, all codes were double-checked by one of the researchers. As an ultimate robustness check, two independent researchers double-coded 10% of the articles. The Fleiss Kappa statistic amounted to 0.779 [0.719; 0.838] in the first coding round, and increased to 0.843 [0.818; 0.868] in the final coding round, which indicates a very good final strength of agreement (Laerd Statistics, 2019).Footnote 10
Description of variables
Dependent variables – Citizens’ attitudes, behaviour and capabilities
The dependent variables used in this study are changes in attitudes, behaviour, and capabilities amongst citizens. We build on prevailing theoretical work on deliberative democracy, participatory democracy and direct democracy that presents theoretical expectations about the effects of democratic innovations on participating citizens, and follow Jacquet and van der Does’ (Reference Jacquet and van der Does2020) classification of those effects into the categories of citizens’ attitudes, behaviour and capabilities. We then further subdivide these categories into sub-types of attitudes (policy, political and process attitudes), behaviour (political behaviour and civic engagement) and capabilities (knowledge, internal efficacy and reasoning).
Attitude changes are measured by attitude questions on a rating scale. The attitude objects include the policy (i.e., policy attitudes), a political entity (i.e., political attitudes), or the decision-making process (i.e., process attitudes). Changes in policy attitudes are most frequently measured as net group shifts, but have over time also increasingly been measured as individual-level opinion change. Political attitude change is measured by looking at shifts in evaluations of and affective orientations towards Dalton's five classical levels of political support (Dalton, Reference Dalton2004). Process attitudes are often captured by measures of perceived procedural fairness, outcome acceptance or perceived legitimacy of the process.
Behavioural changes are either measured as changes in intended behaviour, or as changes in actual behaviour. The first subtype of behaviour, political participation, is measured with variables such as voting or willingness to demonstrate. The second subtype of behaviour, civic engagement, covers variables such as discussing public affairs or involvement in civic organisations.
Capability changes are measured either objectively by asking factual knowledge questions, or subjectively by probing for self-perceptions. The first subtype of capabilities, knowledge, is most often measured by asking factual questions about the policy issue at hand or about politics in general. Internal efficacy is captured by tapping into participants’ sense of internal political efficacy and confidence in their own competences and knowledge to participate in politics. The last subtype of reasoning skills is measured by assessing participants’ attitude coherence or their level of justification.
Independent variables – Types of democratic innovations
The key independent variable of interest employed in this study is participation in a democratic innovation. Democratic innovations have to be conceived of as ‘grand treatments’ that comprise numerous micro-treatments and together cause the observed effect (Luskin et al., Reference Luskin, Fishkin and Jowell2002). This ‘grand treatment’ is measured in two ways. Either a group that receives the treatment is compared to a group that does not receive the treatment, that is, comparing people that participated in a democratic innovation to people who did not participate, or the same treated group is measured before and after treatment, that is, people that participated in a democratic innovation were surveyed before and after participating.Footnote 11
These ‘grand treatments’ either take place in the field or are simulated in a lab. A lab setting provides for a more controlled environment of the features of the ‘grand treatment’ that are implemented, while field studies have higher ecological validity. Both settings however allow for the manifestation of the core features of each type of treatment.Footnote 12
As mentioned earlier, these ‘grand treatments’ can be categorised into three big families, characterised by a specific set of core features and several common featuresFootnote 13 (based on Elstub & Escobar, Reference Elstub and Escobar2019):
-
(1) Mini-publics: mini-publics bring together ‘groups of citizens that engage in facilitated deliberations on an issue, and make public recommendations’ (Harris, Reference Harris, Elstub and Escobar2019, p. 45). Hence, the fact that citizens engage in discussions with one another and come to decisions based on deliberation are the core features of mini-publics. The participants to mini-publics are usually selected via some form of sortition (Elstub & Escobar, Reference Elstub and Escobar2019, p. 26), and the discussions last between several hours to several weeks (Smith, Reference Smith2009). Reason-giving is an important aspect of these discussions, and the recommendations that are made via mini-publics are often advisory (Elstub & Escobar, Reference Elstub and Escobar2019). Often a moderator is present to ensure that deliberative standards of respect and an equal chance to participate are met. Mini-publics entail a high level of information input stemming from information booklets to expert presentations or the questioning of politicians and stakeholders. In this paper, in order for a democratic innovation to be categorised as a mini-public, the modes of participation have to be discursive expression and listening, and the main mode of decision-making has to be deliberation (following Elstub & Escobar, Reference Elstub and Escobar2019).Footnote 14 Democratic innovations that are classified as mini-publics include citizens’ assemblies, focus groups, citizens’ juries, deliberative forums, and deliberative polls and lab experiments.
-
(2) Participatory processes: participatory processes are rooted in theories of participatory democracy, which advocate the ‘[d]irect participation of citizens in the regulation of key institutions of society, including the workplace and local community’ (Held, Reference Held2009, p. 215). The core features of participatory processes are, therefore, that citizens engage in discussions and vote, as the outcome is decided by aggregating the preferences of the individual participants. Usually, participatory processes are open to everyone and they often have a direct impact on decision-making (Elstub & Escobar, Reference Elstub and Escobar2019; Smith, Reference Smith2009). In this paper, in order for a democratic innovation to be categorised as a participatory process, the modes of participation have to be voting and discursive expression, and the main mode of decision making has to be aggregation (following Elstub & Escobar, Reference Elstub and Escobar2019). Democratic innovations that are classified as participatory processes include participatory budgeting processes, town meetings and lab experiments.
-
(3) Referendums and citizens’ initiatives: the core features of referendums and citizens’ initiatives are that citizens vote on or sign a statute, constitutional amendment, or legal act, and that decisions are taken through some form of aggregation (Elstub & Escobar, Reference Elstub and Escobar2019). The use of referendums and citizens’ initiatives ranges from mandatory to facultative, and their impact from binding to advisory. Some are top-down, others bottom-up, requiring a certain number of signatures to be collected before they may occur. Usually, all citizens can participate in them. Participation, however, generally does not happen face-to-face and the act of voting is rather short as compared to participating in other democratic innovations. Nevertheless, referendums and citizens’ initiatives can go on for weeks and campaigns can provide participants with information regarding the proposed policy, even though this information is often not balanced. In this paper, in order for a democratic innovation to be categorised as referendums and citizens’ initiatives, the mode of participation has to be voting and the mode of decision-making has to be aggregation (following Elstub & Escobar, Reference Elstub and Escobar2019). Democratic innovations that are classified as referendums and citizens’ initiatives include popular referendums, citizens’ initiatives, government-sponsored referendums and lab experiments.
Meta-analysis
Unlike most meta-analyses, where the focus lies on identifying key independent variables that explain a single dependent variable with robust effects across studies, this research takes the opposite approach. As shown in Figure 2, we are interested in one independent variable or ‘grand treatment’, that is, the type of democratic innovation citizens participate in, and seek to evaluate a variety of dependent variables that may be affected by citizen participation in different democratic innovations.Footnote 15
The effects of participation in democratic innovations on citizens.
We analyse the impact of participation in different types of democratic innovations on the dependent variables using meta-analytical techniques. A meta-analysis is ‘the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings’ (Glass, Reference Glass1976, p. 4). Because our purpose in this paper is to map the full range of empirical findings on the effects of participating in democratic innovations on citizens, we take a strict approach and focus on analysing whether significant effects on citizens were found at all, rather than the size of effects. In order to do so, we employ the meta-analysis techniques of ‘vote-counting’ for estimating the direction of the effect, and of ‘combined tests’ for estimating the likelihood that the effect is significant (Geys, Reference Geys2006; Smets & van Ham, Reference Smets and van Ham2013).Footnote 16 This method is arguably the most conservative method to estimate the presence or absence of effects in meta-analytical studies, and reduces the likelihood of Type I errors of finding a significant effect when in fact there is no such effect. This is in line with the goal of this study: to gain an overview of which effects prove robust across studies and which effects require further research.Footnote 17 A more detailed description of the method of meta-analysis employed in this paper can be found in online Appendix F.
After conducting the meta-analysis, the results are checked for possible publication bias, which is a common problem in meta-analyses, using the classic method of Rosenthal (Reference Rosenthal1979, Reference Rosenthal1991). In this method, the fail-safe N represents the number of unpublished studies necessary to render the significant effect found insignificant (for a more detailed explanation, see online Appendix H). In doing so, the tolerance for unpublished or future null results is estimated. In case the significant effect estimate is rendered insignificant by adding only a few null results, the effect cannot be considered as resistant to publication bias (Rosenthal, Reference Rosenthal1979).
In addition to testing for publication bias, we perform a series of robustness checks. We check the robustness of our findings by including only field studies (and excluding results from lab experiments). Furthermore, we check the robustness of our results when only including those studies that apply both a pre-test and a control group. Third, we check the robustness of our results if results are tested at the level of datasets, rather than studies. As some studies make use of the same dataset, this is an important additional robustness test of our findings. Finally, we check the sensitivity of our findings to sample selection bias both across countries and over time. All detailed results for robustness checks are reported in online Appendix I and are discussed in section 4.3.
Analysis and results
In this section, the descriptive analysis and meta-analysis are presented. The descriptive analysis allows us to assess which types of democratic innovations and effects on citizens are most thoroughly studied, and which gaps remain. The meta-analysis allows us to test which effects of democratic innovations on citizens prove robust across studies. Finally, we present the results of the robustness checks.
Descriptive analysis
In this section we aim to answer our first research question: how thoroughly has each type of democratic innovation and each type of effect on participants been studied? First, we assess the extent to which different effects on participants are studied. Then, we look at how much attention is devoted to the different types of democratic innovations.
As can be seen in Figure 3, changes in participants’ policy attitudes are the most frequently studied effects of democratic innovations, followed by changes in their knowledge and their political attitudes. The effect of democratic innovations on participants’ process attitudes and behaviour is less frequently considered. This could be due to the fact that behavioural change is not always immediately measurable after the democratic innovation as it may take time to manifest. The scarcity of studies into process attitudes could be attributed to the fields’ focus on explaining process preferences among the broader public, as opposed to changes in those preferences as a result of having participated in democratic innovations.
Share of studies into each type of effect on participants. Note. The sum of the percentages of studies in this graph is higher than 100, because many studies combine research into different types of effects on citizens within one study.
The type of democratic innovation that is most frequently assessed for its effects on participants are mini-publics (see Figure 4). This is not surprising as mini-publics are often set up in a quasi-experimental way. Moreover, even though recently the focus is moving towards the non-participants (Van Dijk & Lefevere, Reference Van Dijk and Lefevere2023), studying the effects on participants is one of the main interests of empirical work on mini-publics. Participants are randomly selected and several types of mini-publics, such as Deliberative Polls, have a pre- and post-survey built into their design, enabling quantitative analysis of their effects on citizens. Much less frequently studied are participatory processes. This can be explained by a general absence of a pre-test and/or a control group in their design, which makes evaluating the effects of participating in the democratic innovation difficult. Indeed, almost 40 per cent of the studies on participatory processes that were excluded dropped out as a result of this criterion (see also: He, Reference He2019). These studies were mostly based on a single survey among participants after participating in participatory processes. In these cases, lacking information on participants before they participated (i.e., a pre-test), and/or lacking comparative data on citizens who did not participate (i.e., a control group), means the effects of participating in the democratic innovation cannot be evaluated. Additionally, 22 per cent of the excluded studies were dropped because the independent variable was not participation in a participatory process, but rather explanatory factors for why citizens would favour certain participatory processes such as in survey experiments. Another 17 per cent was dropped because the dependent variable was not effects on participants, but the policy impact or the outcome of the participatory process. Indeed, it could be argued that effects on participants are only one of the many focuses of empirical work into participatory processes. Another 15 per cent was dropped because the article was not published in a peer-reviewed journal (see online Appendix D).
Share of studies per type of democratic innovation. Note. The sum of the percentages of studies in this graph is higher than 100, because some studies combine research into different types of democratic innovations within one study.
Referendums and citizens’ initiatives are not widely studied in terms of their effects on participating citizens either. Here, many large-N studies had to be excluded because they were conducted at the macro-level, which made it impossible to trace individual-level effects because these studies measure the effect of the availability of or exposure to, and not participation in, referendums and citizens’ initiatives.Footnote 18 As a consequence, they measure the effect on all citizens in the states where referendums and citizens’ initiatives are taking place or are available, both on those who potentially participated and those who did not. Over a quarter of the excluded direct democratic innovation articles were excluded for this reason (see Figure 11 online Appendix D). Additionally, 17 percent of the excluded articles looked at other independent variables than participation, such as media consumption during the referendum or topic salience. Another quarter of the articles was excluded because they assessed the effects on other dependent variables, such as why people vote in favour or against in the referendum or citizens’ initiative; what the overall representativeness of those who voted is; or what policy impact referendums and citizens’ initiatives have. Articles on referendums or citizens’ initiatives thus also focus on other, equally relevant, effects than the effects on participants. Lastly, 22 per cent of the excluded studies on referendums and citizen's initiatives was not published in a peer-reviewed academic journal, but rather in a book or book chapter.
One first conclusion is thus that some herding may be going on. Clearly, more studies using research designs that allow for testing the effects of participation in participatory processes and referendums and citizens’ initiatives on citizens are highly desirable for the broader field of democratic innovations.
Meta-analysis
In this section we will answer our second research question: to what extent do democratic innovations have an effect on participants? For doing so, we test each type of democratic innovation's effect on citizens’ attitudes, behaviour and capabilities. We test effects both at the level of tests and at the level of studies (a more comprehensive overview of the analysis can be found in online Appendix G). In case the approximated average effect is positive and significant at the level of studies, we conclude that a positive effect of participation in a democratic innovation on citizens’ attitudes, behaviour or capabilities exists.
However, as we have demonstrated in the descriptive analysis section, for many potential effects, we lack sufficient data to test them, especially at the level of studies. In that case, the main finding is that a gap in the current empirical literature exists and no claims regarding those effects on citizens can be substantiated yet.
We start our statistical analysis by first briefly discussing and then assessing the effects of participation in a mini-public, we continue by looking at participatory processes, and then discuss and evaluate the effects of referendums and citizens’ initiatives on participating citizens. We finish with a discussion of the results of the robustness checks.
Participation in a mini-public
We start this section by briefly discussing what effects can theoretically be expected. To do so, we consider how the features of mini-publics (see section 3.2) could possibly affect participants. Due to the high level of information input and the quality of discursive exchange between citizens, deliberative democrats expect mini-publics to have a large impact on the participating citizens. First, the exchange of reasons between citizens during deliberation is expected to lead citizens to change their opinion regarding the policy issue they discuss (Fishkin, Reference Fishkin1995, pp. 162−163). Additionally, by promoting mutual respect and understanding, and by aiming to reach decisions that are perceived as justified by participants, deliberative processes are expected to change citizens’ attitudes towards the process and outcome of deliberation (Gutmann & Thompson, Reference Gutmann and Thompson2004; Myers & Mendelberg, Reference Myers and Mendelberg2013). Moreover, by receiving balanced information and taking part in discussions, citizens will increase their political knowledge, reasoning skills and internal efficacy (Burkhalter et al., Reference Burkhalter, Gastil and Kelshaw2002). Overall, there is thus the expectation that participating in mini-publics has a significant effect on participants’ policy attitudes, and a positive and significant effect on their process attitudes, knowledge, internal efficacy and reasoning skills.
Figure 5 provides an overview of our analysis of the effects of participation in mini-publics. Starting with policy attitudes, we observe a highly significant average effect. We therefore find strong evidence that citizens change their opinion about the policy after participating in a mini-public. Moreover, the fail-safe N test demonstrates that there would need to be 415 unpublished insignificant studies to render this effect insignificant. Turning to political attitudes, the average effect at the level of studies is highly significant, which means that participation in a mini-public is positively related to a change in citizens’ political attitudes. Moreover, publication bias does not seem to affect this relationship. For process attitudes the evidence is somewhat mixed: the average effect at the level of studies is significant, but the number of studies is small and this effect is potentially prone to publication bias.
Plot of effects of participation in a mini-public. Notes. t-test with two-tailed significance levels.
$N$ is the sum of the number of participants in the studies into the specific effect.
${{N}_t}$ is the number of tests conducted into the specific effect.
${{N}_s}$ is the number of studies conducted into the specific effect.
${{N}_{FS}}$ represents the number of unpublished insignificant studies necessary to render the average effect insignificant. The size of the square represents the
$N$ of participants. *p < 0.05, **p < 0.01, ***p < 0.001.
Turning to the effects of participating in a mini-public on behaviour, we see that the average effect is significant, yet prone to publication bias. We thus fail to find robust cross-study evidence that mini-publics cause citizens to become more politically active. With regards to the effect on civic engagement, we do find that the average effect is significant. To render this positive and significant effect insignificant, 18 insignificant studies would need to be unpublished. Hence, we can conclude that we find some evidence that mini-publics on average cause participants to become more civically engaged. This effect would however need to be further substantiated to definitively rule out the possibility of publication bias.
Considering the effects of participating in a mini-public on capabilities sheds light on some of the key hypotheses proposed in deliberative theory: mini-publics are often expected to increase political knowledge, internal efficacy and reasoning skills (Burkhalter et al., Reference Burkhalter, Gastil and Kelshaw2002). Political knowledge stands out as the clearest finding of the meta-analysis: a large and highly significant average effect shows that participation in a mini-public has a significant and positive effect on how much citizens know about the policy topic and politics in general. It is highly unlikely that this effect is produced by publication bias, as there would need to be 708 unpublished insignificant studies to render the overall effect insignificant. The results for internal efficacy and reasoning skills are similar but slightly less outspoken. The average effects are highly significant and these results seem to be unaffected by publication bias. Hence, we also find evidence for the positive effect of participating in a mini-public on citizens’ internal efficacy and reasoning skills, even though the effects are less robust across studies than the effect on citizens’ knowledge.
To conclude, this analysis regarding the effects of participation in a mini-public on participants shows that participation in a mini-public has an effect on citizens, as we find significant effects on participants’ policy attitudes, political attitudes, knowledge, internal efficacy and reasoning skills. There is also some evidence that mini-publics render participants more positive toward the process, but we would need more evidence in order to substantiate this finding. Apart from these, we found some evidence that mini-publics could make participants’ more civically engaged, however a further assessment of that effect would be desirable given the low number of studies and its proneness to publication bias.
Participation in a participatory process
The second type of democratic innovation covers participatory processes. Central to participatory democracy is the contention that individuals ‘learn to participate by participating’ and that these individuals will also ‘participate in decision-making in their everyday lives as well as the wider political system’ (Pateman, Reference Pateman2012, p. 10). Participatory democrats thus conceive of participatory events as schools of democracy, during which citizens gain the knowledge and learn the skills they need to participate more in the future (Pateman, Reference Pateman1970). During active participation, citizens enhance their internal political efficacy and self-confidence and come to appreciate the intrinsic value of participation more (Barber, Reference Barber2003). Hence, participation leads to more participation (Pateman, Reference Pateman1970). Overall, there is thus the expectation that participation in participatory processes has a positive and significant effect on citizens’ political participation, civic engagement, knowledge and internal efficacy. Figure 6 provides an overview of our analysis regarding the effects of participation in participatory processes.
Plot of effects of participation in a participatory process. Notes. t-test with two-tailed significance levels.
$N$ is the sum of the number of participants in the studies into the effect.
${{N}_t}$ is the number of tests conducted into the effect.
${{N}_s}$ is the number of studies conducted into the effect.
${{N}_{FS}}$ represents the number of unpublished insignificant studies necessary to render the average effect insignificant. The size of the square represents the 
$N$ of participants. *p < 0.05, **p < 0.01, ***p < 0.001.
Figure 6 shows that only a few studies look at the effect of participating in participatory processes on citizens’ attitudes. For policy attitudes, we find an insignificant, slightly positive effect, which is based on only two studies. Three studies assessed effects on political attitudes resulting in a positive but insignificant average effect. Only one study looked at the effects on participants’ process attitudes and found no significant effect. Hence, even though there are some preliminary indications of a positive effect on participants’ political attitudes, there exists a significant gap in our knowledge with regards to the effect of participating in a participatory process on citizens’ attitudes.
We now turn to the effects of participatory processes on citizens’ behaviour, which is a key expected outcome of participating (Pateman, Reference Pateman2012). Of the two studies that assess this effect, none found a significant effect of participatory processes on citizens’ political participation. For civic engagement, the average affect size is even negative, meaning that participation lowers civic engagement. However, this statistic is insignificant, and the confidence interval is very large due to a low number of available studies.
Finally, regarding the effects of participatory processes on capabilities, we find insignificant effects on both knowledge and internal efficacy. For knowledge, a few studies contained positive and significant effects, but the overall effect is insignificant. For internal efficacy, the results are even clearer with all four studies finding no significant effects. Hence, these results could be a preliminary indication, based on four studies and 12 tests, that participation in a participatory process has a limited effect on citizens’ internal efficacy. There are no studies looking at the effects on participants’ reasoning skills.
Overall, therefore, due to the small number of studies, we find no robust effects (yet) of participatory processes on citizens’ attitudes, behaviour or capabilities. There are preliminary indications that participatory processes could have a positive effect on citizens’ political attitudes and knowledge, and no effect on their internal efficacy. However, no conclusions can yet be drawn based on the current analysis, which may change if more studies on participatory processes with pre- and post-tests or a control group are conducted in the future (see section 4.1 and He, Reference He2019).
Participation in referendums and citizens’ initiatives
The third and last type of democratic innovation comprise the main instruments of direct democracy: referendums and citizens’ initiatives. Referendums and citizens’ initiatives are rooted in the theory of direct democracy. Most importantly, it is argued that such direct processes can change citizens’ political attitudes, such as feelings of external political efficacy (Smith & Tolbert, Reference Smith and Tolbert2004). Indeed, they are linked to an important idea in democratic thought, namely ‘popular sovereignty as a way of addressing the demands of citizens and the dependence of public policies on their preferences’ (Altman, Reference Altman2010, p. 1). Barber (Reference Barber2003) has argued that direct democracy leads citizens to gain a more inclusive identity and feel more part of a political community. However, more recent scholarship has suggested that the aggregative element of referendums and citizens’ initiatives might lead to the ‘winners’ experiencing an increase in positive affect towards political objects, whereas ‘losers’ would decrease their support, especially if the policy issue is highly salient (Skitka, Reference Skitka2002). Procedural fairness theory challenges that view and posits that a decision-making process in which citizens have the final say should enhance political and process attitudes, even if the outcome of such a process is unfavourable. Additionally, the perception that citizens have a direct influence on the outcome could strengthen feelings of political efficacy, regardless of the outcome (Tyler, Reference Tyler2006). In terms of behaviour, direct initiatives are expected to have a spill-over effect on election turnout (Smith & Tolbert, Reference Smith and Tolbert2004). Direct participation, moreover, is expected to boost future citizen participation in policy making (Barber, Reference Barber2003). Furthermore, exposure to the information given during campaigns can increase citizens’ political knowledge (Smith, Reference Smith2009, pp. 130−133). Referendums and citizens' initiatives should stimulate political interest and facilitate learning about politics (Smith & Tolbert, Reference Smith and Tolbert2004). The process would educate citizens on a permanent basis (Barber, Reference Barber2003). Overall, there is thus the expectation that participation in referendums and citizens’ initiatives has a positive and significant effect on citizens’ political attitudes, political participation, knowledge and internal efficacy.
Figure 7 shows the results of the meta-analyses regarding referendums and citizens’ initiatives. As we can see, effects on participants’ process attitudes, civic engagement and reasoning skills have not been assessed in the empirical literature included in our analysis. This lack of studies is in line with theoretical expectations regarding the effects of referendums and citizens’ initiatives. However, also, the effects for which we would theoretically expect an effect on citizens are currently not widely studied. Policy attitudes have only been found to have a positive and significant effect in one study, which means we cannot draw any meta-analytic conclusions. For political attitudes, we find an insignificant slightly negative average effect. This means that there is no consistent significant effect of participation in referendums and citizens’ initiatives on political attitudes found in the three studies included in our meta-analysis. In terms of the effects on process attitudes, only one study considered these and found no significant effect.
Plot of effects of participation in referendums and citizens’ initiatives. Notes. t-test with two-tailed significance levels. 
$N$ is the sum of the number of participants in the studies into the effect. 
${{N}_t}$ is the number of tests conducted into the effect. 
${{N}_s}$ is the number of studies conducted into the effect. 
${{N}_{FS}}$ represents the number of unpublished insignificant studies necessary to render the average effect insignificant. The size of the square represents the 
$N$ of participants. *p < 0.05, **p < 0.01, ***p < 0.001.
Turning to political behaviour, again only one study assessed the effect on political participation and found positive effects. For knowledge, a small positive effect was found based on three studies. Finally, with regards to the effect on internal efficacy, the average effect based on two studies indicates a positive and rather large effect. However, again, these effects proved insignificant due to the small number of studies.
To sum up, the effects of participation in referendums and citizens’ initiatives on citizens cannot be sufficiently tested due to gaps in the empirical literature. There are however preliminary indications that referendums and citizens’ initiatives could have a positive effect on citizens’ knowledge and internal efficacy, and the jury is still out on the effect on political participation. Nevertheless, more studies are needed to test these effects more thoroughly. If more studies would conduct their analyses at the micro rather than the macro-level, we could fill these gaps in our knowledge with regards to the observed effects of referendums and citizens’ initiatives.
Robustness checks
In this subsection, we test the robustness of our findings. As we mainly found substantial evidence for the effects of participating in a mini-public, our robustness checks focus on those findings. The full results of the robustness checks can be found in online Appendix I.
First, we repeated the meta-analysis with field studies only. Our findings hold when excluding lab experiments. Interestingly, the average effects for all dependent variables are more significant for field studies than for lab experiments. This indicates that in the field, when mini-publics actually have some impact on the policy process, the effects on citizens are more apparent than in a lab setting.
Then, we repeated the meta-analysis at the level of datasets instead of at the level of studies. Our findings largely remain the same, except for the effect of mini-publics on civic engagement. At the level of datasets, the average effect became highly significant. Hence, this robustness check constitutes additional evidence that participation in a mini-public leads citizens to become more engaged.
Subsequently, we repeated the meta-analysis including only those tests that apply both a pre-test and a control group. Some of the significance decreases due to the limited number of tests that apply both a pre-test and a control group. However, the effects for those variables that our earlier analyses found to have an effect on citizens remain significant, providing additional evidence that the effects found in this meta-analysis can be attributed to participation in a mini-public and not to confounding factors.
Lastly, we checked whether temporal and country variations had an effect on our findings for mini-publics. Even though our temporal range for inclusion in the dataset goes from January 1980 to January 2021, the studies in our dataset range from 1999 to 2021. Therefore, as a robustness check, we compared studies into mini-publics conducted until 2010 (N = 26), which is the mid-point of the dataset's time range, with studies conducted after 2010 (N = 59). Since 2011, more attention has been devoted to the effects on political attitudes and process attitudes. However, the only significant difference between the effects on participants until 2010 and after 2010 is the effect on process attitudes. Until 2010, only one study looked at the effects on process attitudes which means no meaningful meta-analytic analysis can be conducted. For the other effects on participants, there are no significant differences between studies conducted until 2010 and after 2010.Footnote 19 With regards to country differences, more than one third (40) of the included studies took (partly) place in the United States (US). That is why, as a robustness check, we have reconducted our analyses without the US. When excluding the US, the average effect size for the effect of participating in a mini-public on process attitudes remains the same but becomes insignificant due to the reduced N. The average effects for internal efficacy and reasoning skills become slightly smaller but remain positive. All other effect sizes and significance levels for mini-publics remain highly similar. Temporal and country variations in our dataset thus do not seem to substantively influence our findings.
Conclusion
Democratic innovations are said to expand and deepen citizen participation in democratic decision-making processes. Building on theories of deliberative democracy, participatory democracy and direct democracy, different types of democratic innovations have been developed, ranging from mini-publics, to participatory processes, to referendums and citizens’ initiatives. These democratic innovations aim to achieve a variety of ‘democratic goods’, ranging from citizens that are more engaged and better informed, to citizens that participate more and consider democratic decision-making processes to be more legitimate. But are they actually successful in doing so?
In this paper, we carried out a meta-analysis of empirical research on the effects of democratic innovations on citizens. Spanning four decades, the paper brings together findings from 100 quantitative studies, in order to answer the following two research questions: (1) How thoroughly has each type of democratic innovation and each type of effect on participating citizens been studied? And, most importantly, (2) to what extent (if at all) do democratic innovations have an effect on participants’ attitudes, behaviour and capabilities?
With regard to the overall effects of democratic innovations on citizens, we found that most robust conclusions could be drawn for the effects of mini-publics. Fewer studies evaluate the effects on citizens of participating in participatory processes and referendums and citizens’ initiatives. This might partially be explained by the fact that in these fields, much attention has also been devoted to assessing the effects of participatory processes and referendums and citizens’ initiatives on policy making and the wider public (Papadopoulos, Reference Papadopoulos2001; Wampler, Reference Wampler2012), whereas for mini-publics these effects have only recently gained widespread attention (Suiter & Deligiaouri, Reference Suiter and Deligiaouri2023). Additionally, studying the effects of participation on attitudes and capabilities has a rich history in the study of mini-publics as several types of mini-publics were specifically designed and carried out to test this type of impact (e.g., deliberative polling).
We found that participation in mini-publics affected citizens’ policy attitudes, political attitudes, knowledge, internal efficacy and reasoning skills, but more evidence is needed to substantiate their effect on process attitudes.Footnote 20 Additionally, we found mixed evidence for the effect of mini-publics on participants’ political attitudes, and limited evidence regarding their effect on civic engagement. With regards to participatory processes, no significant effects of participation were found due to the small number of studies. However, some preliminary indications for the existence of a positive effect on participants’ political attitudes and knowledge, and the lack of an effect on their internal efficacy were found. With regards to referendums and citizens’ initiatives, results appear to indicate that there is a positive effect on participants’ knowledge and internal efficacy, but also here the low number of studies limits the robustness of our findings.
This study has some limitations and caveats. First, relatively few studies on participatory processes and referendums and citizens’ initiatives could be included in the meta-analysis, limiting our knowledge of the effects of participation in these types of democratic innovations on citizens. This was partly due to the research design of these studies. Indeed, for studies on participatory processes, the largest reason for being excluded from the current study was that they did not include a pre-test and/or a control group in their design (see online Appendix D). Yet, surveys that are taken only after the participatory process has taken place cannot demonstrate whether results found were due to participation. It has been observed before that there is a general need for more studies into participatory processes that include these research design features to be able to assess participatory processes’ effects on citizens (He, Reference He2019). For studies on referendums and citizens’ initiatives, over a quarter of the excluded studies tested the effects at the macro-level, thereby testing the effects on citizens of exposure to or the availability of referendums and citizens’ initiatives, rather than participation. Additionally, several studies on referendums and citizens' initiatives focus on other important outcome variables, such as the policy impact of the referendum or the factors explaining the outcome of the citizens' initiative, rather than the effects on citizens. This meta-analysis thus highlights the need for further research on the effects of participatory processes and referendums and citizens' initiatives that employ micro-level data and, where feasible, incorporate pre-tests and/or control groups in their research design. This can, for instance, be done by using similar cities or neighbourhoods without referendums as a control group (at the local level, see e.g., Marien & Kern, Reference Marien and Kern2018) or by using panel data with pre- and post-surveys (at the national level, see e.g., Brummel, Reference Brummel2020) (for two specific examples, see online Appendix K).
Second, the aggregation of different variables into one overarching variable eliminates some of the nuance regarding what is being measured. This is not unusual in meta-analyses, as they are by definition the integration of a large number of findings. The researcher's task is to find the right balance between aggregation and differentiation. The aim of this paper was to provide an overview of what we know regarding the effects on citizens’ attitudes, behaviour and capabilities, with sub-sets of types of attitudes (policy, political and process), behaviour (political behaviour and civic engagement) and capabilities (knowledge, internal efficacy and reasoning). In doing so, we built on prevailing theoretical work on deliberative democracy, participatory democracy and direct democracy, that presents convincing theoretical expectations about these effects of democratic innovations on citizens. However, further distinctions can be made within these categories, and more fine-grained analyses on sub-categories of, for example, mini-publics or policy attitudes are possible. In this study, we did not opt for this approach because of the research goal and limitations in the number of studies, but more qualitative analyses of results at those lower levels of aggregation are certainly possible. We provide the full data on Harvard Dataverse should future scholarship wish to take up this task.
Third, though our analysis found overall positive and significant effects of mini-publics on policy attitudes and political attitudes, the fact that we also found negative effects in a number of studies suggests that effects of democratic innovations may be conditional on their design and the context in which they take place. It also suggests that democratic innovations have the potential to positively affect attitudes under some conditions, but may actually have negative effects on those attitudes under other conditions. Further analysis of potential conditional effects would therefore be worthwhile.
Fourth, meta-analytical techniques are not suitable for including qualitative studies, hence this meta-analysis excluded 14 qualitative studies that met our sample selection criteria and studied the dependent and independent variables of interest. This ties into a larger point: it is known that the effects of democratic innovations on citizens might be subtle and perhaps hard to quantify (Mansbridge, Reference Mansbridge, Elkin and Soltan1999). We therefore refrained from making any definitive claims regarding the extent of an effect on citizens but rather focussed on exploring and testing whether the existing evidence for certain theoretical claims is convincing.Footnote 21
Fifth and last, the scope conditions of our meta-analysis imply that the results found in this study are generalisable to long-standing democracies, not to all countries in the world. Future research that broadens this scope condition to also include non-English research in democracies worldwide would be worthwhile. In addition, this meta-analysis focused on the effects of democratic innovations on participants. However, democratic innovations may have important effects on non-participants (i.e., the maxi-public) too, as several studies have shown (see for instance Bowler & Donovan, Reference Bowler and Donovan2002; Van Dijk & Lefevere, Reference Van Dijk and Lefevere2023), and further research into the differential effects of democratic innovations on participants and non-participants would be highly valuable as well.
The good news is that many of these caveats highlight gaps in our knowledge about the effects of democratic innovations on citizens that future research could take on board. First of all, research on participatory processes would benefit from including pre-tests and/or control groups in future studies and research on referendums and citizens’ initiatives would benefit from research using more micro-level data. Second, our results highlight areas of research that appear to have been overlooked by empirical studies, even if theoretical work on deliberative, participatory and direct democracy suggests effects on citizens should occur. For instance, the effect of participation in referendums and citizens’ initiatives on participants’ future political participation is only assessed in one study, even though direct democratic theory suggests there should be a positive effect (Barber, Reference Barber2003). Third, the mixed findings for several effects indicate that more careful consideration of the conditionality of effects of democratic innovations on citizens may help shed light on what types of democratic innovation designs, in what contexts, have positive effects on citizens’ attitudes, behaviour and capabilities.
To conclude, this paper has shown that there are effects of participation in mini-publics, but it also shows that much work remains to be done in the study of democratic innovations’ effects on citizens, especially in the realm of participatory processes and referendums and citizens’ initiatives. Empirically, we need to move ‘beyond the deliberative hegemony’ (Smith, Reference Smith, Elstub and Escobar2019, p. 579). Democratic innovations are increasingly applied to solve policy issues and realise ‘democratic goods’ at the local, national, as well as supra-national level. The question whether these innovations indeed realise those ‘democratic goods’ and decrease democratic disillusionment amongst citizens therefore remains an important one to fully answer, for policy-makers as well as the democratic system.
Acknowledgements
We would like to thank Stella Koenen and Ramon van der Does for their help with the coding of the articles and feedback on the codebook. We would also like to thank Amélie Godefroidt for her methodological insights. We would like to thank the discussants and participants to the COST Action Conference on Constitution-Making and Deliberative Democracy 2021, to the NEXT GDC Symposium 2021, to the ECPR General Conference 2021 panel on ‘Public support for democratic innovations’, and to the NIG Conference 2022 panel on ‘Democratic innovations and citizen participation’ for their feedback on an earlier version of this paper. Finally, we would like to thank the three anonymous reviewers for their insightful comments on this paper.
Data availability statement
The authors confirm that the data used in this study and the script to replicate the analyses are available at Harvard Dataverse (https://doi.org/10.7910/DVN/DZCLWS).
Online Appendix
Additional supporting information may be found in the Online Appendix section at the end of the article:
N
Nt
Ns
NFS
N
N
Nt
Ns
NFS
Open access
