Sampling and Interpretation Bias

The sub-discipline we address is quite fortunate in that it is characterized by regular engagement between political philosophers, political scientists, and practitioners on merit. Many of the movers and shakers in the discipline can justifiably claim expertise across these categories. This is no mean achievement. However, despite the positive relationship that has been built between philosophy and systematic data collection in the sub-discipline, we argue that more needs to be done to refine and integrate the lessons of practice and theory. This is not a new refrain (see, for example Mutz Reference Mutz2008 and Thompson Reference Thompson2008), but it is one whose nature we wish to update. There is a healthy and necessary tension between normative and empirical work in the discipline (Sabl Reference Sabl2015). We argue that the advancement of any such sub-discipline is hampered when this tension is either too strong or lost altogether; and that both problems arise in the context of work on deliberative institutions.

First we contend that some of the pattern witnessed above can be explained by a lacuna of work on conceptualizing so-called democratic innovations themselves in order to establish standards for invoking empirical cases of the class. The meaning of almost every major concept in social sciences can be contested. While a fixation on definitional issues can be poisonous to the advancement of a discipline, it is hard to overstate the difficulty that defining a class by the quality of being innovative creates for the scientific process of comparison. A normative project of democratic deepening presupposes that improvements on the status quo are necessitated. The gestation of the research area within normative circles has had a lasting influence in the naming of the sub-discipline itself and the expectation of what is studied empirically. Studying innovation implies a commitment to understanding the process of experimentation oriented to identifying and implementing improvements. But what counts as innovation for the purposes of systematic comparison can be confusing because a solutionist approach automatically connotes an idiosyncratic improvement on whatever has gone before.

This scenario presents clear dangers as sampling of exceptional cases is incentivized. An entire class is selected on the dependent variable (or a positively skewed almost-constant as it were). A correction towards the mean is made difficult because each case can be presented as both unique and a member of the class—where what defines the class is the quality of being unique in some way. Such an account is stylized in the sense that no serious scholar will make the explicit claim that a case is both unique and the same in terms of characteristics relevant to the study at hand. And of course if a sub-discipline did not depart from what went before there would be no need for the sub-discipline. But it is the lack of lucid conceptual analysis (which requires both theoretical and empirical contributions) that allows for this selection bias to manifest in a dearth of studies of failures. Failures will simply not be as salient where a class is defined by solutionist claims. ^{Footnote 2}

There are very few occasions that we are aware of where empirical scholars have been clearly able to map out or even borrow from theory a set of necessary conditions that distinguish what a non-case of a democratic innovation under investigation is from a case with a negative outcome based on a chosen dependent variable. If the first cases brought to attention are all those with exceptional outcomes, we might expect that in reality the typical case cannot be so, and we cannot be sure of what the typical case looks like and how it differs from our exceptional ones. We would expect much like in Galton’s (Reference Galton1886) original that the child’s height would regress from that of the exceptional parent (say the British Columbia Citizen’s Assembly archetype) towards the mean (something less deliberative or well-planned perhaps). The problem here is that we don’t have a good sense of a population and we don’t know what the mean might look like. If we took a sample of participatory processes over time we would probably expect that the results in Porto Alegre were atypical. It is not that variation does not exist, it is that the standards for communicating and interpreting that variation do not. Current conceptions of deliberative or democratic innovations either are too exclusive of real-life consultations that are regularly replicated by a plethora of governments and agencies; or can be too inclusive of non-deliberative interventions whose effect departs from standard outcomes—a criticism that has been levelled at the burgeoning deliberative systems literature (Owen and Smith Reference Owen and Smith2015). Where that tension between normative and empirical work is weakened, conceptual clarity will suffer.

This leads us to the second related problem which manifests not when the tension between normative and empirical analysis is too weak, but when it is too strong. What should researchers do when presented evidence that falsifies a hypothesis? Where normative commitments are strongly held, it is all too easy to categorize failures post-hoc as instances of non-deliberation or unintended consequences of otherwise successful processes.

We should be wary of temptations towards concept-shifting (i.e., oh well that wasn’t really deliberation then) as a response to evidence of negative effects on democratic outcomes generated by democratic innovations. If deliberative scholars are overzealous in their normative commitments, evidence that reduces the odds that mechanisms designed to improve democratic deliberation are successful in certain contexts will be ignored or downplayed and the project of democratic deepening will suffer. Where such a situation persists a vicious circle is created whereby negative portrayals of deliberative democracy in certain contexts are not seen as part of the field of democratic innovation (Hibbing and Theiss-Morse Reference Hibbing2002, Shapiro Reference Shapiro2003); and the sub-field becomes no more than a self-referencing echo-chamber. Following Mutz (Reference Mutz2008) we reiterate calls that contributors to debates on democratic institution-building should make clear distinctions between what they expect deliberative democracy to deliver, and what democratic goods they expect specific instances of deliberation to deliver. In other words, deliberative democrats should reflexively consider the scope of their arguments in light of the evidence. That is, we need a better idea of how deliberative democracy is likely to achieve democratic goods in different contexts; at the expense of wasting time trying to interpret what might make a context ‘deliberative’ or not. It is only once this work is done that we can begin the crucial empirically-informed normative work of deciding what kind of deliberation should be prioritized and when.

Pressure at the Gate—Supply Failure

Another factor that contributes to the scarcity of interest in failures and fragilities of democratic innovations is increasing constraints on relationships between researchers and gatekeepers that generates a low supply of these type of studies.

The competitive nature of the emerging democratic innovation ‘industry’ and the scarcity of long-term funds implies that firms that specialize in the facilitation and support of democratic innovations crave academic validation, that is, research that cheerleads their efforts and products. This system is born not of excessive greed or narcissism on the part of for-profits or non-profits that make up the sector, but of a survival imperative. Perverse incentives are generated by market forces, political competition and austerity measures. The problem is not to compete with other providers of similar services, but to survive a crowded marketplace for political reform, beholden to ideological competition. Often democratic innovations are adopted in the midst of fierce political competition and thus any critique, no matter how small and abstract, might be used to justify abandoning the project.

Inevitably the opportunity spaces for research are limited or mollified, when small organizations are tasked with collecting process data that can be used to review their own performance. The critique of evaluation in the industry is not new (Lee Reference Lee2015), but intensifies as the public participation professional (PPP) space becomes more crowded and competition among small organizations for survival becomes more pronounced in times of austerity. Academics are often involved in these evaluations but their skill-set can be used as much to avoid serious scrutiny as to provide it. The request for signing of a confidentiality agreement (see an example in appendix 2) is becoming common for academics that join the research boards of organizations implementing democratic innovations.

Thus academics when invited to evaluate democratic innovations are forced to inhabit a difficult space because they have to generate knowledge that is rigorous, but at the same time cannot be used to damage the reputation of the innovation itself. In many cases the innovation is promoted by well-meaning organizations that open their doors to research. The incentive for the academic is to stop asking the difficult questions (or at least answering them publicly for now), and concentrate on ancillary debates in which significant positive impact can be shown. Most evaluations of deliberative institutions, for example, show increases in learning and internal efficacy of participants—something that is quite important—but very few enter into system-level questions of real policy impact and long-term empowerment.

In order to overcome these problems, academics have developed and implemented innovations themselves. The developer/researcher is a role that has become well understood in the discipline and which funding bodies have supported. However, internalizing development does not automatically shield research from market or political forces in the long-run. When initial exploratory grants run dry, the innovation developed by an academic has to self-fund on the basis of its own merits and not on the basis of the research output. Often developer/researchers have been criticized for not being forthcoming with their data or allowing external impact evaluation of their innovations (Lupia Reference Lupia2004). As a community we are left sleepwalking into a scenario where researchers who have fought long and hard to develop good reputations are criticized because of information constraints not entirely of their own making.

Thus the bias towards generating evidence that is positive is rooted in a mix of moral and personal incentives. Developers, activists and academics themselves are for the most part committed to the normative project of democratic deepening and are interested in the proliferation of these programs so that they can generate more studies and obtain more data.

Publication Bias—Demand Failure

Debates above may of course be exacerbated by a familiar bias for publishing ‘findings’ (Gerber, Green and Nickerson Reference Gerber, Green and Nickerson2001) and innovation studies that have desirable consequences (Rogers Reference Rogers1983, Sveiby et al. Reference Sveiby, Gripenberg, Segercrantz, Eriksson and Aminoff2009). In 1983 Rogers in reviewing the literature on the diffusion of policy innovation found only 0.2% of studies investigated unintended negative consequences. This evidence bears out what we suggested about disciplinary framing above. If you don’t find an ‘innovation’ you don’t have much to shout about.

Our exploratory analysis is the first tailored to the specificity of the subfield of democratic innovations and it highlights that the top five journals in political science do not contain a single empirical paper analyzing a failure of a democratic innovation in the past ten years. We find such papers in Politics and Society, (49th in the last Thompson and Reuters ranking) and in the Journal of Public Deliberation (unranked). Are studies of failures not conducted? Or are journals not publishing them because they are less likely to receive attention?

All new fields have to prove their worth in terms of demand. But beyond that there are also very specific characteristics of the democratic innovation sub-field that might be exacerbating this problem. A failure to reach out across sub-disciplinary boundaries has likely resulted in a lack of excitement among editors of generalist journals about what is at stake in replications of democratic innovations which tend to be small-scale with little immediate policy impact (Goodin and Dryzek Reference Goodin and Dryzek2006). To most social scientists it is not surprising that these experiments fail, and therefore the prospects for learning from expected failures may seem limited (unless they are deemed exceptional). The field has established itself as a normative critique but not yet an empirically grounded one, even if we are now witnessing hundreds of thousands of new deliberative institutions bubbling up around the world.