Deliberative polling and policy making

Efforts to consult the mass public confront a dilemma. What with most people not knowing much about most policy choices, direct consultation will harvest mostly uninformed opinion. But if policymakers only consult policy elites, who are more knowledgeable, the consultations would hardly yield a public voice. The choice, it appears, is between representative but uninformed mass opinion and informed but unrepresentative elite opinion – between the democratic values of political equality and deliberation.

‘Deliberative Polling’ offers a way out of this dilemma. Recruitment through random sampling gives each member of the population an equal chance of participating, and moderators ensure opportunities for equal participation in small group discussions. At the same time, the small group discussion and plenary question-and-answer sessions give the participants the opportunity to consider competing points of view, to become more informed about trade-offs affecting their choices, and then to express their considered opinions in confidential questionnaires.

Deliberative Polls have been conducted locally and nationally in a variety of countries and policy contexts, ranging from the United States and Britain, to Canada, Australia, Denmark, Italy, Bulgaria, Hungary, Northern Ireland and transnationally in a Europe-wide project for the entire European Union. China, however, poses a distinctly different political and policy context.

Local public consultation in China

Efforts at public consultation in China have increased markedly in recent years, with the public being invited to express its views in ‘public hearings’ on such local issues as the prices of water and electricity, park entry fees, the relocation of farmers, the development of historic sites and even the possible relocation of the famous Beijing zoo.Footnote 4 In the mid- to late 1990s, it became increasingly common for villages to hold meetings in which village representatives discussed major decisions on local issues.Footnote 5 These practices have now spread to more urban communities. In the Shangcheng district of Hangzhou City, for example, a consensus conference or ‘consultation meeting’ is held regularly once a month.Footnote 6 In one state-owned factory, a representative council of staff and workers deliberated for several months to decide the allocation of new departments to workers and managers.Footnote 7

The setting for the first Chinese Deliberative Poll was Zeguo Township in Wenling City, which had previously held numerous deliberative consultations (called kentan: ‘sincere heart-to-heart discussion’). From 1996 to 2000, there were 1,190 of these at the village level, 190 at the township level, and 150 in governmental organizations, schools and the business sector.Footnote 8 Some were connected to decision making through the local People’s Congress.

Some similar practices have even sprouted at the national level. In 1996, the first national law on administrative punishment introduced an article on holding public hearings before decisions about punishments were taken.Footnote 9 The famous Article 23 of the Law on Price passed by China’s National Congress in December 1997 specified that the price of public goods should be decided through public hearings. This was followed by the Law on Legislature, passed in 2000, which required public hearings before passing any legal regulations or law.Footnote 10 More than fifty cities have now held legislative public hearings.

But public hearings have the same limitations in China as anywhere else. Above all, the participants are unrepresentative. Thus, Cai Dingjian complains that public hearings do not really involve ordinary citizens and urges ‘popularizing’ them, and Yang Zhongxin, the Director of the Price Bureau at Qinghuangdao City, argues that public hearings on prices often decide to increase them because the hearings are usually dominated by business interests.Footnote 11 In addition, the procedural requirements are often vague and there may not be sufficient time for deliberation. In China, moreover, there is the additional danger of the dialogue’s being manipulated, or of officials’ selectively mobilizing the participants to support a pre-determined conclusion.Footnote 12 Deliberative Polling, with its random sampling, extended deliberations, balanced briefing materials and expert panels, and clear aggregation rules for determining results, is designed to overcome these limitations.Footnote 13

The deliberative poll in Zeguo Township, Wenling City

Zeguo is part of Wenling City, a county-level city with a vibrant private economy. Zeguo has an area of 63 square kilometres, of which the town centre is 6.5 square kilometres. It contains eighty-nine villages, and there are nine urban residential committees. The permanent local population is roughly 120,000, and the floating (migrant) population another 120,000. The four major types of employment are in the manufacture of shoes, water pumps, air compressors and materials for the construction industry.

The question participants in the Zeguo Deliberative Poll were asked to consider was which of a set of thirty possible infrastructure projects should be funded in the coming year. The projects, designed by local officials, included new bridges, roads, a school and city gardens. Altogether, these projects would cost roughly RMB 137,000,000, compared to the estimated RMB 40,000,000 that could be raised for them. Thus, the local government had to prioritize. The available funding could cover only ten to twelve of the thirty possibilities (depending on which were chosen).

The idea, from the beginning, was to use Deliberative Polling as a way of democratizing local policy making. Thus, the Zeguo Town leadership made – and carried through on – an explicit commitment to fund the projects the sample rated highest after deliberating. Therefore, this deliberation was effectively binding.

A working committee, composed of the deputy head of the department of propaganda in Wenling City, Dai Kangnian, Officer Chen Yiming, Party Secretary Jiang Zhaohua of Zeguo, and Deputy Party Secretary Wang Xiaoyu of Zeguo, organized an expert committee that carried out a preliminary study of, wrote the feasibility reports for, and drafted briefing materials on the infrastructure projects. We helped local officials prepare the questionnaires and briefing materials, which contained arguments for and against each project.

We assess this application of Deliberative Polling under the following headings:

1. (1) The representativeness of the sample.

2. (2) The occurrence and magnitude of net policy attitude change.

3. (3) The extent to which the policy attitude changes appear to rest on normatively desirable processes of deliberation. In particular,

   1. (a) The avoidance of distortions from unequal social influence,

   2. (b) The absence of uniform polarization,

   3. (c) The development of public-spirited preferences,

   4. (d) The occurrence and magnitude of learning, and

   5. (e) The extent to which that learning drives the attitude change.

4. (4) The extent to which the post-deliberation attitudes or pre-to-post-deliberation attitude changes influence public policy.

The experiment began with an initial survey in March 2005. A simple random sample of 275 Zeguo residents was drawn from a household registration list. As an inducement to attend, the participants were paid a modest fee.Footnote 14 The response rate (the proportion completing the initial interview) was a pollster’s idea of heaven, the participation rate (the proportion of those who attended the deliberations) a Deliberative Pollster’s idea of the same. Of the 275, 269 completed the initial questionnaire, and 257 showed up on the day (9 April 2005). Of the latter, 235 also completed the final questionnaire.Footnote 15

Table 1 compares the 235 interviewees who attended the deliberations and completed the final questionnaire (‘participants’) with the thirty-four who did not (‘non-participants’). It shows the participants to be an attitudinally representative subsample of the whole interview sample. On only one of the thirty projects (roughly the 5 per cent expectable by chance) did the participants and non-participants enter the process with significantly different attitudes.

Table 1

Attitudinal Representativeness (T1)

Note: Entries are means; p-values are two-tailed.

Table 2

Demographics

Note: Entries are percentages except for age, which is in years.

*Difference between participants and non-participants significant at the 0.10 level.

†Difference between participants and non-participants significant at the 0.01 level.

§There are no statistically significant differences between the participants and the whole sample.

To be sure, there were also some sizeable and statistically significant differences between the participants and non-participants with respect to socio-demographic characteristics.Footnote 16 But there were only thirty-four non-participants, and the participant sample still closely resembles – and in no wise differs significantly from – the whole interview sample (which is in turn virtually the same as the entire sample).

In one respect, however, the sample did manifestly differ from the population. There were too far too many men (although the participants were less unrepresentative in this respect than the non-participants). This resulted from a failure to implement one customary element of Deliberative (and other careful) Polling, namely random selection within the household. Instead, the household members exercised some discretion as to who would take the questionnaire. Subsequent Chinese Deliberative Polls (see below) have corrected this problem by sampling individuals from the electoral list rather than from households.

The deliberation lasted just one day. As in other Deliberative Polls, the design alternated small group and plenary sessions. In the small group sessions, the participants considered the advantages and disadvantages of each project and formulated key questions to put to the panels of competing experts in the plenary sessions. There were sixteen small groups, averaging about sixteen participants apiece. They were led by moderators (teachers selected from Zeguo high schools) trained not to give any hint of their own opinions, to foster equal and civil discussion, and to facilitate the process of forming questions for the expert panels. At the end of the day, the participants completed an augmented version of the same questionnaire as they were given on first contact.

Attitude change (project priorities)

The participants were asked to rate each of the thirty projects on a ten-point scale, with 0 being extremely unimportant, 10 being extremely important, and 5 being neither important nor unimportant. Table 3 shows the mean ratings before and after deliberation. The scores are translated to a 0 to 1 scale. The results for twelve of the thirty projects showed a statistically significant change at the 0.1 level (two-tailed).

Table 3

Project Priorities

Generally speaking, the participants became more interested in sewage treatment and road construction that would affect their daily lives. All three sewage treatment projects received much higher support after deliberation. Some of these changes appear to reflect an increase in something like public spiritedness, about which we say more below. The average support for Wenchang Main Avenue, a new road that would cross a number of villages, increased by almost a full point. In contrast, roads more specific to particular villages received diminished support. When it came to parks, a ‘People’s Park’, for recreation, gained support, but Wenchang Park, a kind of town square that was touted as good for the city’s image, lost support, as did commercial roads designed to connect factories with main roads.

For most ensuing analyses, we boil these projects down to ten broader categories, captured by mostly multi-project indices. Five are road-related. Industrial Roads includes roads in industrial areas and they are targeted to improve these industrial zones. Village Roads includes road constructions within specific villages. Main Roads includes roads traversing the whole town or important to most villages. Commercial Roads includes roads connecting factories with main roads. Wenchang Main Avenue is a single-item index. Two indices concern parks: Recreational Park, a single-item index, refers to a park for the entire township. Other Parks includes park constructions for specific villages. Sewage Treatment includes four sewage treatment projects, all designed to serve the entire township. Township Image contains projects aimed at improving the township’s appearance, for example, by planting greenery and flowers. Cultural Heritage contains two projects (the reconstruction of Old Street and second stage construction of Wenchang Park) using traditional cultural architecture and designs. Appendix A lists the variables in each index and provides the inter-item correlations (for the two-item indices) and the Cronbach’s alphas (for the multi-item indices). Consistent with Table 3, Table 4 shows the participants as coming to give greater priority to Wenchang Main Avenue and Sewage Treatment and lesser priority to Village Roads, Commercial Roads, Township Image and Cultural Heritage. Four of these six indices changed significantly at well below the conventional 0.05 level, while another missed only by a whisker (at 0.066 and 0.076).

Table 4

Project Indices

Public spiritedness

Discussions of deliberation and political participation have long speculated that forms of public consultation that involve shared discussion and decision about public issues will foster ‘public spiritedness’ – a greater support for policies of broad rather than narrow public interest. J. S. Mill, building on Tocqueville’s account of town meetings and juries in America, praised institutions that serve as ‘schools of public spirit’ – local decision-making bodies where the interests of the whole community are discussed and individual citizens have some role in decisions. More recent writers, like Jane Mansbridge, have continued the speculation but have encountered difficulty finding clear empirical confirmation.Footnote 17

The Zeguo Deliberative Poll provides a good opportunity, in an unexpected context, to test these speculations. The projects varied a great deal in the proportion of the town’s population they would benefit. A five-point scale was used to rate the extent to which each project would benefit the whole of Zeguo Township. Projects benefiting only a small number of villages were rated as 1, projects benefiting a large number of villages as 5. The ratings are displayed in Appendix A. The correlation, across the thirty policy priorities, between this shared-benefit rating and the change in policy priority is 0.655. After deliberation, the participants’ priorities shifted towards projects benefiting the entire town. In that important sense, at least, they appear to have become more public spirited.

Knowledge gains

The questionnaire contained four questions tapping the participants’ knowledge of the policy context in Zeguo Township. These asked: (1) whether Zeguo Township’s revenue had increased by 10.2 per cent, 20.1 per cent, 33.7 per cent, or not at all between 2003 and 2004; (2) whether Zeguo Township’s floating population is 50,000, 120,000, 200,000 or 300,000; (3) whether any of the following (water pumps, shoes, plastic products, or air compressors) are not a major product of the township; (4) whether the township has zero, one, two, five or seven parks. The correct answers were (1) 33.7 percent, (2) 120,000, (3) plastic products, and (4) two. The participants gained on all four items, significantly so on three of them. On average, the percentage answering correctly increased by 11 per cent, which is highly significant.

Social influence

Some critics of deliberation, including Lynn Sanders and Iris Marion Young, have argued that the more privileged will dominate discussions and disproportionately influence the results, which should thus incline towards their views.Footnote 18 Such a skew would undermine the aspiration of deliberative democrats that everyone’s views get appropriate consideration on merits. Critics of previous, less structured Chinese public consultations have noticed the same danger there.Footnote 19

One simple empirical approach to this question is to examine whether the sample as a whole tends to move towards the initial opinions of the more privileged or higher status participants. For the purposes of this test, we take the more privileged to be men, the more highly educated, and those in the most privileged occupations – in this setting, the entrepreneurs and merchants. Table 6 shows that, far from moving towards the positions of the more privileged, the sample moved away from the Time 1 position of the more highly educated on half of the indices, away from the Time 1 position of the men on three-fifths of them, and away from the Time 1 position of the entrepreneurs and merchants on four-fifths of them. At least in this setting, Deliberative Polling seems to create an environment in which inequalities in the broader society do not distort the deliberative process. The more privileged could hardly be said to dominate the process when opinions move away from their views.

Table 5

Knowledge Gains

Note: n = 235, p-values one-tailed.

Table 6

Social Influence

Polarization and consensus in small groups

Cass Sunstein has argued that there is a ‘law of group polarization’, according to which discussion predictably moves participants towards more extreme positions. A group beginning on one side of the mid-point will move further out in the same direction. This poses a normative challenge to deliberative democracy by implying that deliberation may change attitudes as a predictable artefact of group psychology rather than on the merits as the participants see them. Sunstein believes that there are two basic mechanisms by which discussions produce polarization in this sense. First, if the group begins on one side of the mid-point, the arguments voiced are likely to be weighted in favour of that side. Secondly, there is a social comparison effect. People will feel social pressure to agree with the perceived majority. Sunstein and various collaborators have confirmed these hypotheses with experiments with mock juries.Footnote 20

While we have not found this pattern in previous Deliberative Polls,Footnote 21 Sunstein has argued that the pattern applies to deliberative processes generally, including those resembling Deliberative Polls.Footnote 22 We suggest that the relationship between deliberation and polarization depends on institutional design. Two features of Deliberative Polling may limit the problem there. First, the arguments to which the participants are exposed tend to be relatively balanced, thanks to balanced briefing materials, moderated small group discussions aimed at considering competing arguments, and balanced panels of competing experts. Secondly, there is minimal social pressure, since the participants’ final opinions are solicited only in confidential questionnaires, and there is no common verdict to be reached. This makes Deliberative Polling very different from the mock jury deliberations from which Sunstein largely draws his evidence.

Does Deliberative Polling display its usual absence of polarization in China? Table 7 reports the movements towards or away from the mid-point for the ten priority indices in the sixteen small groups. Overall, only 47.5 per cent of the 160 group-issue combinations move away from the Time 1 mid-point, about what one would expect by chance. In this Chinese context, too, Deliberative Polling belies the ‘law of group polarization’.

Table 7

Polarization and Convergence

Another worry is that the members of given small groups might always converge on a single position. Deliberation might tend to produce consensus, even if not steered towards it. Much presumably depends on the degree to which relevant interests and values are shared. In past Deliberative Polls, the within-group variance of opinion has not typically decreased in much more than half the group-issue combinations.Footnote 23 But what of these Chinese deliberations? The right-most column of Table 7 shows that the variance within a larger than usual percentage (70.6 per cent) of the small group-issue combinations do shrink.

This is not large enough to be worrisome but does leave the question of why it is larger than in most previous Deliberative Polls. It may well be something about the nature of the issue. At a glance, the projects that particularly stood to benefit the whole community tended to be those for which the percentage of groups whose within-group variance decreases was largest. The mean percentage for the Sewage Treatment, Wenchang Main Avenue, and Recreational Park indices is 89.6 per cent. For the remaining seven indices, it is only 62.5 per cent. At the level of the thirty individual projects, the correlation between the shared-benefit scale introduced above and the percentage of groups showing a decrease in within-group variance is 0.429. At the level of the ten indices, the correlation is 0.142. In this light, the tendency towards increased agreement, concentrated as it is on projects benefiting the whole community, would appear to be a consequence of the increase in public-spiritedness.

Knowledge gain and attitude change

A simple model can permit us to estimate the extent to which the participants who emerged with the most knowledge were the ones who changed the most.Footnote 24 The model is:

where P ₁ and P ₂ are the participant’s positions at T1 and T2 (before and after deliberation), K ₂ is his or her knowledge at T2, G ₁ is the mean position of the participant’s small group (disregarding the participant himself or herself) at T1; γ ₀, γ ₁ and γ ₂ are the parameters; and u is a disturbance.Footnote 25 Luskin has shown that under three plausible, indeed compelling conditions, T2 knowledge is actually a better proxy for true information gain than is T2–T1 knowledge. The three conditions are: ceiling effects (respondents answering every question correctly at T1 cannot show any gain); item sampling bias (the knowledge questions asked tend to be very easy compared to the universe of possible knowledge questions on the issues); and the ‘rich getting richer’ (the well-known tendency for those who begin with more knowledge to acquire more).Footnote 26

Normatively, we should want the T2 knowledge coefficient γ ₁ to have the same sign as the mean opinion change P ₂–P ₁, meaning that those who emerge knowing the most are disproportionately responsible for the overall change. Theoretically, we should also expect, though not necessarily want, the small group coefficient γ ₂ to be negative, meaning that participants are narrowing the gap between their own and their small group’s T1 position.

Table 8 reports the ordinary least squares estimates for the six project indices showing significant or borderline significant change. The signs of the estimated coefficients are all as expected – those for the small group variable P ₁–G ₁ always negative, those for T2 knowledge always sharing the sign of the overall opinion change. All six of those for the small group variable are highly significant (p<0.01), as are three of those for T2 knowledge. A fourth estimated T2 knowledge coefficient just barely misses (p<0.10). At least on the surface, the participants do seem to be narrowing the gaps between their own and their small group’s mean T1 opinion. It should be noted, however, that if the P ₁–G ₁ variable is split apart, and the model re-estimated with K ₂, P ₁ and G ₁ as separate regressors, almost all of (P ₁–G ₁)’s effect turns out to belong to P ₁, whose negative coefficient can be interpreted as mere regression towards the mean.Footnote 27 How much to make of the P ₁–G ₁ coefficient estimates is therefore unclear. What is clear, from the K ₂ effects, is that the changes in the priorities awarded these projects are, in large measure, learning-driven. Those emerging the most knowledgeable contribute most to the overall opinion change.

Table 8

Knowledge Gain and Attitude Change

Note: Cell entries are coefficient estimates with estimated standard errors in parentheses. The parenthetical signs in the column headings indicate the direction of net change for the sample as a whole and thus the expected sign of the information coefficient.

†T2 knowledge is the mean of the four information items.

‡The group mean variables are calculated on the other group members, excluding the respondent.

*Significant at the 0.10 level, **significant at the 0.05 level, ***significant at the 0.01 level.