Nonresponse in Survey Research

There are two types of nonresponse: unit nonresponse and item nonresponse. Unit nonresponse occurs when the respondent refuses or fails to participate in the survey. Neil Munro finds that unit nonresponse, also known as refusal bias, may have contributed to overestimates of trust in government by as much as 6 per cent,Footnote ⁴ despite advances in sampling techniques that seek to minimize the effect of refusals.Footnote ⁵ In addition to unit nonresponse, item nonresponse is common in surveys in China and may impact substantive conclusions.Footnote ⁶

Item nonresponse, or non-substantial response, can be defined as “the failure to obtain information for a question within an interview or questionnaire” and can occur owing to respondent and/or question characteristics.Footnote ⁷ We use both terms, “item nonresponse” and “non-substantial response,” to indicate a response of “do not know,” “refuse to answer” or for missing data in lieu of answering a question. Several respondent characteristics have been associated with item nonresponse: education, age, sex and cognitive ability.Footnote ⁸ NSRs sometimes suggest that the respondent perceives the question as sensitive, which could also mean that preference falsification could be higher for these questions.Footnote ⁹

NSR is understudied in the China field but the few existing studies suggest that politically sensitive questions are associated with NSRs and that changes in political dynamics can change NSR rates. NSRs for questions concerning satisfaction with the government increased by approximately 10 to 20 per cent following the 1989 government crackdown in Tiananmen Square.Footnote ¹⁰ Using the World Values Survey, Liying Ren finds that China has comparable rates of NSR to the US in general but higher rates of NSR with political questions.Footnote ¹¹ With the exception of Ren's study, research on NSR in China relies on data from the 1990s.Footnote ¹² Experimental studies are divided on the degree to which Chinese self-censor through nonresponse or preference falsification.Footnote ¹³ Recent survey research only briefly discusses NSR.Footnote ¹⁴

Until recently, survey research in China was uncommon: Melanie Manion identifies only eight scientific political science surveys conducted between 1978 and 1995.Footnote ¹⁵ In 1996, survey research increased but publications often avoided discussing nonresponse. We found that in 67 studies published between 1998 and 2016 that use survey data collected in China related to politics, 61 per cent did not specify an NSR rate, nor did they explain how they addressed NSR in their analysis (Figure 2). Some had a very high response rate. Others dropped the missing data, while a few recoded missing data or used imputation.

Figure 2:

Nonresponse in Survey Research in China

Notes:

Sample includes 67 academic publications from 1998 through 2016 that use survey research in China on political or social science topics.

We focus on survey questions on political trust, i.e. whether respondents perceived government leaders at the village, county, provincial and central levels to be trustworthy.Footnote ¹⁶ A commonly held belief in China is that Beijing's policies are good but that local government is corrupt and implements central policies poorly. Central officials generally enjoy high levels of trust while local leaders are blamed for problems.Footnote ¹⁷ Therefore, questions regarding trust in government leaders are particularly salient in the Chinese context.

Nonresponse and Survey Design: HEPS

We deliberately sought to reduce nonresponse in HEPS. First, we designed questionnaire items to be consistent with local linguistic expressions. Second, to the extent possible, we selected teams of enumerators who were gender balanced, local to the region and who spoke local dialects. Third, enumerators underwent three days of training, which included a pre-test in each province, following best practices.Footnote ¹⁸

A common preconception about villagers is that they suffer from suzhi di 素质低, which roughly means “lacking cognitive ability and education overall” and is often indelicately translated as “low quality.” Many of our enumerators were concerned that villagers would not understand or would not have considered the issues in the questionnaire, rendering the data meaningless. This attitude presented two challenges. First, we had to convince the enumerators that this preconception was neither relevant nor accurate, so that enumerators’ biases would not affect how they asked or recorded questions. Second, enumerators would have to actively pre-empt rural respondents’ expectations that villagers were regularly perceived as inferior, having often experienced urban hauteur. Enumerators had to convince respondents that they were trustworthy and that they respected villagers’ contributions while asking them personal and sensitive questions. We worked intensively with enumerators on how to build a rapport with villagers.

Our analysis of the HEPS data demonstrates first, that with careful survey design, NSR need not prevent survey research in an authoritarian regime, and second, that villagers’ individual characteristics can partly explain the likelihood of an NSR but in some unexpected ways. There are two major factors that would lead us to expect a higher rate of NSR among Chinese villagers as compared to respondents in a democratic context. First, interviewees may fear social and political costs, including retaliation from officials, for responses that challenge the status quo of state control. In rural areas, most villagers have never participated in a survey and may not feel confident that their responses will be kept anonymous and confidential, which compounds pre-existing fears of political persecution. The second factor relates to the aforementioned discussion of suzhi di and lower levels of education in the countryside. A low level of education may mean that respondents have limited cognitive resources to process and evaluate the question, which has been established as a source of higher NSRs elsewhere.Footnote ¹⁹

Explaining NSR: Hypotheses

Previous research in the US has found that respondents with fewer socio-economic resources are more likely to give a non-substantial response.Footnote ²⁰ Following this research, we hypothesize that villagers will be less likely to give an NSR if they occupy a relatively privileged position in society – that is, if they are male, have a higher level of education, are more cosmopolitan and a Chinese Communist Party (CCP) member. Previous studies of nonresponse in China have found a “gender effect” that leads to higher rates of NSR among women, but the reasons for this effect are unknown.Footnote ²¹ We hypothesize that:

Previous research indicates that education is inversely associated with the likelihood of giving an NSR.Footnote ²² Scholars in China typically affirm that this factor is even more salient in the context of rural China because villagers have suzhi di, yielding Hypothesis 2:

We hypothesize that CCP members will be less likely to provide an NSR. CCP members are more likely to have interactions with higher-level government officials and attend Party meetings that provide them with broader perspectives. Because the CCP is an elite party (about 6 per cent of the population) and recruitment strategies deliberately select more educated individuals, CCP members tend to be more educated and wealthier than non-members.Footnote ²³ In our sample, fewer than 10 per cent of villagers are Party members (see Table A.1). We hypothesize that even when controlling for education, Party membership will be associated with lower rates of NSR because Party members are more cosmopolitan. CCP members may also feel more confident to critique the government as part of the party-state apparatus because their status is more secure and supported by their networks. Or, they may choose to strategically inflate their evaluations of the government, rather than give an NSR.

The Chinese concept of suzhi di encompasses more than an individual's educational attainment; it connotes that an individual is parochial, that is, not exposed to issues beyond a small and limited community. Therefore, we expand on H₂ by testing whether villagers who are less parochial – or more cosmopolitan – are also less likely to provide an NSR. We measure a villager's degree of cosmopolitanism through several indicators depending on the variables available in each survey. These include having engaged in migrant labour, having family or close friends in a large city or foreign country, following major foreign events, using the internet at home or at an internet café, frequency of watching news on TV, and frequency of reading the newspaper. Migrants who leave the village to work in a larger city will be exposed to (but excluded from enjoying) higher standards of living. Villagers who have close friends or family in large cities or abroad are more likely to have heard about life beyond the village and to have considered larger questions about politics and society and, therefore, likely to have more cognitive resources to respond to complex survey questions. Similarly, villagers who follow foreign events, use the internet, watch TV news or read the newspaper are likely to be more cosmopolitan. Based on this logic, we propose Hypothesis 4:

Explaining NSR: Findings

We tested these hypotheses using questions about whether government officials are trustworthy.Footnote ²⁴ We constructed a variable for NSR by coding a substantial response as 0 and an NSR (“do not know,” “refuse to answer,” or missing) as 1. As available for each survey, we included relevant control variables: age, age-squared, whether the respondent belongs to an ethnic minority,Footnote ²⁵ whether the respondent was born in the village (local), how trustworthy the respondent believes other villagers are, whether the respondent had prior interaction with the officials in the question, experience with corrupt officials, and an index to capture innate propensity for an NSR. Where available, we included county or provincial fixed effects to capture unknown factors associated with the locality.Footnote ²⁶ Details regarding the questions and variable measurements for each survey are provided in the online Appendix. Tables 1 through 5 summarize the results of the logistic regressions for each survey.

Table 1:

Logistic Regression of Non-substantial Response in HEPS, 2012

Source:

Authors’ survey of Chinese villagers.

Notes:

*p < 0.10, **p < 0.05, ***p < 0.01.

Table 2:

Logistic Regression of Non-substantial Response in Lianjiang Li's Survey, 2014

Source:

Li Lianjiang's survey. Data are used with permission.

Notes:

This dataset did not provide adequate information to use provincial or county fixed effects. * p < 0.10, ** p < 0.05, *** p < 0.01.

Table 3:

Logistic Regression of Non-substantial Response in the Asian Barometer Survey, 2011

Source:

Asian Barometer Wave 3, www.asianbarometer.org. Data are used with permission.

Notes:

The Asian Barometer does not provide adequate information to use provincial or county fixed effects. *p < 0.10, **p < 0.05, ***p < 0.01.

Table 4:

Logistic Regression of Non-substantial Response in the China Survey, Rural Subset, 2008

Source:

China Survey. Available upon request through Texas A&M University, https://pols.tamu.edu/data-resources/. Data used with permission. The China Survey is a project of the College of Liberal Arts at Texas A&M University, in collaboration with the Research Center for Contemporary China (RCCC) at Peking University.

Notes:

Provincial fixed effects are used rather than county fixed effects because the number of observations for each county would be very small. *p < 0.10, **p < 0.05, ***p < 0.01.

Table 5:

Logistic Regression of Non-substantial Response in the SSMSC, Rural Subset, 1993

Source:

Shi, Tianjian. 1993. “Survey on social mobility and social change,” available at the China Archive at Texas A&M University, http://chinaarchive.tamu.edu/portal/site/chinaarchive/.

Notes:

The dataset does not provide adequate information to use provincial or county fixed effects. *p < 0.10, **p < 0.05, ***p < 0.01.

Overall, our analyses support the argument that marginalized groups are more likely to give an NSR, with some exceptions. Women are more likely to give an NSR in three of the five surveys (Asian Barometer, China Survey, SSMSC), supporting H₁.Footnote ²⁷ Education is negatively associated with giving an NSR in four surveys (HEPS, Asian Barometer, China Survey, SSMSC), supporting H₂. That said, in the model of trust in village leaders for the China Survey and in two models of Lianjiang Li's survey data, more educated respondents were more likely to give an NSR, suggesting the need for further research on the role of education and survey participation. Party membership is negatively associated with giving an NSR in three surveys (HEPS, Lianjiang Li, SSMSC), supporting H₃.Footnote ²⁸

Does this, then, confirm the suspicions of those who claim that villagers have suzhi di? In fact, measures of cosmopolitanism reveal a more complicated relationship. Villagers who followed foreign events regularly were less likely to give an NSR for the Asian Barometer and villagers who watched more TV news were less likely to give an NSR for the SSMSC, suggesting that more cosmopolitan villagers are less likely to give an NSR, as expected. However, other measures indicate that more cosmopolitan villagers are more likely to give an NSR. Villagers who have close friends or relatives in large cities or abroad were more likely to provide an NSR in HEPS. In Lianjiang Li's survey, villagers who used the internet at home were less likely to give an NSR, although villagers who used the internet in an internet café were more likely to give an NSR. Other measures did not have a statistically significant association with NSR, including having undertaken migrant labour (HEPS), contact with foreigners (China Survey) and frequency of reading a newspaper (SSMSC). Thus, these data do not support a clear association between cosmopolitanism and NSR, although this issue merits further study.

Implications for Estimates of Political Trust

We have established that marginalized individuals tend to be underrepresented in surveys in China, but does their absence affect substantive conclusions? We examine the substantive implications of our findings in two ways: first, by examining whether marginalized individuals tend to hold different views as compared to their more privileged counterparts, and second, by comparing estimates of political trust with and without NSR.

First, if marginalized individuals tend to have systematically different opinions as compared to their more privileged counterparts, then we may misjudge public opinion in China.Footnote ²⁹ We use the HEPS data from those who responded to examine whether marginalized individuals – women, non-Party members and the less educated – hold different opinions regarding the trustworthiness of government officials. Table 6 summarizes these results.

Table 6:

Expression of Distrust in Government Officials in HEPS, 2012

Source:

Authors’ survey of Chinese villagers.

Notes:

Bolded percentages indicate that the difference in the proportions expressing distrust between the two categories is statistically significant at the p < 0.05 level based on a z-test. Percentages calculated based on those who responded to the question. NSRs (“don't know,” refuse to answer, and missing data) are excluded.

Focusing on the statistically significant differences in bold in Table 6, women are more likely to express distrust in the central leaders; non-Party members are more likely to express distrust in village and central leaders; and less educated villagers are less likely to express distrust in government leaders above the village level. Overall, marginalized individuals hold different views as compared to their privileged counterparts. Imputation may reduce the effect of nonresponse bias but we remain concerned that the complexities of respondents’ views on sensitive issues and the possibility of preference falsification could reduce the effectiveness of this technique.

Second, failing to account for NSR may have caused scholars to overestimate political trust and exaggerate the gap in trust between central and local leaders. Table 7 and Figure 3 show the proportion of respondents who replied that they trust or approve of the government, according to each of the questions that we studied in the five surveys (see online Appendix for details). In Figure 3, the dark grey bars show this statistic with NSR included in the calculation, whereas the light grey bars show the resulting higher proportion when NSR is excluded. Once NSR is taken into consideration, trust and approval of government decrease in all the surveys we analysed. Focusing on instances where the estimate dropped by at least 10 per cent, trust in the central government decreased by 14 per cent (China Survey), 31 per cent (HEPS) and 10 per cent (Li) when nonresponse was included in the calculation, with similar reductions in the estimates of trust in the province (HEPS, The China Survey) and county (HEPS, The China Survey). In the SSMSC, the proportion of respondents that indicated a positive impression of the local people's congress and the National People's Congress diminished by 19 per cent and 20 per cent, respectively, when NSR was included.Footnote ³⁰

Figure 3:

Trust in and Approval of Government in Rural China, with and without Non-Substantial Response

Sources:

SSMSC; the China Survey; Asia Barometer Survey; HEPS; Li Lianjiang's survey.

Notes:

Figure shows the percentage of respondents who replied that they trust or approve of the government. The dark grey bars show the calculation as a proportion of all respondents, including NSR (don't know, refuse to answer, missing), and the light grey bars show the higher proportion that results if NSR is dropped from the sample.

Table 7:

Trust and Approval of Government with and without NSR

Sources:

The China Survey; HEPS; Lianjiang Li's survey; Asian Barometer Wave 3; SSMSC.

Notes:

For the SSMSC, the percentages indicate the proportion of respondents who agreed with the statement for questions 61J and 43D. For 58E, 58F and 58G, the percentages indicate the proportion of respondents who said that the people's congress or government officials were “good” (see Appendix for further details on these questions).

Political trust in China still appears to be relatively high but these results call into question the received wisdom. Focusing on the China Survey, HEPS and Lianjiang Li's survey, trust in central leaders may range from 62 to 80 per cent, as compared to 89 to 90 per cent if NSR is dropped. Meanwhile, estimates of political trust at the local level are less affected by NSR. Trust in the village leaders may range from 60 to 65 per cent with NSR included, as compared to 67 to 71 per cent with NSR dropped. Averaging the results of these three surveys, without accounting for NSR, we might conclude that 90 per cent of villagers trust central leaders, while only 69 per cent trust village leaders. However, the new estimates with NSR included suggest that only 72 per cent trust central leaders and 63 per cent trust village leaders – a far less dramatic gap. Thus, inattention to nonresponse may have caused scholars to both overestimate political trust and exaggerate the gap in trust between central and local leaders.