Background: The Boom in Democracy Surveys

The dramatic expansion of democratic regimes throughout the world has produced a boom in the field of survey research. There are at least six reasons for this boom. First, democracy brings with it elections, and with elections, parties and candidates who want to know where and how to campaign and contributors who want to know on which “horse” to place their bets. Second, democratic governments care very much about public opinion since not only does their reelection depend upon the public will, but their ability to govern depends to a great degree on how well they are able to gauge public reaction to their policies. Third, democratic governments want objective information to help them plan their programs and to be able to gauge their impacts once implemented. Fourth, international donors increasingly carry out surveys of “users” or potential users of public services to help them plan their investment strategies.¹

Fifth, international donors regularly examine program impacts as a means of evaluating project success and as a means of targeting future grants and loans. Finally, within the field of political science at least, there is a growing consensus that political culture matters for sustaining democracy and that the entire “democracy game” goes beyond finding the right institutions to having citizens believe in democratic principles. There is, of course, no unanimity on this point, raised so forcefully years ago in Dahl's “Preface to Democratic Theory,” and there will be those among the “new institutionalists” who entirely dismiss the role of what Dahl called the “consensus on the polyarchal norms” (1956, 135). Yet, broadly speaking, it is fair to say that most democracy experts would agree that in democracies, the public matters; publics vote, protest, and even rebel, and to exclude them in the calculus of the study of democratic consolidation is to risk missing an important part of the story.

Social scientists, it turns out, are major beneficiaries of this boom in polling in the developing world. However, cost is a major factor constraining our abilities to advance the frontiers of science. Survey methodology is very costly to implement in advanced industrial countries. Indeed, the cost per interview has grown so high that few surveys are conducted face-to-face. Instead, social scientists in the United States and Western Europe rely heavily on telephone interviews, self-administered mail questionnaires, or one of several variations of Internet polling. Yet, we are all aware of the problems in alternative modes of survey research; in some phone surveys a very high percentage of all calls are diverted by answering machines, callblocking devices, or flat refusals by respondents, not to mention the growing challenge of cell phone-only households. Since cell phones are by definition mobile, designing a sample that includes cell phones is extremely difficult since one does not know precisely into which sampling unit each cell phone belongs. Without that knowledge, it is very difficult to define sample parameters and determine sampling errors. A further problem in all phone surveys is that both non-contact and refusal rates are increasing. Moreover, the duration of interviews in telephone (and Internet) surveys is constrained since it is very difficult to prevent respondents from hanging up when the interviews go beyond 15 or 20 minutes. This forces researchers to limit the data that they can obtain from each respondent. Survey research faces a special constraint in much of the developing world where large portions of the population, especially the poor, do not have phones and therefore are missing from surveys that exclude door-to-door interviewing. Mail surveys have certain advantages, including cost, but it is difficult to ascertain who actually responded to the questionnaire in a multiple-individual household, and mail questionnaires assume a lingua franca which may not be the operative language for minority populations. In the developing world, postal systems are often unreliable, household addresses unreliable or unknowable, and significant proportions of the population illiterate or only semiliterate, thereby excluding them entirely from the sample.

For all these reasons, face-to-face surveys in the developing world have become especially attractive; in much of Latin America, the region I know best, face-to-face interviews of an average of 45 minutes can be conducted for about $15–$25 each and sometimes less, including costs of sample design, coding, and data entry. This is in contrast to prices averaging 10 times that for comparable, in-person interviews in the United States. Cost varies, of course, by country depending on exchange rates, labor costs, country size, and population density. It is far cheaper, for example, to carry out a national sample in El Salvador (only 21,000 Km² and 292 persons per square kilometer), than in Bolivia (1.1 million Km² and 7 people per square kilometer) (World Bank 2000, 10).²

Yet this variation is trivial when compared to the cost of carrying out face-to-face interviews in advanced industrial nations. Elite interviews, which one normally expects to carry out face to face, are almost always far less costly in developing than in the advanced industrial nations—again, because of lower labor costs.³ In short, dollar for dollar, one can learn far more about a far wider range of national populations by carrying out research in the developing world rather than in the advanced industrial world. If we seek scientific advancement, the ability to find patterns and the information to make generalizations, then surveys in the developing world, ceteris paribus, will bring the most rapid progress.

The Dark Side of the Boom in Surveys: The “Normalization of Deviance”

The boom in survey research in the developing world has led researchers in both scholarly and applied settings to, in a number of cases, incorrectly assume that the data being generated in the developing world are of the same quality as that from first world sources. Some recognize that the quality is not as good as the data that they have been accustomed to analyzing, but they believe that it is at least of acceptable quality. In fact, as this article shows, there is reason to be very concerned about the quality of the data emerging from those surveys. Even when analysts know that the data are not sound, some nonetheless argue that “it is good enough, and poor quality data capturing some of the reality of a country about which nothing is known is better than none at all”—an argument to which I take strong exception. Using poor quality data inevitably leads to a greater incidence of “Type II” errors by which a null hypothesis (i.e., that there exists no significant relationship among a given set of variables) is falsely accepted.

In many ways, the situation we face today is not unlike the picture described by Diane Vaughan in her brilliant analysis of the Challenger launch decision that ended up killing the entire space shuttle crew and destroying the spaceship (Vaughan 1996). Even though the O-rings that sealed the booster rockets had suffered various degrees of damage in seven of the launches prior to the fatal 25th launch that ended in tragedy, the launches were allowed to continue. At NASA, she argues, the engineers and scientists had come to accept a “culture of the normalization of deviance,” so that deviance in the performance of the spacecraft became acceptable, indeed normal. Tragically, even though the O-ring problem was corrected after the Challenger disaster, the cultural problem was not; in 2003 foam falling from a fuel tank during launch, traveling at some 500 mph, smashed into the wing of the Columbia causing a break in its skin that caused it to burn up on reentry, once again with the complete loss of the crew. Some reports state that foam had fallen off of the fuel tanks in all prior launches of the space shuttle, but because nothing had happened until the Columbia disaster, such deviance was accepted as “normal.” Are we making the same error by accepting survey data with little or no evidence of its quality? Is the temptation to use a “free” data set that we can download from one source or another so great that data quality issues are ignored and that known violations of basic survey research norms are becoming accepted as the norm? Do we view such data as being “good enough”? Do we sweep these concerns under the rug and tell ourselves that the difficulties of conducting survey research in the conditions of the developing world are such that we just need to be content with what we get? If so, both social science and social policy will surely suffer, producing consequences that, while far less dramatic than the shuttle disasters, could seriously affect the lives of hundreds of millions of people.

Researchers must weigh the scientific benefits, owing to their striking cost advantage, of carrying out surveys in developing nations against the important hidden costs of how surveys are actually carried out in these countries. Today, surveys can easily be contracted to thousands of universities, government agencies, think tanks, and private firms in the developing world, which can transmit the data file obtained (normally in SPSS format) through email attachments. The convenience of receiving a nicely formatted and labeled data file, however, can mask serious errors in the data collection process. The social scientist accustomed to downloading the latest National Science Foundation National Election Study (NES) from the University of Michigan is likely to be in for an unsettling surprise with much of the survey data collected from the developing world. One cannot automatically expect that the quality of the data will be anything like the norm for the NES, General Social Survey from NORC at the University of Chicago, the Eurobarometer, or similar surveys on which modern social science have come to rely. Poor quality survey data can also be obtained, of course, in the United States, as it can be in Western Europe, but in the developing world, good quality survey data sets are more generally the exception than the rule.

Recounting my personal experiences with survey data in Latin America I hope will effectively illustrate the common problems that I expect are similar to those that would be encountered in much of Africa, the Middle East, and some regions of Asia. I first became aware of the quality problem a number of years ago when I was asked to carry out a multi-country survey in Latin America. Prior to that time, I had been accustomed to close, personal involvement in the design and fieldwork. As a graduate student, I planned and carried out the dissertation field research myself, as was then the norm (increasingly today, with the widespread availability of survey data sets housed in archives, students write dissertations based on surveys conducted by others). This meant obtaining the census data and the census maps, selecting the sample (that first time with the help of Leslie Kish, the author of the classic work on modern survey sampling [Kish 1965]), and then conducting the interviews (along with my wife, who was planning to use some of the data for her own Ph.D. dissertation). The experience of interviewing a sample of 531 respondents (a number I will never forget) proved invaluable for future research. I learned, for example, that census maps are not always up-to-date and respondents selected to be interviewed are often not at home. I also learned that some neighborhoods or dwelling units looked too foreboding to enter, and we had to force ourselves, on occasion, to overcome fears and enter places where there was reason to be concerned for our safety. I discovered that non-response is a common occurrence when respondents are first read a question, and that the well-developed techniques taught to me at the University of Michigan for using non-directed probes to increase response rates to questions are extremely valuable. I learned the hard way about errors in data entry and the need for verification of each questionnaire. In short, it became obvious that ignoring any one of a wide variety of problems in the data collection process for survey research can seriously affect the quality of data in many ways and produce misleading results.

Later in my career, when the magnitude of the survey tasks expanded, to cover multiple nations, I began carrying out surveys by working closely with my bilingual graduate students, training them, supervising their work, and having them travel to the countries involved, where they would draw the samples, recruit and train the interviewers, supervise the pretests, and insure the quality of coding and data entry. Eventually, however, I began a project in which the size of the samples, the number of countries involved, and time deadlines became too great for hands-on supervision of the training and fieldwork; the project had exceeded the capacity of the graduate students working with me at the time. As a result, for the first time, I contracted with a firm local to the region in which the survey was to take place. The firm was not only well established but was affiliated with one of the large transnational survey firms with affiliates all over the world. When the files arrived I was aghast. As soon as I began running frequency distributions on the variables I discovered that approximately 15% of all responses were out of range codes, that geographical locator codes were often meaningless, and that the files and the printouts sent by the firm did not match. When I confronted the firm with these findings, I was told, “This is the first time that anyone has ever asked us for the raw data, and it is our policy to recode all out-of-range codes to missing so that the printouts we send look clean. That is what our clients expect and demand.”

That experience compelled me to take a much closer look at the firms I hired in the future. Let me describe my recent experiences that raise alarm bells for me about the quality of surveys in the Latin America region—experiences that I think apply to other developing regions of the world.

Experiences

Recently, when planning to carry out a national sample in a country, I decided to target the university survey institute that was carrying out the periodic employment surveys for the central bank of that country. My assumption was that if the central bank was relying on this university to estimate employment/unemployment figures (a fact that the university prominently advertised), then I could be confident in their sampling procedures. In discussing my sample needs with the head of the survey institute, I asked how he handled household selection within a given primary sampling unit (PSU). I was told: “We send the interviewer to the neighborhood selected and then she determines which doors to knock on and who to interview in each home. Our interviewers know their neighborhoods.” This means, of course, that not only are the respondents not probabilistically selected (or even selected by some sort of quota system), but the unemployment figures for that country produced from these surveys may well be wildly inflated. Respondents found at home are far more likely to be unemployed than the population as a whole. They are also more likely to be female than male and very young or very old rather than representative of the national age pyramid. In short, the data emerging from surveys based on that sample design would systematically be unrepresentative of the population of the country.

• In another case, where a deadline was very important for my survey, I was directed to a firm with an excellent reputation for getting the work done on time. I learned that they could deliver 1,500 interviews—all conducted on the same day—and could do so by having a team of 1,500 interviewers, each conducting one interview. No supervision and no training. No rules were to be applied by the interviewer who was selecting respondents. Sadly, this firm produces survey data that is being used by major international donors to develop nationallevel governance indicators.
• For another project, the survey firm was selected for me, so I had to work with what I was given. Initially I was optimistic about the outcome since the firm had the reputation of being the leading survey firm in the country. But, when we got into a technical discussion of the questionnaire and sample design, major problems emerged. I was told, for example, that no in-home survey could be conducted in which interview time was longer than 10 minutes. I explained that from a cost-effectiveness point of view, the main costs were accrued in locating the correct respondents and that the additional benefits of adding some more minutes to the survey far exceeded the costs. Their response was that the people in this country were psychologically incapable of answering more than 10 minutes worth of questions. I was also told that a “good” sample of the country would not need to include significant numbers of respondents from the highlands because those people, many of them indigenous, did not really count. When I argued these points (and others), I was warned very firmly to back off, since the director of the organization held an M.A. from the John F. Kennedy School of Government at Harvard. Apparently, those credentials were supposed to tell me all I needed to know.
• Another project required that I search for a firm that would be especially sensitive to questions of ethnicity; I was directed to the leading NGO with survey experience in the country, one that came highly recommended to me by anthropologist colleagues in the United States. In this country the census data were many years out of date; when I asked their technical team how this problem would be dealt with, I was told that if any of the sample segments fell in regions that had been unpopulated in the last census (such as the peripheral areas of major cities), they would correct for this omission by choosing a segment in the center of the city where lots of people live. When I suggested that this might underrepresent the recent urban migrants, who tend to be poorer and less well educated than those who live in the urban center, they said that they had never thought of that problem before and could not suggest a way to deal with it. Apparently the notion of updating census maps had never occurred to them.
• On another occasion, I was directed to an organization that touted its long-standing association with the U.S. Bureau of the Census, which had invested heavily in designing a national sampling frame for this organization. When I asked them to design a sample that varied from the U.S. Bureau of the Census template, one covering different regions, each with a given confidence interval for sampling errors, they told me that the U.S. Census expert no longer gives them advice and they did not know what sampling errors were or how to vary the design that he had left with them.
• In another case, a firm presented its sample design in which they had decided to select two urban segments for every one rural. When I asked if in fact the urban population was known to be twice as large as the rural, they responded by saying “no,” but rural folk have problems answering survey questions, so they did not want to waste a lot of time interviewing them.
• In another study, I was asked to reanalyze a survey by an organization that was a well-known demographic research arm of a major university. The long-standing tradition in demographic surveys is to include modules that only a subset (normally half) of respondents answer so that the total number of variables studied can be increased. When the results of a commissioned democracy survey were turned over to the international donor, however, the report did not distinguish between non-response and questions not asked, so the international donor saw response rates lower than 50% on large numbers of items referring to opinions on democracy. The baseless conclusion that they drew was that in that particular country, the population was very uninformed about democracy and uninterested in politics. In fact, the high non-response rate was merely a reflection of the fact that half of the respondents were never asked the questions of interest to the donor.

More generally speaking, when it comes to interviewer training, I would say it is almost the rule for firms in Latin America to train their interviewers to take the questionnaire as a loose guide and to give the interviewer a great deal of flexibility to change the questions. They train interviewers by telling them: “Put it in the words that the people will understand.” When I argue against this policy, I am told, “we have very good interviewers, who know what they are doing.”

Causes of the Data Quality Problems

This list of problems in surveys conducted in Latin America could go on, and I doubt that my experiences over the past 15 years (I have been doing surveys in Latin America for 30 years but using firms only for the past 15) are atypical. Rather, I think they reflect the growing pains of a region in which surveys used to be a rarity and suddenly have become the coin of the realm. Moreover, I suspect that these problems are also found, to varying degrees, elsewhere in the developing world.⁴

There are two major causes for the problems of quality in this field, one deals with the supply side and the other with the demand side. On the supply side, that is, the supply of organizations who offer to carry out surveys in the developing world, the dearth of welltrained experts in the field of survey research is a serious problem.

This problem gets to the demand side, those paying for surveys, the second factor affecting quality in survey research. The credibility of survey findings within the developing nations is a serious issue. Local firms live in a world in which pre-election surveys have become a high-profit line of business, and represent among one of the largest sources of demand. In Latin America, many newspapers and even some TV stations have formed alliances with firms to commission regular surveys, similar to the CBS/New York Times poll. Unfortunately, the media in Latin America and in many parts of the democratizing world do not yet have the traditions of independence in news reporting enjoyed by the major media sources in the United States. We need to recall that until recently in the United States unconscionable political influence of newspapers was common (as the infamous case of New York's “Boss Tweed” so well illustrated). Today, however, when we hear results from a CBS/New York Times poll, or a recent Gallup poll, few of us think that they have been “cooked” to favor one party or the other. Sadly, this is precisely what many Latin Americans think when they read poll results in their newspapers, and even more sadly, many times they are right. Political parties carry out surveys and release data to the media, or to “their media,” that show their candidate far ahead in the race in the hope of generating a pollbased “coat tail effect.”⁵

When the returns ultimately reveal radical differences between the poll results and the actual vote, readers lose faith in survey results, sparking a credibility problem that is difficult to overcome.

What Is to Be Done?

I have thought about solutions to help resolve the problem of data quality for a long time and recently I organized a series of meetings at the University of Michigan, the Midwest Political Science Association (Chicago), and the Wilson Center in Washington, D.C. to help think through them. Based on that input, I have come to consider the possibility of a dual track approach, one emphasizing both a short-term and a medium-term solution. The short-term solution involves the establishment of international standards for surveys. The medium-term solution would focuses on building human capital in the survey research area.