Shortly before writing this review, I watched a charming but perplexing movie. It mixed styles with abandon, deploying tropes from buddy movies, slapstick comedy, road trip films, social satire, and even magical realism. The disparate genres bumped up breathlessly against one another, without transitions, leaving members of the audience grumbling about what exactly they were watching.

I experienced the same tension when I read Robert Wuthnow's book, Inventing American Religion: Polls, Surveys, and the Tenuous Quest for a Nation's Faith. The book comprises an assault (possibly justifiable) on commercial polling about religion, a somewhat well-worn argument against positivism, a confession of faith in response rates, a paean to qualitative research, and more. By the end, I still couldn't see the point of the author's thesis that polling has a deleterious effect on how we view religion. It seemed a thin rationale for such a sustained assault on pollsters.

I came to this book with predispositions. I have enormous respect for Robert Wuthnow whose scholarship has enriched my understanding of American religion even when I haven't agreed with him. I am also probably one of few social scientists who has routinely taught a graduate seminar on religion and politics and another on survey research methodology, so I share the author's concern about the treatment of religion in polls and surveys. Finally, although I do run survey data, I also soak and poke in tangible religious environments and have spent a lot of time in musty archives to get a deeper understanding of my research topics.

While reading the book, I engaged the author in a running argument. The copious comments scribbled on the pages of my copy included (slightly snarky) admonitions/challenges to the author, indignation about several broad claims resting on slender evidence, identification of occasional factual errors, admiration for pithy insights conveyed in elegant prose, even some introspection about my own scholarly assumptions. I'm glad Politics and Religion has given me a chance to bring that internal conversation into public space.

Nicely setting the tone, the book starts by noting a disjuncture connecting two buildings on Wuthnow's Princeton campus. Nassau Hall, where John Witherspoon taught moral philosophy to members of the American founding generation, is the birthplace/antechamber where an authentic American religiosity was born, nurtured and transmitted to posterity by Witherspoon and his students. They forged a faith anchored in the belief that “the [divine] rights of individual persons must be the basis for the nation's fragile understanding of democracy” (2). But the building across the street from Nassau Hall once housed another “icon in American religious history” (3), George Gallup, whose pioneering polls about American religion seemingly reduced this vibrant inheritance to mere “numbers and percentages and notions of what is typical and what is atypical …” (4). Thinking of these two buildings standing a stone's throw from one another, it's not hard to visualize the direction that Wuthnow would like the rocks to fly. One gets the impression of a fall from grace.

The remainder of the book asserts the baneful influence of the polling industry on popular understanding of American religion. The second chapter nests the viper (my term) that would become polling in the idealistic social survey movement of the early twentieth century, a campaign to collect systematic information about religious communities in the service of evangelism and, like other Progressive Era innovations, to bring business-like efficiency to church outreach. As chapter three recounts, this impulse was eventually transmuted into an effort to comprehend religion through scientific polling. This shift, Wuthnow concludes, produced a tendency to reify the numbers and percentages derived from poll questions into something called “American Religion.” Gallup, an active church layman, was the foremost advocate of this new way of thinking about religion in the United States.

Wuthnow turns in chapter four to the roughly parallel development of academic studies of religion in the post-World War II era. While commercial and academic survey researchers favored similar methods, the latter were much less interested in marginal frequencies (e.g., what percentage of Americans believed in God), favoring deeper inquiries that disaggregated survey results and emphasized the contexts in which religion was lived. Wuthnow cut his own academic teeth as a student of Charles Glock and Rodney Stark who deployed large surveys of church members in California. Despite optimism that such intensive studies would lead to law-like statements about the social impact of religion, many scholars were disappointed by the elusiveness of generalizations about religious effects on behavior. Due to scarce funding to support their research, academic scholars of religion were often forced to depend on the very polls that they had considered superficial. That increasingly meant a scholarly focus on large, nationally-representative samples that obscured much of the local diversity in the religious landscape.

Returning to commercial pollsters in chapter five, Wuthnow documents their transformation from collectors of scientific information to pundits and prophets. Much of this can be explained by the resurgence of religion in American politics. Influenced by the election of a Catholic president in 1960 and seemingly blind to the religious foundation of the civil rights movement, many scholars had assumed that religion no longer mattered in American political life. The political mobilization of evangelical Protestants in the late 1970s and 1980s suggested otherwise and pollsters quickly began fielding surveys about what became known as the New Christian Right. Unfamiliar with the base constituency for this movement, pollsters badly mis-measured it and their poll results seemed to offer bizarrely inconsistent estimates about the size, beliefs, and popularity of the protagonists in what was called the Culture War.

Chapter six reviews the crisis of confidence afflicting public opinion research in the late 20th century, a development which Wuthnow claims also reduced public confidence in polling about religion. Although more commercial/private polling was being done about religion, polling as an industry lost credibility. Even as the media remained eager to publicize poll findings about religion, the techniques that yielded the data confronted growing doubts about accuracy and trustworthiness. Most ominously, scientific research seemed to demonstrate that respondents’ answers to survey questions varied with virtually every element of survey design — mode (how the respondent was contacted) and instrument (question wording, order) in particular — and were notoriously unstable over time. Were surveys measuring signal or noise?

While chapter seven summarizes how interested parties — church leaders and other religious elites — make sense of (or don't) poll findings about their “industry,” chapter eight is essentially a jeremiad that takes stock of polling about religion and, true to the genre, finds its defects more glaring than its virtues.

Let me be clear that I share many of Wuthnow's misgivings about the commercial polls on religion. In fact, the book brought to mind screeds such as Lindsay Rogers’ acerbic The Pollsters or Michael Wheeler's outraged Lies, Damn Lies, and Statistics. The poor quality of reporting about polls on religion should come as no surprise to readers of Philip Meyer's Precision Journalism. This clarion call for sophistication in reporting social science data apparently influenced journalism professors more than working journalists. Anyone who draws conclusions about religion in the United States from news reports (even more so from blogs) will be sadly misinformed about the subject. The lack of nuance and comprehension remains striking decades after religion gained renewed power as a political factor.

Developing his own uncertainty principle, Wuthnow contends that polls about American religion created the very concept they purported to study. This central claim of the book is amplified by its subtitle. The idea of “American religion” conveyed by pollsters assumed a constancy and uniformity that never existed. George Gallup and his fellow pollsters missed the rampant pluralism of American religiosity because they largely ignored the “local, personal and familial identities” in which religion flourished (65). That omission could be traced to the pollsters “external” stance, which prompted them to look for religion in all the wrong places. By asking about opinion rather than personal experience (67), polling produced an image of religion in America that was demonstrably untrue to what scholars would later call “lived religion.”

Like Rogers, Wuthnow does not merely criticize the state of the art in polling. His criticism is much more fundamental. Polls (whether called surveys or not) may be sufficient to measure political opinion but they are incapable of assessing in any meaningful way “the deeply personal, complex, highly diverse beliefs and practices embedded in the many religious traditions of the nation” (4). Forget for a moment the implied assumption that political attitudes are simple, impersonal, coherent, and consensual: Wuthnow offers a broader epistemological critique of positivism as arid and utterly insensitive to the real worlds inhabited by people who practice (or imagine themselves as practicing) American religion. The whole survey enterprise directed to religion, not just poor quality commercial work, is doomed to failure. The preferred solution to this problem, I infer from his tone, is more thick description, participant observation, grounded theory, ethnography, and other forms of qualitative research which can assuredly penetrate the interior religious core.

It bears repeating that Wuthnow delegitimizes not just low-hanging fruit like commercial polls but even the best academic surveys which, he says, are afflicted with fatal shortcomings. He is particularly exercised about threats to validity resulting from the low response rates of the modern era. How can surveys that obtain responses from 9% of target samples possibly represent the views of large populations? The claim to external validity that has always been the principal argument in favor of surveys now appears unsustainable.

The issues raised by this book cannot be confronted adequately even within the generous page limits allotted to this review so let me simply raise some concerns.

I start with Wuthnow's thesis that commercial polls have done harm by misconstruing and misrepresenting the nature of religion in the United States. Wuthnow avers that pollsters have presented a misleading picture of the nation's true spiritual life by assuming consensus and drawing narrow boundaries around the religious field. The Gallups (father and son) certainly offered facile commentaries on religion in the United States and worshipped their marginal frequencies, unwilling to recognize that “referendum-type” survey questions are saturated with measurement error (“Ordinary Questions, Survey Questions, and Policy Questions,” Public Opinion Quarterly, vol. 50, no. 3 [1986]).

Even if that is so, where's the beef? Wuthnow is not very specific on how these portrayals did harm to religion apart from calling attention to some facets of religion and ignoring others. Granted, we'd prefer polls not to distort reality but it's a stretch to assume that what pollsters report about religion frames what audiences think about it. Indeed, Wuthnow provides some fairly persuasive evidence about the limits of polling's influence on the public and religious elites. Given how little Americans know about the content of religious traditions (Religious Literacy: What Every American Needs to Know — and Doesn't), including their own, it's unlikely that polls drive public understanding of religion or popular religiosity.

Can we measure religion meaningfully through survey methods? It may be that religion has a core of some kind that is too deep for survey methods to penetrate. It is equally possible that most individuals do not have something that we could label as meaningful religious attitudes and are incapable of articulating the meaning of faith to an observer regardless of research method. Decades ago, Philip Converse argued that most Americans did not have political attitudes that could be dignified as such, that their political views even on longstanding issues were so evanescent as to be called “non-attitudes” (Ideology and Discontent). Subsequent research has suggested that people carry around in their heads a bundle of “considerations” about politics that can be called to mind (or not) by the survey interview (“A Simple Theory of the Survey Response: Answering Questions Versus Revealing Preferences,” American Journal of Political Science, vol. 36, no. 3 [1992]). These ideational strands may have a central tendency and dispersion that amounts to a kind of attitude. Religion may well be experienced in the same manner.

While I would be the first to agree that much about religion is mysterious and probably not amenable to either surveys or qualitative interviews, I do think that we can and do ask useful questions about how people try to live and practice their religious traditions and that such views are increasingly transmitted to the public through quality journalism. In the field of political behavior, we've developed a rough kind of consensus on the value of measuring the three Bs — belief, belonging, and behavior. The beliefs can encompass a range of matters that seem relevant to public issues while belonging and behavior are also accessible to well-designed survey questions. “Religiosity,” once a composite term for all things religious or for discrete forms of behavior, is now seen as one component that encompasses such diverse activities as attendance at worship, private devotionals, encounters with other members of the religious community, and a range of other phenomenon that are not only cognitive but behavioral as well. We also have emphasized the role of context, the places, and times where religion is shaped and its implications for secular life. For the most part, we begin with congregations as the operative environments for political learning and have now moved more deeply into informal social groupings within the religious subcommunity (The Political Influence of Churches). I don't know whether Wuthnow believes that these constructs can be plumbed with standardized interviews but they do tap, I think, “the “manner and form in which religious phenomena appear in human experience” (Ways of Understanding Religion).

Are surveys afflicted with fatal errors due to declining response rates? Is it impossible to generalize with confidence beyond the small percentage of targeted respondents who become part of our sample? Here Wuthnow is on incredibly shaky ground. His obsession with response rates has been seriously undermined by a growing body of research that Wuthnow dismisses out of hand. Robert Groves demonstrated that response rate is largely irrelevant to what we need to be concerned about regarding survey quality — sample bias (“Nonresponse Rates and Nonresponse Bias in Household Surveys,” Public Opinion Quarterly, vol. 70, no. 5 [2006]). Surveys with low response rates may well be quite representative of the population from which they are drawn while those with much higher levels of success in reaching and recording respondents may, in fact, be more biased. Ironically, steps taken to increase response rate may sometimes introduce additional bias that impairs a sample's representative quality. Moreover, we have tools available that might blunt some of the representational defects of our samples, sample matching and post-stratification among others.

Even if we assume that surveys face daunting challenges, it is naïve to assume that qualitative approaches are somehow immune to them. Paul Lavrakas shows how questions about validity and reliability are just as relevant to qualitative as quantitative research (“Applying a Total Error Perspective for Improving Research Quality in the Social, Behavioral, and Marketing Sciences,” Public Opinion Quarterly, vol. 77, no. 3 [2013]). Given such awareness, I find it increasingly anachronistic to assert the intrinsic superiority of qualitative research or to define “mixed methods” as anything that excludes quantitative analysis. I am equally skeptical of those who claim that survey data alone will give us all the insights we need. Surely we have moved beyond these stale arguments.

For all my misgivings, Inventing American Religion is an interesting and provocative book about the sociology of knowledge. I remain unpersuaded that the author has made his case that polling somehow injures public understanding of religion but I still learned a great deal about the evolution of commercial polling and the development of scholarly research on religion.