THE TROUBLE WITH MEASURING EVANGELICALS

While it is possible to trace the origins of the “evangelical” back to the decades after the death of Jesus or to the writings of the Protestant Reformation (Eskridge Reference Eskridge2006), the modern American evangelical finds its roots in the American colonial revivals led by John Wesley and Jonathan Edwards (Noll Reference Noll2010). These preachers extolled the value of having a “born again” experience and inculcated a belief in their followers that the most important activity for a believer was to pass that “good news” on to others in the community. While the amount of evangelicals in the United States ebbed and flowed in the United States for the next 200 hundred years, the word would see a resurgence during the tent revivals of famed evangelist Billy Graham in the 1950s (Marsden Reference Marsden1990), who also focused on the “born-again” experience of his followers (Whalin Reference Whalin2014) though he too had trouble defining the term evangelical (Mattingly Reference Mattingly2013).

While Graham was struggling with a definition, so were social scientists. Some early practitioners divided American Protestants into two geographic categories, Northern and Southern, as a sort of proxy for liberal/conservative theology (Stouffer Reference Stouffer1955). From this early conception, other scholars took a different tack. Tom Smith created a tripartite FUND measure (fundamentalist-moderate-liberal), which he sorted denominations into one of those three camps (Smith Reference Smith1990). This, however, drew sharp criticism as being overly reductive by fellow scholars (Green et al. Reference Green, Guth, Smidt and Kellstedt1996). In its place, a new scheme was devised that sorted respondents into one of seven traditionsFootnote ² based on a respondent's church affiliation (Steensland et al. Reference Brian, Park, Regnerus, Robinson, Wilcox and Woodberry2000). This scheme, known as RELTRAD, has become the most widely used in social science having been cited over 1,100 times and has been largely validated by subsequent testing (Dougherty et al. Reference Dougherty, Johnson and Polson2007; Frendreis and Tatalovich Reference Raymond2011). However, the approach is incredibly lengthy and somewhat complicated to replicate in a survey setting and takes over 150 lines of code to implement during data analysis. In addition, RELTRAD employs the use of racial filters in some cases and church attendance filters in others. In a follow-up article published in 2012, the original authors assess the scheme after 12 years and note that RELTRAD needs significant updating to consider the rising number of nondenominational Christians, the difficult interaction between race and affiliation inside RELTRAD, and the constant evolution of Christian denominations that appear on the General Social Survey (GSS) (Woodberry et al. Reference Woodberry, Park, Kellstedt, Regnerus and Steensland2012).

While most academics were moving forward with using religious affiliation as a means to sort out religious traditions, other observers of American religion took a different approach. As early as 1976, the Gallup polling organization began asking respondents if they had “a born-again experience”, which was replicated in a large-scale survey conducted by Christianity Today in 1979 (Schmalzbauer Reference Schmalzbauer2002). In subsequent surveys, Gallup would employ a question that equated a born-again experience with being evangelical,Footnote ³ and used that question for its regular tracking benchmark of evangelical political behavior (Hackett and Lindsay Reference Hackett and Lindsay2008). In addition to allowing individuals to self-identity as born-again, other surveys have asked individuals to self-identify as evangelical. A number of scholars extol the values of allowing self-identification (see Lewis and De Bernardo Reference Lewis and De Bernardo2010), believing that individuals are well equipped to understand where they exist in the religious landscape (Kellstedt and Smidt Reference Kellstedt and Smidt1991; Smith Reference Smith1998), while some subsequent analysis has indicated that self-identifying evangelicals are more likely to be politically conservative (Lewis and De Bernardo Reference Lewis and De Bernardo2010).

In just the last few years, a number of evangelical organizations including Christianity Today and Lifeway Research have attempted to bring clarity to the definition of evangelicalism by looking through the lens of religious belief. While David Bebbington offered a list of four essential evangelical convictionsFootnote ⁴ (Bebbington Reference Bebbington1988), Lifeway Research operationalized those beliefs into four statements that they contend provides the necessary components of evangelical belief if one affirms them strongly (Smietana Reference Smietana2015). This four statement typology has been endorsed and adopted by the National Association of Evangelicals (“What Is an Evangelical?” n.d.).

Taking a step back, this literature has taken two distinct methodological approaches (self-identification vs. affiliation) as well using three different conceptions of evangelicalism (behavior, belief, or belonging). Several recent articles have tried to understand what impact these design choices have had on our understanding of evangelicalism. Work by Hackett and Lindsay conclude that measurement strategy can widely distort the number of evangelicals in the population, with this number ranging from as low as 9% to as high as 38% (2008). Other work has used attempted to use a self-identification approach, for example Smidt (Reference Smith1998) used a question regarding whether individuals considered themselves fundamentalists or not. More recent scholarship suggests that excluding self-identified evangelicals from the evangelical movement by limiting the categorization to affiliation alone might limit researchers' understanding of evangelical Christianity, especially as applied to politics since evangelicals by affiliation have similar attitudes to evangelicals by self-identification. This approach suggests that evangelical belonging be defined as a “spectrum” and self-identification, along with affiliation, can aid in the process of understanding evangelicals (Lewis and De Bernardo Reference Lewis and De Bernardo2010, 124). While the recent work highlighted here does an exemplary job of showing the methodological pitfalls of different measurement strategies, none of it provides practical, prescriptive advice to scholars of politics and religion. What follows is an empirically grounded discussion regarding the best practices for measuring evangelicalism through survey questions.

DATA

Beginning in 2008, a number of social scientists joined together to provide large-scale, comprehensive survey data under the umbrella of the Cooperative Congressional Election Study (CCES) (Vavreck and Rivers Reference Vavreck and Rivers2008). The CCES is a national stratified sample of over 35,000–65,000 Americans fielded online in two waves each election year. The CCES is especially valuable because it has spanned three presidential election cycles, as well as containing a number of different measures of religious affiliation.Footnote ⁵ It allows researchers to analyze the born-again self-identification strategy, as well as the RELTRAD denominational approach, both of which have been used extensively in the previous literature. In addition, we will utilize a well-established survey to compare the two measures, the GSS. The GSS is a nationally representative sample of 1,900–3,000 Americans conducted via in-person interviews typically every 2 years.Footnote ⁶ The RELTRAD coding classification was originally created to sort the denominational variables and that syntax has been reviewed and updated in recent years (Steensland et al. Reference Brian, Park, Regnerus, Robinson, Wilcox and Woodberry2000; Stetzer and Burge Reference Stetzer and Burge2016). In addition, the GSS has asked respondents if they have had a born-again experience consistently beginning in 2004.

To be clear, here are the criteria that will be employed to create our two groups:

1. (1) Using RELTRAD denominations to identify evangelical churches. (Affiliation Measure)

2. (2) Those who responded in the affirmative to: “Would you describe yourself as a born-again or evangelical Christian, or not?”Footnote ⁷   +  those who indicated that they were Protestant. Furthermore, all African-Americans were excluded from this measure so as to minimize the conflation of evangelical Protestants with Black Protestants. (Self-Identification Measure)

We chose to further include anyone who said they were born-again and also Protestant, as that concept is a uniquely Protestant one which does not fit well into Catholic or Mormon faiths (Hunter Reference Hunter1981; Smith Reference Smith1998; Hackett and Lindsay Reference Hackett and Lindsay2008). This approach is distinct from the exit polls that Pew analyzed (discussed above) as they include anyone who affirms a “born-again” experience, irrespective of religious tradition. Another point of divergence between the exit polls’ methodology and social science is the issue of race. Pew reported results regarding evangelical Protestants who were white, however, RELTRAD evangelicals are not exclusively white; nor does the self-identification measure include a racial filter. Our approach self-identification approach does not limit evangelicals to only whites, instead just eliminating blacks from the evangelical category. Because of this, there is a significant level of racial diversity. In fact, the 2016 CCES indicates that just 71% of those who are classified as evangelicals were white, while the self-identification measure was 68% white.Footnote ⁸

FINDINGS

The most appropriate place to begin to understand the differences between the measures is to look at their distribution in each of the surveys. We chose to use the prior five waves of the GSS (biannually from 2008 to 2016) and the CCES survey conducted biannually since its inception in 2008. The percentage of the total sample that was included in each measure can be found in Figure 1 using the appropriate weights included by the survey authors. In order to aid interpretation, each histogram contains error bars representing 95% confidence intervals. It is clear that the difference between the two samples is small or statistically non-existent, especially in the case of the GSS in which the samples were not statistically distinct in any of the five waves. In three of the five waves of the CCES, the self-identification approach does classify a larger percentage of individuals as evangelicals and this difference is statistically significant. However, this provides an incredibly rigorous statistical test as the total sample size of the CCES is exceptionally large and therefore the margin of error shrinks to ±1%.Footnote ⁹ If any trend emerges between these two measures, it seems that the self-identification approach generates a slightly larger sample. However, both measurement strategies result in an overall proportion that is relatively stable. It is worthwhile to note that across the ten surveys we analyzed the percentage of respondents using the affiliation measure is never less than 19% and never more than 24% which comports with other research that has utilized a similar affiliation approach in a different sample (Alwin et al. Reference Alwin, Felson, Walker and Tufis2006). In regards to measuring self-identified evangelicals, only the 2008 CCES stands as a true outlier.Footnote ¹⁰

Figure 1.

Percentage of evangelicals using both affiliation and self-identification measures.

In order to further test the differences between the two groups, we analyzed two variables important to research on both politics and religion: self-described party affiliation and self-reported religious attendance. Figure 2 displays the mean party identification along with 95% confidence intervals for each of the ten surveys that were analyzed. There are several notable findings on this dimension. First, note that in all five GSS surveys the differences between the means is not statistically significant at the p > 0.05 level, indicating that either approach is statistically indistinguishable when it comes to the measurement of party identification. The CCES samples do indicate that self-identification group is slightly more conservative than the group generated using the affiliation strategy, however, this difference is never more than half a point on a seven-point scale. While the CCES samples are somewhat more conservative than the GSS waves, the more recent surveys of both have indicated a somewhat rightward drift among both self-identifiers and those with an evangelical affiliation.

Figure 2.

Mean partisanship of evangelicals using both affiliation and self-identification measures.

The other dimension, church attendance, can be found in Figure 3. The pattern between the party ideology and church attendance is similar. Again, none of the difference of means in the GSS samples was significant at the p > 0.05 threshold, indicating that the measures are not statistically dissimilar for church attendance. However, the differences are statistically different for the CCES, but the magnitude of these differences is still small. In fact, the mean attendance never diverges by more than one-quarter of one point on a six-point scale. The other result that is worth considering is that all ten samples are statistically very similar for church attendance for both measurement approaches. Taken together, the pattern is clear: using either a self-identification strategy or the affiliation measure based on RELTRAD, there is not a noticeable difference in respondents’ reported political affiliation or their worship attendance.

Figure 3.

Mean church attendance of evangelicals using both affiliation and self-identification measures.

To further explore the differences between these measures of evangelicals, a series of regression models limited only to the sub-set of evangelicals (separately by affiliation and self-identification) were specified. These models used a political issue with strong religious overtones: abortion (Evans Reference Evans2002; Jelen and Wilcox Reference Jelen and Wilcox2003) as the dependent variable. Both the CCES as well as the GSS asks respondents if they favored a woman's ability to obtain an abortion for any reason of her choosing.Footnote ¹¹ If self-identified evangelicals are similar in composition to those classified evangelicals through RELTRAD we would expect to see that coefficients in a regression analysis should be statistically similar (i.e. signed in the same direction and with coefficients of roughly the same magnitude).

In these models of the evangelical subsample, four basic independent variables were included: education, gender, age, and party identification.Footnote ¹² In order to provide a direct comparison all variables were coded on the same scale (0–1), and a logit model was specified as the dependent variable was dichotomous in both surveys. Figure 4 displays the results of two of these models as a coefficient plot.

Figure 4.

Regression analysis predicting support for abortion using both affiliation and self-identification measures.

In looking at the results from the GSS (2016), three of the four variables (education, gender, and age) do not reach statistical significance for either of the two measurement approaches. The only variable that is statistically significant is party identification. In this case, a strong affiliation with the Republican party drives down support for access to abortion services. Note that for both the self-identification strategy and the affiliation approach the coefficients are not statistically distinct from one another.

Turning our attention to the panel on the right, which displays the results from the 2016 CCES, a somewhat similar pattern emerges. The education variable is statistically significant for the affiliation approach, but not significant for the self-identification strategy. Note, however, that the magnitude of the coefficient is not statistically distinct. The male variable reaches statistical significance in both of the two subsamples and predicts lower support for access to abortion. Age, for both measurement strategies, is not statistically significant but identifying with the Republican party is significant and signed in the direction of less support for abortion. The affiliation subsample does evidence slightly less support for abortion access as they become stronger identifiers with the Republican party than does the self-identification group, but the coefficient only deviates by less than 0.05. Taken together, with eight total coefficients across two surveys, the two measurement strategies behave in a statistical similar manner. In addition, regression analysis was conducted for 2008, 2010, 2012, and 2014 versions of the GSS and the 2008 and 2012 waves of the CCES. Taken together, these results are similar in nature and magnitude to those described in Figure 4.Footnote ¹³

ADVICE FOR FUTURE SURVEYS

Taken as a whole, we feel confident in saying that either the affiliation measure or the self-identification approach provides a theoretically and statistically sound measurement of evangelical Protestants in the United States. Analyzed from both descriptive as well as multivariate angles, we find that, in most cases, there is no real difference in the overall size or composition of either of these classification techniques. We offer this prescriptive to researchers of American political behavior: if space is running short on a survey it is possible to add a religious dimension to the instrument through the addition of two straightforward questions (plus race). However, if greater specificity is desired, using the full denominational approach (with appropriate follow-ups) is a sound collection technique, as well. To aid researchers in this endeavor, we have included question-wording (borrowed from Pew Research and replicated in the CCES) for new surveys in the online Appendix. The 2016 presidential election showed that white evangelical Protestants are one of the most stable electoral coalitions, as well as the most important. Yet unlike more easily identifiable demographic groups, measuring evangelicals suffer from a multitude of measurement strategies that are often plagued by measurement error, inconsistency, and complexity, this much is evident in the findings of the Pew Research Center.

This disparity in results might lead other social scientists to balk at wading into the waters of measuring religion. It seems likely many survey authors would like to include a religious dimension in their questionnaire but have hesitated when they discovered the complex follow up a structure that is employed by surveys such as the GSS. By suggesting a simplified strategy of combining religious affiliation with self-identification as “born-again or evangelical” and showing its statistical reliability, it is our hope that many more surveys will choose to include consistent religion variables. Our aim was to provide a coherent, clear, and rigorous approach to understanding an extremely difficult concept to measure: evangelical Christianity. We hope that researchers who were wary of including religious questions on their surveys for fear of incorrect operationalization will feel confident in using measures that have been analyzed and considered to be “best practices” in the social sciences.