INTRODUCTION
Surveys show that many Americans—between 20% and 25%—are no longer affiliated with a religious tradition (Hout and Smith Reference Hout and Smith2015; Newport Reference Newport2015; Pew Research Center 2015).Footnote 1 This shift makes the unaffiliated one of the largest “religious” categories in the United States, rivaling evangelical Protestants (25%) and Catholics (21%), and larger than mainline Protestants (15%) and those of other (6%) faiths.
However, recent research argues that many of these ostensibly unaffiliated Americans do have an affiliation but report “none” due to a survey artifact (Dougherty, Johnson, and Polson Reference Dougherty, Johnson and Polson2007; Lim, MacGregor, and Putnam Reference Lim, Ann MacGregor and Putnam2010). This artifact, rooted in question wording or context, encourages some respondents, mainly political liberals, to report being unaffiliated. Liberal respondents report “none” to distinguish themselves from religious conservatives, focused primarily on conservatives’ intolerant attitudes toward nonnormative sexuality in general and lesbian, gay, bisexual, and queer (LGBQ) issues specifically (Baker and Smith Reference Baker and Smith2009a; Putnam and Campbell Reference Putnam and Campbell2010; Vargas Reference Vargas2012; Hout and Fischer Reference Hout and Fischer2014).
This paper investigates sexuality-focused political polarization as a cause of the survey reports of religious nonaffiliation. In contrast with the observational data used by the extant research, the current study uses a survey experiment to test the polarizing effect of a context manipulation: questions on sexuality-focused political issues (e.g., same-sex marriage, gay rights, and “religious freedom” ordinances). Rates of religious nonaffiliation are compared between two conditions, measuring affiliation either before or after sexuality-focused political issue questions, controlling for political orientation, sexual orientation, and the importance of religion to the survey respondent.
Nonaffiliation as survey artifact
Research has argued that the high percentage of religiously unaffiliated Americans is at least proximately caused by a survey artifact linked to the wording of the affiliation question. Borrowing an approach taken in the measurement of attitudes (Schuman and Presser Reference Schuman and Presser1981; Bishop, Tuchfarber, and Oldendick Reference Bishop, Tuchfarber and Oldendick1986; Gilljam and Granberg Reference Gilljam and Granberg1993), this research contends that questions offering a “none” option encourage weakly religious respondents to report being unaffiliated (Dougherty et al. Reference Dougherty, Johnson and Polson2007; Lim et al. Reference Lim, Ann MacGregor and Putnam2010).Footnote 2 Yet, many of these self-reported unaffiliated individuals continue to believe, perform religious behaviors, and feel a connection to a religious tradition, even if the connection is weak, tangential, or residual (Lim et al. Reference Lim, Ann MacGregor and Putnam2010).
Survey artifacts like this one are common in measures of normative traits and behaviors, including other religious measures such as church attendance (Hadaway, Marler, and Chaves Reference Hadaway, Long Marler and Chaves1993, Reference Hadaway, Long Marler and Chaves1998; Presser and Stinson Reference Presser and Stinson1998; Brenner Reference Brenner2011) and prayer (Brenner Reference Brenner2014). However, these measurement biases are typically positive, inflating survey estimates relative to their true value. Respondents tend to report more frequent church attendance and prayer than is warranted by actual behavior. But the research on religious affiliation argues that bias is negative, reducing survey estimates relative to their true value (Dougherty et al. Reference Dougherty, Johnson and Polson2007; Lim et al. Reference Lim, Ann MacGregor and Putnam2010). Thus, in contrast to religious behaviors, which are well understood to be over-reported, this approach assumes that affiliation is under-reported; that respondents are apt to inaccurately claim “none” when reporting an affiliation would be closer to their true value.
The proximate, survey-based cause is a manifestation of a distal cause which is arguably rooted in political polarization. Liberals are motivated to choose “none” as a “symbolic statement” against the perceived coalition between conservative politics and religion (Hout and Fischer Reference Hout and Fischer2002, 165). Moreover, recent research confirms this causal order: a liberal political identity causes reported nonaffiliation rather than the reverse (Hout and Fischer Reference Hout and Fischer2014).
Although liberals take issue with many conservative positions, differences over sexuality-focused politics, especially over LGBQ rights and anti-LGBQ discrimination, remain an important point of contention (Pew Research Center 2016; 2017), arguably catalyzing the polarization that leads to (reports of) unaffiliation (Baker and Smith Reference Baker and Smith2009a; Vargas Reference Vargas2012; Hout and Fischer Reference Hout and Fischer2014). Putnam and Campbell (Reference Putnam and Campbell2010) point to a cascading set of reactions, which they call “aftershocks,” to the original “shock” of the sexual revolution of the 1960s. The first aftershock was the religious revival of late 1970s and 1980s, and the second aftershock, occurring presently, is the religious disaffiliation of liberals caused by sexuality-focused political polarization. But this research views claims of unaffiliation skeptically, recasting the burgeoning percentage of unaffiliated Americans as “still religious,” arguing that some continue to practice and believe (Baker and Smith Reference Baker and Smith2009b; Lim et al. Reference Lim, Ann MacGregor and Putnam2010; Hout and Fischer Reference Hout and Fischer2014).
The current study investigates this causal link between sexuality-focused political polarization and the increasing propensity to claim no religious affiliation. If the political polarization hypothesis is true, it should be possible to experimentally manipulate the reporting of liberals’ religious (non)affiliations. By priming a random subsample of respondents with the polarizing sexuality-related reasons that they claim no affiliation, higher rates of nonaffiliation should emerge after the manipulation in comparison with a control group. Questions about LGBQ rights are used as a contextual manipulation rather than other potential foci (e.g., contraception, abortion, extramarital sex) given the continued high level of polarization over LGBQ rights post-Obergefell (e.g., “religious freedom” bills) (Pew Research Center 2016).
My research question is: do respondents claim no religious affiliation at a higher rate when primed by questions that ask about sexuality-focused politics? Moreover, does the context effect differ by sexual orientation, political orientation, or the strength of the respondent's connection to religion? From this research question follows the first hypothesis based on prior research: asking questions about LGBQ rights and anti-LGBQ discrimination before asking about religious affiliation will increase self-reports of nonaffiliation. More specifically, the research suggests that it is liberal respondents who will alter their reports in reaction to the linkage between conservative politics and religion. Thus, the first hypothesis focuses on the interaction between question order condition and political orientation:
H1:
Liberals will be more likely to report “none” when the affiliation question is preceded by questions about sexuality compared with conservatives who will not differ between conditions.
But not all liberal respondents should be equally likely to report nonaffiliation. Rather, it is those who feel only a tangential or residual connection to their faith who are most likely to alter their reports of religious affiliation in reaction to question framing (Lim et al. Reference Lim, Ann MacGregor and Putnam2010). Thus, the second hypothesis is focused on the interaction between question order condition and religious importance.
H2:
Respondents with relatively lower reported religious importance will be more likely to report no religious affiliation when the affiliation question is preceded by questions about sexuality.
DATA AND METHODS
A sample of 2,708 English-speaking, U.S.-based respondents from a nonprobability SurveyMonkey panel began a 15-question web survey in January and February 2016. Breakoffs and item nonresponse reduced the analytic sample by 17% to 2,238 respondents with complete data for the variables analyzed here. Incomplete cases are approximately evenly distributed across the experimental conditions. As with any nonprobability panel, these respondents do not necessarily resemble the general population of the United States nor are their responses generalizable to that population. But estimating population parameters is not the goal of this research. Rather, this study's purpose is testing the effect of question context on responses to the religious affiliation question in line with the hypothesis derived from the political polarization theory. When paired with random assignment, nonprobability samples from online panels can be effectively used to test survey design features such as question order and context (Callegaro et al. Reference Callegaro, Villar, Yeager, Krosnick, Callegaro, Baker, Bethlehem, Göritz, Krosnick and Lavrakas2014; Mullinix et al. Reference Mullinix, Leeper, Druckman and Freese2015).
The dependent variable, religious nonaffiliation, is dichotomous; (0) no religion, (1) any religious affiliation (i.e., Protestant, Catholic, Jewish, or other religion; see Appendix 1 for question text).Footnote 3 The manipulation, question order is also dichotomous: (1) religion questions asked first, or (0) sexuality-focused political questions asked first. The sexuality-focused political questions addressed the civil rights of and discrimination against LGBQ individuals in employment, housing, public accommodations, and wedding services (e.g., florists, bakers). These questions are used only to contextualize the religion measures for respondents in the treatment condition (see Table 1 for the distribution of the outcome across treatment conditions across sampling strata).
Percent of religiously unaffiliated respondents, by question order (religion first or sexuality first), political orientation, and sexual orientation
Independent variables include an 11-point scale measuring importance of religion and political orientation, coded in three categories: liberal or very liberal; moderate; and conservative or very conservative. Sexual orientation (LGBQ or straight/heterosexual), is also included as a blocking variable as approximately half of the respondents were sampled from a stratum of LGBQ panelists. An LGBQ oversample was necessary to achieve large enough cell sizes to permit including sexual orientation in these analyses, an important covariate given LGBQ respondents’ potential to be differentially affected by the manipulation, their political liberalism and relative irreligion (Sherkat Reference Sherkat2016) (see Table 2 for descriptive statistics).
Descriptive statistics
Logistic regression models predict nonaffiliation using the aforementioned covariates. Models test the hypothesized interaction between question order and both political orientation (model 1) and religious importance (model 2). Hypotheses are tested using key interactions: liberal political orientation and question order; religious importance and question order.Footnote 4
RESULTS
The first model tests the main and interaction effects of question order and political liberalism. Statistically significant main effects emerge for liberal and moderate political orientations compared with conservatives (see Table 3, model 1). Liberals and moderates both have nearly three times (e logit = e 1.05 = 2.9) the odds of conservatives to report no religious affiliation. Similarly, LGBQ respondents have 45% higher odds of reporting no affiliation compared with heterosexual/straight respondents. The main effect of question order is not statistically significant (−0.07; p = 0.64). However, this does not mean that question order fails to influence the survey reports of religious affiliation. A significant interaction suggests that liberals who received the religion question first had about half-again the odds of reporting “none” than other respondents.
Logistic regression predicting nonaffiliation with political orientation, sexual orientation, religious importance, question order condition, and interactions
Comparison category in italics. ***p⩽0.001, **p⩽0.01, *p⩽0.05. Dashed lines indicate that variable was not included in the model.
The second model includes religious importance and its interaction with question order to test for differences in how respondents with varying connections to religion report their religious affiliation between conditions.Footnote 5 With religious importance in the model, neither the main effect for liberals nor that for question order are significant but their significant interaction term increases the odds of nonaffiliation for liberals receiving the religion question first by nearly double that of other respondents. Each increase in religious importance reduces respondents’ odds of nonaffiliation by 39%, but its interaction with question order is not significant. Comparison of the two models, using likelihood ratio χ2 (χ2 = 978.0, p < 0.001), AIC (ΔAIC = −974.0), and BIC (ΔBIC = −962.6), strongly prefer model 2.
To help with interpretation of this model including interaction effects, Figure 1 presents the predicted probabilities of nonaffiliation for each question order condition by political orientation based on model 2. Neither moderates (0.43 and 0.41; p = 0.38) nor conservatives (0.37 and 0.34; p = 0.30) differ in their predicted levels of nonaffiliation between question orders. However, liberals who receive the sexuality questions first (0.37) are less likely to report being unaffiliated compared with liberals who receive the religion questions first (0.43; p < 0.01).
Predicted probability of nonaffiliation, by question order condition and political orientation.
DISCUSSION
The political polarization hypothesis suggests that priming liberal respondents with the focus of their disaffection from conservatives should generate more polarization. Thus, liberal respondents receiving questions about LGBQ rights and anti-LGBQ discrimination before the question about religious affiliation should report higher rates of nonaffiliation than those receiving questions in the reverse order. However, this did not occur. Rather, liberals claim a lower rate of nonaffiliation when religious affiliation is measured after a set of questions about sexuality and anti-LGBQ discrimination.
A second hypothesis suggests that respondents with weak links to their religion, the so-called “liminal nones,” are more likely to report being unaffiliated in the face of sexuality-focused polarization. However, no support for this hypothesis is evidenced here. While those with lower religious importance are indeed (and unsurprisingly) more likely to be unaffiliated, the interaction between religious importance and question order is not significant.
These findings fail to support the political polarization hypothesis, but why did they occur? It is possible that asking a series of questions about LGBQ rights and anti-LGBQ discrimination allowed liberal respondents to express their liberal political identity by endorsing progressive positions. In this context, liberals could safely report a religious affiliation without fear of being mistaken for a conservative. Similarly, it is also possible that the sexuality questions primed religious liberals to focus on the social gospel or social justice focus of their religion and acceptance of or support for LGBQ rights.
These post-hoc explanations are unsatisfying for a couple of reasons. First, no difference emerges between question order conditions for religious importance. If demonstrating their acceptance of LGBQ people allowed liberals to claim their religious affiliations more validly, why are there no changes to reported levels of religious importance between conditions (see Table 2)? Second, previous research has demonstrated that respondents tend to carry forward attitudes expressed in prior questions to subsequent ones (Salancik and Conway Reference Salancik and Conway1975; Tourangeau et al. Reference Tourangeau, Rasinski, Bradburn and D'Andrade1989). Context effects are influenced by the degree to which the context and dependent variable are related or in conflict (Tourangeau et al. Reference Tourangeau, Rasinski, Bradburn and D'Andrade1989). These highly related concepts (LGBQ rights and religion) and the high degree of conflict as suggested by the political polarization hypothesis suggest that the carryover effect from the sexuality questions should have increased the reports of religious nonaffiliation for liberal respondents.
The difference between the reports of affiliation for liberals raises another question: which estimate is more valid? Split-ballot experiments, like that used here, can identify with high internal validity the role of context and question order in influencing answers. However, they may not necessarily identify the most valid estimate. Prior research on the growth of nonaffiliation in the United States expresses a preference for the report that achieves the highest rate of affiliation and the lowest rate of nonaffiliation (Dougherty et al. Reference Dougherty, Johnson and Polson2007; Lim et al. Reference Lim, Ann MacGregor and Putnam2010). This work implies that recent estimates of affiliation include a high rate of under-reporting; that some respondents are failing to report their actual affiliation. Accordingly, whatever question wording or context, mode or method that best pushes the respondent to choose an affiliation (other than “none”) is assumed by this extant research to improve measurement validity.
However, this approach operates counter to the balance of research on the measurement of religious and other normative behaviors where over-reporting is the norm; respondents tend to make unwarranted positive claims. When considering artifacts in the measurement of normative behavior and traits, the estimate that represents the lowest social desirability is typically assumed to be closest to the truth. Thus, the question wording, order, or context that delivers the estimate with the lowest rate of religious behavior is preferred (Presser and Stinson Reference Presser and Stinson1998; DeBell and Figueroa Reference DeBell and Figueroa2011). Similarly, research on error in the survey reports of voting (Belli, Moore, and VanHoewyk Reference Belli, Moore and VanHoewyk2006; Duff et al. Reference Duff, Hanmer, Park and White2007; Presser Reference Presser1990), physical exercise (Williams et al. Reference Williams, Klesges, Hanson and Eck1989; Taylor et al. Reference Taylor, Coeffey, Berra, Iaffaldano, Casey and Haskell1984), and other normative behaviors proceeds from this assumption. Why has the extant research on “nones” treated measures of religious affiliation so differently?
The very nature of questions about affiliation and those measuring other religious concepts make important presumptions about that which is being measured. Do you believe in God? How often do you attend religious services? What is your religious affiliation? Each of these questions presumes its referent—belief, behavior, and belonging—in a directive way. Even if negative options are included (i.e., no, never, none), all at least subtly cue agreement with the pretense of the question. Through the biasing effects of acquiescence and socialization into a culture that views religion as important, estimates of religiosity are inflated. But it would be difficult to measure these concepts without a direct question referencing its object. Try to imagine a reasonably well-worded question about belief without reference to god (or a synonym) or a question about affiliation without reference to denominations or religious traditions.
Affiliation may be better measured indirectly. Time diaries and other chronological measurement techniques have been effectively used to validly measure religious behaviors like church attendance (Presser and Stinson Reference Presser and Stinson1998; Brenner Reference Brenner2011; Brenner and DeLamater Reference Brenner and DeLamater2016) and prayer (Brenner Reference Brenner2014). These methods work well because they eschew direct questions about the normative behavior of interest, thereby avoiding priming, acquiescence, and other effects that bias estimates and harm data quality. Religious affiliation could also be measured indirectly by masking it in a question about social memberships of all types. Respondents could be asked to name five organizations or groups to which they belong. Since Americans belong to fewer voluntary organizations today than in the past, and because churches are perhaps the most prevalent among these affiliations (Putnam Reference Putnam2000), an approach like this one could potentially decrease bias in our estimates of affiliation by measuring religious affiliations without asking directly about it. This approach may have the added benefit of measuring the strength of the individuals’ attachment to the religion in the context of other affiliations, if the order in which groups are mentioned is meaningful.
Two limitations of this study warrant discussion. The experimental design employed permits causal inference about the role of question order in the reports of affiliation, the estimates themselves are not necessarily generalizable, given the use of a nonprobability online panel. Indeed, these respondents do differ from the general population as they are more liberal and less religiously affiliated (Pew Research Center 2015; 2017). This nonprobability design was used to ensure adequate numbers of LGBQ individuals, a small and hard-to-sample population. Although it is not technically impossible to generating equivalent numbers of LGBQ individuals using a probability sampling design, such a design would be logistically difficult and prohibitively expensive. Moreover, while the evidence on panel conditioning effects is mixed (Toepoel, Das, and Van Soest Reference Toepoel, Das and Van Soest2008), experienced web survey panel members may answer survey questions differently than a fresh, probability-based sample from the general population (Van der Zouwen and Van Tilburg Reference Van Der Zouwen and van Tilburg2001). Thus, an important avenue for future work is an extension of this research to a probability-based sampling design to ensure that findings are both internally and externally valid.
A second important limitation is the narrow focus and potential weakness of the manipulation. Although the sexuality questions directly addressed the topics of current political debate and did so in a way to prime the respondent to think about the linkage between conservative politics and religion, other potentially stronger primes could be used to further test the hypothesis. Future research could use nonverbal primes, like photographs (e.g., a gay pride parade, a same-sex wedding, or a same-sex couple kissing), or vignettes of varying strength. Alternatively, the manipulation could focus on a different highly polarizing topic (e.g., abortion, contraception, and reproductive rights). Finally, future research could test the effect of this prime on measures of related religious concepts, such as spirituality in combination with or in lieu of affiliation.
CONCLUSION
Recent surveys suggest that nearly one-quarter of Americans are religiously unaffiliated. Further research into the validity of these survey estimates argues that they result, at least partially, from a survey artifact. Liberal respondents report being unaffiliated in reaction to the perceived link between religion and conservative politics. This study used a question order experiment to test the political polarization hypothesis. Respondents were randomly assigned to one of two conditions: they were asked to report their religious affiliation either before or after a battery of questions on sexuality-focused politics that measured approval of legal same-sex marriage, LGBQ rights, and anti-LGBQ discrimination. A measure of religious importance was used to test a related hypothesis: that respondents with the weakest association to religion would be most likely to choose “none” as a result of the sexuality-focused political prime.
Counter to the hypotheses based in the political polarization explanation, the experimental manipulation failed to reduce the reports of religious affiliation. Political liberals were more likely to report being unaffiliated when the question was asked before the sexuality-focused political questions. Moreover, while religious importance is strongly related to the reports of affiliation, question order did not differentially affect the respondents who viewed religion as less important or unimportant in their lives.
In summary, the political polarization hypothesis did not operate as anticipated. Thus, these findings suggest that the reports of religious nonaffiliation are unlikely to be a survey artifact caused by sexuality-focused political polarization. Future research should continue to test this important association and other potential causes of nonaffiliation.
APPENDIX 1: QUESTION TEXT
DEPENDENT VARIABLE
Religious Affiliation
What is your current religious affiliation? (1) Catholic, (2) Protestant, (3) Jewish, (4) none (“I don't have a religious affiliation.”), or (5) other (“Some other religion.”)
INDEPENDENT VARIABLES
Religious Importance
How important is religion to you in how you live your life? (0) Not at all important to (10) Extremely important
Political Orientation
How would you describe your political point of view? (1) Very liberal, (2) Liberal, (3) Neither liberal nor conservative, (4) Conservative, and (5) Very conservative
Sexual Orientation
How do you identify yourself? Lesbian, Gay, Bisexual, Straight or heterosexual, Questioning, Asexual, Queer, Other (specify)
CONTEXTUAL FACTOR
Sexuality-Focused Political Questions
Employment
• Do you think that religious organizations should be allowed to not hire a job applicant based on the applicant's sexual orientation?
• Do you think that employers other than religious organizations should be allowed to not hire a job applicant based on the applicant's sexual orientation?
Housing
• Do you think that the owner of an apartment building should be allowed to refuse to rent to a person based on the renter's sexual orientation?
• Do you think that the owner of an apartment building should be allowed to refuse to rent to a person based on the owner's religious beliefs?
Public Accommodations
• Do you think that a business that provides services to the public, such as a restaurant or hotel, should be allowed to refuse those services based on a customer's sexual orientation?
• Do you think that a business that provides services to the public, such as a restaurant or hotel, should be allowed to refuse those services based on the owner's religious beliefs?
Wedding-Related Services
• Do you think that a business that provides wedding services, such as a caterer or florist, should be allowed to refuse those services to same-sex couples?
• Do you think that a business that provides wedding services, such as a caterer or florist, should be allowed to refuse those services based on the owner's religious beliefs?
