Political scientists, particularly those who use experimental methods, increasingly conduct research using online surveys. This increased reliance on a comparatively new survey format raises important questions about recruitment, respondent engagement, and data quality. A striking feature of many web-based interviews is the prominence of the study’s sponsor (often an academic institution), at the start of the interview and often throughout the survey. This prominence is unique to the online format. While telephone polling has historically included a brief introductory message describing the sponsor, and face-to-face interviews involve an introductory letter or mention of the research team leading the project, online surveys (and online experiments) typically contain detailed introductory text describing the study, its sponsorship, the name of the principal investigator, and human subject regulations. In many surveys, the name and logo of the sponsoring academic institution appears in the header of every page of the survey questionnaire. Participants in online studies are, therefore, shown a much stronger exposure to the sponsor than in other survey modes historically used by political scientists.
While previous research has examined the effect of sponsorship, including academic sponsorship, on response rates (Tourangeau, Presser, and Sun Reference Tourangeau, Presser and Sun2014), no extant research examines how sponsorship affects response behavior in the online context. If sponsorship effects exist, they could seriously undermine the external validity of findings. Studying sponsorship effects is difficult, however, because participants may be more or less willing to participate in studies from different sponsors, thereby confounding selection effects and response biases. To test whether prominent and repeated reminders of academic research sponsorship change the behavior of online participants, we conducted an online experiment that randomly assigns sponsorship after participants have agreed to respond. Using a variety of measures of data quality, we find minimal influence of sponsorship by either a university or marketing firm, nor any effect of past survey experience on survey data quality. The results give us confidence in the quality of web-based data collection for political science research.
SPONSORSHIP AND BRANDING IN ONLINE RESEARCH
Relatively recently, conducting a survey required contracting with a survey firm that maintained control over many aspects of the instrument. Today, the growth of online survey tools has “democratized” social and political research, allowing investigators to program and distribute their own surveys on platforms such as Mechanical Turk, Qualtrics, Prolific Academic, social media sites, or email listservs, or by contracting with online panels to recruit respondents.
There are three main points from which respondents may learn about the sponsor of an online survey: in the recruitment message, the consent form, or the body of the survey itself. When recruiting subjects (e.g., from Mechanical Turk), many researchers choose to include their university affiliation in the recruitment text. After opting in, respondents are almost always given at least a brief exposure to the university’s name in the text of the consent form as required by institutional review boards. Finally, many researcher-designed online surveys feature the university’s logo at the top of each page (several examples of this practice can be found in Supplementary Material: Appendix A). In contrast to this relatively prominent display of sponsorship in online surveys, most telephone interviews name the sponsor only once, at the beginning of the interview (see Supplementary Material: Appendix A).
Past research suggests that surveys sponsored by academic institutions yield higher response rates compared with those administered by commercial firms (Fox, Crask, and Kim Reference Fox, Crask and Kim1988; Jones and Linda Reference Jones and Linda1978). Universities are perceived as being more trustworthy, likeable, and having higher authority status compared with commercial companies, all of which increases the likelihood that a potential respondent will accede to the request (Groves, Cialdini, and Couper Reference Groves, Cialdini and Couper1992). Indeed, surveys sponsored by within-state universities garner higher response rates compared with those sponsored by out-of-state universities (Edwards, Dillman, and Smyth Reference Edwards, Dillman and Smyth2014). However, the same factors that improve response rates for university-sponsored surveys may also bias results in several ways.
When the survey sponsor is an institution that the respondent likes and respects (such as a university), respondents may be more concerned with social desirability. “Social desirability bias” occurs when respondents misrepresent themselves in order to appear compliant with social norms (Nederhof Reference Nederhof2006). For example, social desirability often causes people to over-report voting (Belli et al. Reference Belli, Michael, Young and McGonagle1996). While the size of the over-reporting varies by question wording, estimates suggest that it can run as high as 20% (McDonald Reference McDonald2003). In health-related surveys, social desirability can lead people to under-report behaviors such as overeating or drug use. Social desirability bias is reduced in self-administered surveys compared with telephone or in-person surveys (Kreuter, Preser, and Tourangeau Reference Kreuter, Preser and Tourangeau2008). This mode effect suggests that a respondent’s sense of who is “listening” to their answers shapes how they respond. Prominent displays of university sponsorship could thus heighten social desirability effects.
Respondents’ attitudes towards the survey sponsor might also increase demand characteristics, specifically the “good-subject effect,” in which subjects behave in a way that will “help” the researcher (Weber and Cook Reference Weber and Thomas1972). For example, when unobtrusively told of an experiment’s hypothesis, respondents were significantly more likely to behave in a way that confirmed that hypothesis, and this effect was heightened among respondents who had a positive attitude towards the experimenter (Nichols and Maner Reference Nichols and Maner2008).
Finally, university sponsorship might also increase attentiveness and reduce satisficing. Satisficing occurs when respondents expend minimal energy in answering questions. This may include careless reading of response options or a less effortful memory search (Krosnick Reference Krosnick1999).
We hypothesize that a survey’s sponsor affects respondents’ response behavior in several ways. In particular, we expect that, compared with a no-sponsor control group, the university sponsor will increase socially desirable responding and demand effects (Nichols and Maner Reference Nichols and Maner2008) and decrease satisficing behavior (Krosnick Reference Krosnick1991). Similarly, we expect that, compared with the control condition, a commercial marketing sponsor will decrease socially desirable responding and demand effects and increase satisficing.
Design
There are two challenges to assessing sponsorship effects in surveys. First, the questionnaires must be identical apart from the sponsor so that response behavior is not affected by question or response wording. Second, sponsorship may affect who is willing to participate in a survey: different sponsors may yield samples that differ in unobserved ways (due to the sponsor’s reputation or variations in recruitment techniques). These differences may in turn affect response behavior. Thus, a test of sponsorship bias requires a randomized experiment where all respondents complete the same questionnaire only after opting in to survey participation.
We recruited 852 respondents from Amazon’s Mechanical Turk (MTurk), an online opt-in crowdsourcing platform based in the United States.Footnote 1 While MTurk does not demographically reflect the U.S. population as a whole, it does offer two advantages. First, it is a population frequently used by social scientists to field experiments and surveys. Second, MTurk respondents (or “workers”) have a wide range of previous experience: while some have taken hundreds of surveys, others have taken only a few. This variation allows us to examine whether response bias varies depending on the respondent’s survey experience.
U.S.-based respondents were offered $1.00 as compensation to “complete a 10–15-minute survey about your attitudes and opinions.” The MTurk “requester” (sponsor name that was displayed to potential participants) was “Aarhus Research,”Footnote 2 and no other information about the study was available to respondents until they agreed to participate in the study.
We additionally employed quota sampling to obtain a sample of respondents stratified by the amount of their experience on MTurk.Footnote 3 Each task an MTurk “worker” completes is called a HIT, and we used MTurk’s “Qualification Requirements” to create separate survey opportunities (i.e., separate HITs) for workers with varying numbers of completed HITs (<100, 100–500, 500–1,000, 1,000–2,000, and >2,000).Footnote 4 We are, therefore, able to detect whether any observed effects of sponsorship are moderated by previous task experience, thereby addressing concerns about different response patterns among individuals who participate in many online surveys (Binswanger, Schunk, and Toepoel Reference Binswanger, Schunk and Toepoel2013).
After agreeing to participate in the survey, respondents were randomly assigned to one of five conditions. In the first (control) condition, the survey was not attributed to any particular sponsor and the survey included no logos or images. In the next two conditions, respondents were told the survey was being conducted by a marketing research firm called “Aarhus Market Research.” In one of these conditions (marketing light), participants were briefly told the name of the sponsor and then continued with the survey. In the other (marketing heavy), participants were also given a longer description of the firm and asked whether they had previously completed any surveys for this firm.Footnote 5 In both heavy and light conditions, respondents saw the logo at the top of each page of the survey.
In the final two conditions, respondents were told that the survey was being conducted by the University of Aarhus and shown a university logo. In one condition (university light), respondents were told only the name of the university sponsor, while in the other (university heavy), respondents were given a longer description of the university and asked whether they had participated in any research for the university before. The light and heavy versions of our two sponsor treatments were designed to test the conditionality of any effects and to better correspond to real-world survey conditions, in which the extent of exposure to the survey sponsor can vary a great deal.
Data Quality Measures
To measure the influence of sponsorship, we employed four categories of survey and experimental measures: questions likely to evoke socially desirable responding, measures of the “good subject effect,” attentiveness measures, and two knowledge batteries that also serve as measures of satisficing. These sets of measures were presented to respondents in random order. See Supplementary Material: Appendix F for full question wordings.
We implemented four tests of socially desirable responding: self-reported vote history, a “double list experiment” modified from Glynn (Reference Glynn2013), a “check all that apply” question listing a variety of socially desirable and undesirable items, and self-reported interest in politics and public affairs. The survey also included an experimental test of the “good subject effect,” in which respondents alter their behavior to confirm the researcher’s hypothesis after learning the purpose of a study. In this measure, adopted from Nichols and Maner (Reference Nichols and Maner2008), respondents viewed a sequence of 10 pairs of neutral images and were asked to select the image they preferred from each pair. Half of respondents were randomly assigned to receive introductory text telling them that “we believe that when people are choosing between two images that are very much alike, they prefer images on the left.” We then compared the number of “left” images selected in this group to the number of “left” images selected in a control group that did not receive the hypothesis explanation.
The third category of measures assessed attentiveness. Respondents read a short excerpt from a news article about politics and were then asked to write down everything they could remember about the text in an open-ended text box. This task yielded three variables: (1) total time reading, (2) total number of characters typed, and (3) total number of correct and incorrect pieces of information recalled.
The final category, assessing satisficing, consisted of 10 knowledge questions (both open-ended and true–false) concerning politics and scientific understanding. Every question included an explicit “don’t know” option. These questions yielded two measures: the number of correct responses and the number of “don’t know” responses. The advantage of using knowledge questions to assess satisficing is that they require a more effortful information search, thereby increasing the temptation to satisfice by selecting the “don’t know” option. Finally, we measured effortful responding by asking respondents explicitly if they had used outside help to answer any of the knowledge questions (by using the internet to look up correct answers; see Jensen and Thomsen Reference Jensen and Thomsen2013).
The survey also included basic demographics (sex, race, education, party identification, ideology, and measures of the need for cognition and need to evaluate) and four final questions measuring perceptions of the survey interview. These included: (1) a feeling thermometer measures of respondents’ attitudes towards different types of organizations, including universities and market research companies; (2) a measure assessing respondents’ perceptions of the quality of the survey compared with others they have responded to; (3) an open-ended measure asking participants how much they felt they should have been paid for participation in the survey; and (4) a yes–no question asking whether respondents would be willing to receive further emails from the investigators. Participants in the marketing conditions were then debriefed.
RESULTS
Demographic characteristics of the 852 participants are available in Supplementary Material: Appendix C. To assess the impact of sponsorship on our measures of survey response behavior, we employed a series of regressions. Table 1 summarizes the results, which are available in full in Supplementary Material: Appendix D. For each analysis, one measure of survey responding was regressed on indicators for each of our treatment conditions (with the control group as the baseline), controlling for respondent experience (our blocking factor). We present a joint significance test for group differences across experimental conditions, with the resulting F-statistic and p-value reported for each survey measure. Overall, we find almost no group differences, strongly suggesting that the survey sponsor has a minimal effect on participants’ behavior. The effect of sponsorship did not vary by the experience level of the respondent, by the identity of the sponsor (marketing vs. university), or by the intensity of the treatment (exposure to sponsor identity only at the beginning of the survey vs. throughout).
Summary of Effects
Cell entries are F-tests for joint hypothesis of differences across experimental conditions from an OLS regression controlling for respondent experience on MTurk. Full results are included in Supplementary Material: Appendix D. DK = don’t know answers.
Respondents in the marketing sponsorship conditions reported lower rates of voting compared with those in the control group or in the university sponsorship conditions. This is the only significant effect for any of the three social desirability measures. Regarding the good-subjects effect, we see a large and positive treatment effect that replicates the results of Nichols and Maner (Reference Nichols and Maner2008): when told about the researchers’ hypothesis, individuals engage in behavior that strongly conforms with that hypothesis. While this effect holds across all conditions, its magnitude is unaffected by the survey sponsor.
The knowledge and attentiveness measures show that there is no statistically significant effect of sponsorship on respondent engagement. In addition, sponsorship did not affect the rate at which respondents passed the more explicit attention-check questions. Respondents with the most MTurk experience (>2,000 HITs) were slightly less likely to give “don’t know” responses to, or report cheating on, political knowledge questions.Footnote 6
Across all five aspects of respondents’ evaluations of survey experience, sponsorship has no effect. However, more experienced respondents are likely to agree to receiving emails in the future. It is also worth noting that in each of the conditions (including the control condition), respondents were significantly more favorable towards “colleges and universities” than “market research firms,” which is in line with previous research suggesting that universities are perceived more positively compared with other common survey sponsors (Fox, Crask, and Kim Reference Fox, Crask and Kim1988).
DISCUSSION
The results indicate that sponsorship had little effect on socially desirable responding, attentiveness, satisficing, or demand effects. These minimal effects are encouraging given the increasing use of researcher-designed online surveys in political science, although more research is needed to better understand the circumstances under which sponsorship could affect survey error. For example, interactions between the sponsor and the survey topic (such as a university-sponsored survey about student debt) could create more bias compared with the relatively neutral content in this study. Finally, we caution that while our university and market research firms did not affect response quality, effects may exist for other sponsors, especially those perceived as being partisan or having a distinct agenda (Presser, Blair, and Triplett Reference Presser, Blair and Triplett1992; Tourangeau et al. Reference Tourangeau, Groves, Kennedy and Yan2009; Tourangeau, Presser, and Sun Reference Tourangeau, Presser and Sun2014). These results are positive in that they suggest that adhering to the AAPOR (American Association of Public Opinion Research) ethical standards, which prohibit making “false or misleading claims as to a study’s sponsorship or purpose,” does not compromise survey quality.Footnote 7
Several limitations should be highlighted. First, the sample here leaned towards liberal Democratic and had higher levels of education compared with the U.S. general population. It would be worth examining whether those with low levels of education might respond differently to survey sponsorship. Second, it is possible that the lack of observed effect was due to a weak treatment rather than a true lack of sponsor-induced bias. However, the “treatment” in this survey (the sponsor mentioned in the consent form as well as via a header throughout the survey) is a relatively realistic portrayal of sponsorship in an online context. Thus, while a more prominent display of the sponsor (e.g., a video showing a professor giving instructions) might, in fact, increase bias, such displays are uncommon in most real-world surveys. A similar concern might be raised about the identity of the sponsor (“Aarhus University”), which is likely unfamiliar to most respondents, weakening the treatment. However, because past research suggests that within the academic context, perceived authority/prestige does not affect response rates, we would expect similar results for other university sponsors (Porter and Whitcomb Reference Porter and Whitcomb2003).
A final limitation is the sample itself. Participants recruited via MTurk are more attentive compared to other online survey participants (Hauser and Schwarz Reference Hauser and Schwarz2016). This might affect the results in one of two ways: either by heightening the effect of the treatment (because of participants’ increased attentiveness to the survey sponsor) or by reducing the magnitude of effects, because subjects are simply less likely to engage in problematic behavior. While the inclusion of respondents’ level of MTurk experience mitigates these concerns by testing for differential effects by level of “professionalization,” it is still possible that samples that are overall less attentive might respond differently to sponsorship. Overall, these results are encouraging for researchers aiming to reduce potential sources of error when conducting online surveys.
Supplementary Material
To view supplementary material for this article, please visit https://doi.org/10.1017/XPS.2019.25.
