Inferential Issues in Survey Research

Generalizing from survey research to broader populations of interest often presents a number of challenges. The “total survey error” framework (for a recent review, see Groves and Lyberg Reference Groves and Lyberg2010) elaborates the conditions under which survey results accurately characterize population-level parameters for constructs of interest. These error components stem from two sources (Groves et al. 2004): measurement (owing to the validity of the instrument, the precision with which the core concept is measured, and how accurately responses are processed) and representation (owing to non-response bias and how well the sampling frame represents the target population). These potential sources of error threaten the accuracy of the survey statistic with respect to the population of interest.

Concerns about the representativeness of samples obtained via random digit dialing (e.g., Brehm 1993; Curtin, Presser and Singer 2000; Holbrook, Green and Krosnick Reference Holbrook, Green and Krosnick2003; Berinsky Reference Berinsky2004; Groves 2006; Keeter et al. Reference Keeter, Kennedy, Dimock, Best and Craighill2006; Blumberg and Luke Reference Blumberg and Luke2007; Keeter et al. Reference Keeter, Kennedy, Clark, Tompson and Mokrzycki2007) have led many researchers to turn to internet-based surveys, though evidence is mixed about whether online surveys adequately remedy these concerns about representativeness.Footnote ¹ Though the use of opt-in internet panels generates serious concerns about survey sampling and representativeness (see AAPOR 2010), they are not the only potential sources of error in internet surveys, nor are they the only possible explanation for differences that arise between surveys administered over the internet or, for instance, over the phone. For instance, Malhotra and Krosnick (Reference Malhotra and Krosnick2007) compare the 2000 and 2004 American National Election Studies administered over the phone with responses obtained from internet-based samples, and find significant differences between the samples. However, they cannot identify whether differences between various administrations of the same survey are due to differences in sampling or the form in which the surveys were administered. Likewise, mixed-mode studies that allow respondents to select the mode by which they complete the survey are unable to distinguish mode effects from selection effects. Although the former refers to the differential measurement error present in the mode of administration, the latter is the result of the distinctive characteristics of the groups of respondents that choose to take the survey in a certain mode.

Mode Effects in Survey Research

Survey mode may affect survey responses in several important ways. First, mode may affect respondents’ level of engagement with the survey, and better-engaged respondents may provide more accurate responses. Verbal surveys likely induce higher levels of engagement because the interviewer’s voice can communicate enthusiasm and stimulate interest (Cannell, Miller and Oksenberg Reference Cannell, Miller and Oksenberg1981; Chartrand and Bargh 1999). Although internet surveys allow respondents to complete the survey on their own time and at their own pace, the lack of engagement with an interviewer and the inability to monitor the respondents’ attentiveness to the survey could decrease respondents’ interest in the survey, and thus the quality of the responses they provide.

Though verbal interviews may increase respondent engagement, satisficing theory (Krosnick Reference Krosnick1991; Holbrook, Green and Krosnick Reference Holbrook, Green and Krosnick2003) suggests that the cognitive burden associated with answering questions may be greater in verbal interviews, and thus decrease data quality (Tourangeau and Smith Reference Tourangeau and Smith1996). For instance, questions with five-point response scales (i.e., “strongly agree” to “strongly disagree”) require the respondent to listen to each of the five categories before responding, which could encourage greater satisficing and thereby decrease the quality of the answers. In contrast, interviews conducted over the internet display visually each of the five response options.Footnote ²

Variation in levels of engagement or uses of satisficing strategies across survey mode could produce several observable implications. First, survey mode could affect response patterns. Survey participants who are not well engaged may demonstrate higher levels of item non-response and skip those items in which they are less interested. If phone surveys induce higher levels of engagement, we would expect greater levels of item non-response among internet survey respondents. In addition, survey mode can affect the likelihood of a respondent choosing a no opinion response when the option is provided.Footnote ³ Holbrook, Green and Krosnick (Reference Holbrook, Green and Krosnick2003) attribute this to the lower level of satisficing present in the face-to-face interview situation. Thus, internet respondents may answer more survey questions with “don’t know” compared with phone respondents.Footnote ⁴

Furthermore, if predominantly visual surveys, such as internet questionnaires, and predominantly aural surveys, such as telephone interviews, induce different cognitive burdens on survey respondents, response quality could vary systematically across modes.Footnote ⁵ Satisficing could manifest in, for instance, an abnormally large number of “somewhat disagree” responses to questions that use five-point Likert scales, or an increased propensity to rate political figures as a “50” on items that use 101 point feeling thermometers.Footnote ⁶ Thus, levels of item differentiation may differ across modes.

Survey mode could also affect the substantive responses that are provided by survey participants, especially for questions that are cognitively more difficult or ask about attitudes or behaviors that could be influenced by social desirability.Footnote ⁷ Respondents have appeared to be more willing to admit to undesirable attitudes and behaviors in self-administered modes (e.g., Tourangeau and Yan Reference Tourangeau and Yan2007; Chang and Krosnick Reference Chang and Krosnick2009; Kreuter, Presser and Tourangeau Reference Kreuter, Presser and Tourangeau2009; Chang and Krosnick Reference Chang and Krosnick2010). Thus, internet respondents may feel fewer pressures from social desirability because they are not directly interacting with another individual, suggesting that we may find higher rates of what may be considered socially undesirable attitudes and behavior among internet respondents.

Our discussion above identifies a variety of ways in which survey mode may affect response patterns and response quality. Table 1 summarizes these expectations more formally. With respect to response patterns such as unit non-response, the use of “no opinion” options, and differentiation in the use of scales, we have competing expectations as to which mode, telephone or internet, will lead to the greatest measurement error. By contrast, we have a clear expectation that social desirability biases should be greatest among telephone respondents owing to the presence of a live interviewer.

Table 1

Summary of Expectations

In addition, survey instruments themselves may condition the magnitude and direction of mode effects. Just as surveys administered across various modes may impose varying cognitive demands of varying strength, the instruments used to elicit survey responses also vary in the cognitive burdens they place on respondents. Moreover, the cognitive burdens of these instruments may vary across modes. For instance, standard “feeling thermometer” questions may be relatively easier for internet respondents, as these questions are typically accompanied by a visual representation of a thermometer or scale along which the respondent evaluates the attitude object. Lacking such a visual, these questions may induce greater cognitive demands for phone respondents, and thus lead to higher levels of satisficing. Thus, mode effects may be greater for this question type compared with survey instruments that present respondents with simple “yes” or “no” response options. Generally, the magnitude of mode effects may vary across different survey instruments, depending on the cognitive difficulties associated with the instrument and how these cognitive burdens differ across mode.

Data

The 2011 YPP survey was administered by Knowledge Networks (KN) through online and telephone modes in both English- and Spanish-language versions from February 9 to July 14, 2011. The target population for the survey comprised young people between 15 and 25 years of age living in the United States from four ethnic/racial groups: non-Hispanic whites, non-Hispanic blacks, non-Hispanic Asians, and Hispanics of any race. Figure 1 describes the sampling procedures, where the sample of this population was drawn from two sources: KN’s probability-based internet panel and an address-based sample (ABS).Footnote ⁸ The KN panel was used to draw a direct sample of persons aged 18–25 from the four racial/ethnic groups, as well as to draw a sample of parents with offspring between the ages of 15 and 25. From the latter group, the parent was asked to identify the race and ethnicity of each person aged 15–25 in the household, and if any individuals belonged to the target population, one eligible household member was selected into the sample. In all, 993 completed surveys were obtained online from the KN direct panel and another 789 online surveys were administered to respondents recruited through the panel of parents. In all, 65 percent of individuals sampled from the KN panel to receive the internet survey completed the screener survey, and among those who qualified for the main survey, the completion rate was 95 percent. An additional 284 phone interviews were obtained from a separate sample drawn from the KN panels. In all, 50 percent of those sampled completed the screener, and of those who qualified for the main survey, the completion rate was 42 percent. The most salient characteristic of the samples drawn from the KN panels was that both the internet and telephone samples were drawn from the same underlying pool of potential respondents, though, as discussed in greater depth below, the different response rates for the two modes raises the possibility that the two samples could differ in ways that might affect their response patterns.

Fig. 1

Overview of sampling procedures

The KN sample was supplemented by an ABS in order to over-sample minority groups. The sample frame was drawn from the US Postal Service’s Delivery Sequence File, which was combined with additional database sources to over-sample households believed to contain members of the targeted racial/ethnic and age groups. Targeted households were sent a letter inviting eligible household members to participate in the web survey. Households that did not respond to the initial invitation or follow-up were contacted by professional interviewers who attempted to administer the screener and main survey by telephone. Ultimately, 14 percent of selected households completed the screener and 47 percent of qualified individuals completed the main survey. In all, 462 individuals selected through the ABS sample completed the survey online and 392 took the survey over the phone. Unlike the KN samples, the distinguishing feature of the ABS sample was that respondents had the opportunity to select into their mode of administration.

The Substantive Consequences of Survey Mode

Thus far, our results provide some evidence of differential survey response patterns across mode. In this section, we demonstrate how survey mode affects the substantive responses elicited from the respondents for two different types of questions. First, we explore the effects of mode for questions about racial attitudes in which social desirability biases may influence response patterns. Second, we examine the presence of mode effects for fact-based questions in which respondents are asked to provide answers that are subsequently used to assess their knowledge in a given domain.

Racial Attitudes

Social desirability bias refers to conditions in which respondents feel pressure to answer survey questions in socially acceptable ways, and research on racial attitudes has found that social desirability may lead respondents to over-report levels of racial tolerance (e.g., Kuklinski, Cobb and Gilens Reference Kuklinski, Cobb and Gilens1997). In an experimental design that assigned survey respondents to mode, Krysan (Reference Krysan1998) finds that whites interviewed face to face tend to report more racially liberal attitudes than whites who completed a mail survey. Social desirability biases may play a smaller role for internet respondents because the anonymity of the online survey condition could reduce the pressure respondents feel to misrepresent behaviors or attitudes that are perceived as less socially desirable. We analyze response patterns to two short batteries of questions to examine this possibility. We examine responses from white respondents only, as it is less straightforward to identify the socially desirable response options among respondents of color. The first set of questions assesses views about racism, racial minority groups, and immigrants. In particular, we asked respondents to indicate the extent to which racism is an important problem in contemporary society,Footnote ²⁵ whether the increase in population among people of color strengthens or weakens the country, and whether they believe that immigrants take jobs, housing, and health care away from native-born citizens. If the importance of social desirability bias varies across mode in the way we outlined above, we expect to see higher levels of racially liberal responses among phone respondents than among internet respondents.

The second set of questions addresses the racial composition of respondents’ interpersonal interactions. These questions asked respondents whether they interacted with “mostly whites,” “mostly blacks,” “mostly Hispanics/Latinos,” “mostly Asian,” or “mostly multiracial people” at work or school, in social settings, and in groups or organizations to which they belonged. The dependent variable in this analysis is an indicator for whether respondents reported interacting with “mostly whites” for each of the five questions, as we believe that the socially desirable response is for whites to indicate that they interact with people from a variety of races.Footnote ²⁶

Across both sets of analyses, we find strong, consistent, and robust evidence that mode is associated with racial attitudes and self-reported behaviors. The top panel of Table 5 shows that phone respondents provided more racially liberal responses than did online respondents.Footnote ²⁷ Online respondents reported that racism is less of a problem in contemporary society, were more likely to report that the increase in minority populations weakens the country, and were more likely to agree that immigrants take away jobs, health care, and housing from native-born citizens. For each question, the distributions of responses are statistically distinguishable by mode. Moreover, these differences hold up when examining them in a multivariate context. When regressing the survey responses on survey mode and a battery of demographic controls (education, age, gender, geographic region, and metro/non-metro residence), the coefficient for mode is statistically significant in each model.Footnote ²⁸ Thus, we find that phone respondents provided the socially desirable response option at considerably higher rates than online respondents, even while controlling for other factors that might also influence racial attitudes.

Table 5

Social Desirability and Survey Mode

Note: ^aWhite respondents only. Entries are the percentages of survey respondents who report a given response. The text of the question for the first set of results is: “Some people say that racism no longer exists in American society and politics. Would you say that...?” The second question is: “As you may know, Hispanics/Latinos, Blacks, Asians, and other minorities are expected to make up more than half the U.S. population by the middle of this century. Do you think this increase in minority populations...?” The text of the third question is: “Please indicate how much you agree or disagree with the following statement. Immigrants, especially immigrants from Latin America, Asia, and Africa, take jobs, housing, and healthcare away from people who were born in the United States.” Data are unweighted, but similar results are found when weights are applied.

^b White respondents only. Entries are the percentages of survey respondents who reported interacting mainly with other whites at work or school, when socializing with friends and family, and when they are participating in groups and organizations of which they are a member. Data are unweighted, but substantively identical results are obtained when weights are applied.

Similar results emerge when examining questions that ask respondents to report the racial composition of their interpersonal interactions. The bottom panel of Table 5 shows the proportions of white respondents who reported that they interact mostly with other white people. Across all three contexts, phone respondents were at least ten percentage points less likely than online respondents to report interacting mostly with other white people. All of these differences are statistically significant. Furthermore, when regressing a dichotomous indicator for whether respondents reported interacting mostly with other whites on survey mode and demographic controls, the coefficient for survey mode is negative and statistically significant, indicating that phone respondents were considerably more likely than online respondents to report that their social interactions occurred with a more racially diverse set of individuals.Footnote ²⁹

These analyses consistently indicate that interviews conducted over the telephone generate more racially liberal and socially desirable survey responses compared with the responses elicited via internet surveys. Although we are unable to directly test whether these patterns are a result of social desirability bias, they are exactly the patterns we would expect to find if social desirability pressures lead phone respondents to report artificially liberal racial attitudes and behaviors. That these modal differences are robust to accounting for other demographic characteristics that are also associated with racial attitudes suggests that analysts should take mode into consideration when comparing responses to questions that may be susceptible to social desirability.

Political Knowledge

The presence of a live survey interviewer might also introduce another sort of bias that results in mode effects even for fact-based questions. Specifically, phone respondents seeking to oblige an interviewer or to conceal their ignorance may eschew “don’t know” responses to questions that test knowledge of political facts and offer an answer choice even when they do not actually know the answer (Luskin Reference Luskin2011). The consequence of this measurement error is that the level of political knowledge of phone respondents may be overestimated relative to that of online respondents. We evaluate these patterns using a standard battery of five political knowledge questions that are commonly asked in survey research.Footnote ³⁰

As shown in Table 6, we find that phone respondents provided correct responses to a larger number (3.03) of these questions compared with internet respondents (2.85). This difference is especially interesting because the effects of self-selection that we outlined above would suggest that internet respondents (particularly from the ABS sample) would demonstrate higher levels of political knowledge. Moreover, the internet sample is more highly educated (44 percent attended or graduated from college) than the phone sample (36 percent attended or graduated from college). However, the difference in political knowledge observed across modes can be attributed to differential response patterns. Summing the number of “don’t know” answers and non-responses for the five political knowledge items, we see that phone respondents were much more likely to provide an answer: on average, online respondents did not attempt to answer 1.43 questions, compared with a mean of 0.51 for phone respondents. Moreover, the mean number of incorrect responses among phone respondents (1.46) is twice that of online respondents (0.73).Footnote ³¹

Table 6

Responses to Political Knowledge Questions by Survey Mode

We also estimated multinomial logistic regressions of responses on mode and demographic controls.Footnote ³² We again find that phone respondents are less inclined than internet respondents to give a non-answer (“don’t know” or unit-missing) for all five questions. Further, among respondents who answered the questions, online respondents were usually much more likely to supply the correct answer, which suggests that phone respondents were more likely to guess if they did not know the answer to the question, thus, inflating estimates of their political knowledge.Footnote ³³

To assess how this differential propensity to guess affects the estimation of levels of political knowledge across survey mode, we conduct a Monte Carlo simulation of the distribution of correct answers had all respondents given substantive responses to all five knowledge questions. Specifically, following the recommendation of Mondak (Reference Mondak1999), we randomly assign all “don’t know” and missing responses to one of the three substantive answer choices provided for each question. We then sum the number of correct responses and calculate the means for online respondents and for telephone respondents. We repeat this procedure 10,000 times to obtain our estimates of political knowledge by survey mode. As shown in the last row of Table 6, once this correction is made, online respondents display higher levels of political knowledge than do phone respondents (a mean of 3.32 “correct” answers calculated across the 10,000 simulations compared with 3.20).Footnote ³⁴ In all 10,000 simulations, online respondents were estimated to provide more correct answers than phone respondents, and the “average” p-value associated with the difference in means across the 10,000 simulations is 0.049.

Discussion and Conclusion

Internet surveys offer a number of important advantages over more traditional telephone surveys. The relative ease of reaching large numbers of respondents enables researchers to quickly and efficiently compile sizable data sets. They are likely to be especially valuable in researching populations, like the youth surveyed in the YPP study, for which representative samples are increasingly difficult to obtain by telephone. As survey research moves increasingly onto the internet, though, it is important to understand how this change affects the quality of the data that is collected and the inferences they generate. These changes have important implications not just for mixed-mode surveys such as ours but also for any internet surveys that seek to compare their results to those of earlier studies administered via an interviewer.

In particular, the mode of administration appears to have substantial consequences for general response patterns and data quality. Online respondents are more likely than phone respondents to skip survey items and choose the “don’t know” or “no opinion” response option. Online respondents also exhibit less differentiation in their use of rating scales, again suggesting that internet surveys may engender less active cognitive involvement. The latter finding appears to stand in contrast to Chang and Krosnick (Reference Chang and Krosnick2010), who find that computer respondents display greater differentiation than respondents who were administered their survey aurally over an intercom. However, it should be noted that their study measured differentiation with respect to eight feeling thermometer ratings; similarly, when we compared response patterns across different types of survey questions, a battery of feeling thermometer ratings stood out as an exception in which online respondents exhibited greater differentiation and less unit non-response than phone respondents. For these questions, it is likely that the task of providing a rating along a 101-point scale without visual assistance meant that a phone respondent faced a much more cognitively demanding challenge than did an online respondent, who saw a visual representation of the thermometer that changed as she moved her mouse. This is entirely consistent with the interpretation that greater satisficing will lead to less differentiation, and it seems that which survey mode tends to induce more satisficing will depend critically on the format of the question. Generally, future research could more fully examine how question format influences the relative level of satisficing, whether non-response is associated with particular kinds of questions or particular question wordings, and whether instruments can be devised to reduce respondents’ tendencies to use the “no opinion” response option.

Our results show that both the selection process and the mode itself are responsible for differences in response patterns. Our findings echo previous work that warns against making population-level inferences using opt-in internet panels (AAPOR 2010). In particular, among respondents who were allowed to select into their survey mode, we find that online respondents appear to be systematically different from telephone respondents in characteristics associated with access to and facility with personal computers and the internet. The differences in response patterns between online and phone respondents, controlling for the effects of selection, indicate that even when comparing or combining the results from probability-based internet samples, such as the KN panels analyzed here, with those from phone samples, analysts should be attentive to the differing levels of measurement error across mode.

Our substantive findings regarding mode effects and inferences about racial attitudes and political knowledge highlight the need to take mode effects seriously. Although internet surveys may reduce the effects of social desirability biases for questions on sensitive topics such as racial attitudes, online respondents’ greater propensity to answer “don’t know” may artificially depress evaluations of their political knowledge relative to that of phone respondents. Future research could further explore how various question wordings may help alleviate modal differences in how social desirability is encoded in the responses provided by internet and phone responses.

The findings presented in this paper do not refute the validity of either internet or telephone survey research. Rather, they indicate that telephone responses tend to perform better on some criteria (e.g., less item non-response), whereas internet responses seem superior on others (e.g., alleviating social desirability biases). Mixed-mode surveys may indeed be the best solution to negotiating the trade-offs associated with each mode, and we have highlighted some of the considerations researchers will want to engage when comparing responses with the same survey instrument collected across different modes. Some issues can be addressed in the survey design stage. Specifically, thinking carefully about the format of question responses, the ways in which questions are presented to respondents, and the process by which samples are recruited can all help to improve comparability between responses elicited across different modes. Even still, our evidence suggests that survey mode is likely to have considerable effects on patterns of response. Consequently, analysts of mixed-mode surveys should account for mode effects, especially if their variables of interest are confounded by selection into mode. This is only possible if the sampling and survey administration procedures provide some leverage to distinguish mode effects from selection effects, as in the YPP survey analyzed here.

Finally, additional research is needed to investigate whether the mode effects observed here are generalizable to other populations. Given the characteristics of the sample used in this study, the differences we uncovered may understate the potential magnitude of mode effects when surveying a sample from the general population. To the extent that the young people in our sample grew up in the Internet Age, we suspect that they will tend to be more comfortable in taking surveys online than are older generations. Thus, we suspect that the magnitude of any differences would be even larger when examining mode effects across a wider range of age groups. On the other hand, it is also possible that young people’s facility with quickly accessing internet content (and, perhaps lower levels of experience with telephone surveys) could have manifested in larger mode differences than would be observed in the general population. Examining how patterns of media usage condition the effects of survey mode is an important question for future research. As internet surveys become increasingly common, it is essential for survey researchers to know whether generational differences in survey responses are owing to genuine shifts in attitudes or are the artificial product of differential measurement error between groups with varying levels of comfort with online surveys.