COLLECTING DIVERSE LARGE-N DATA IN NATURAL SETTINGS

Collecting large samples of diverse respondents in a natural setting is a challenge for our discipline. Although nationally representative surveys can achieve this end, they are generally very expensive. Students, however, often are willing to participate and are far more affordable. Yet, they present at least two concerns for external validity (Mintz, Redd, and Vedlitz 2006; Peterson Reference Peterson2001).

First, student samples are not representative of general adult populations (Oakes Reference Oakes1972; Sears Reference Sears1986). This concern often is overstated, however, because students tend to resemble adult populations across a range of important covariates, such as partisanship and media use (Druckman and Kam Reference Druckman, Kam, Druckman, Green, Kuklinski and Lupia.2011, 51). Moreover, if scholars are interested in estimating relationships between variables, they can use student samples to create valid inferences—even in cases in which the sample differs substantially from the population. If a treatment effect of interest is homogeneous in the population, any sample can produce an unbiased estimate. However, even if the treatment effect varies, it can be modeled as long as the sample provides variation across the relevant moderating variables. Thus, unbiased estimates of treatment effects require diverse but not representative samples. For example, in the case of presidential debates, the effect of candidate attention to immigration on viewers’ attitudes toward the candidate might depend on a viewer’s ideology and race. In this case, unbiased estimates would depend on obtaining a sufficient number of respondents across the ranges of ideology and race but would not require the sample’s percentage of conservatives or African Americans (for instance) to equal those in the population (Druckman and Kam Reference Druckman, Kam, Druckman, Green, Kuklinski and Lupia.2011). Many single-campus student samples may lack this needed variation.

Second, student-based studies generally are conducted in artificial settings—often a computer lab. Laboratory environments tend to eliminate distractions, resulting in treatment effects that are larger than those in natural settings (Jerit, Barabas, and Clifford Reference Jerit, Barabas and Clifford2013). One solution is to allow participation in more natural settings (Kinder Reference Kinder2007) in which distractions introduce variation in participant attentiveness (e.g., Albertson and Lawrence Reference Albertson and Lawrence2009). However, technological and logistical limitations often impede this approach.

Crowdsourcing data collection can mitigate both concerns. A relatively new concept in business and an even newer concept in academia, crowdsourcing is “a strategic model to attract an interested, motivated crowd of individuals capable of providing solutions superior in quality and quantity to those that even traditional forms [can]” (Brabham Reference Brabham2008). ^{Footnote 1} Our approach, described in detail below, builds on crowdsourcing work by reaching out to the political science community to access a more diverse student-respondent pool participating in more natural settings. Of added benefit, this approach provides instructors with resources to facilitate classroom discussions—and may even heighten student engagement in the political process.

COLLEAGUE CROWDSOURCING FOR THE 2012 PRESIDENTIAL DEBATES

Our substantive interest is to understand how candidate debate behaviors affect viewers’ attitudes (Boydstun et al. Reference Boydstun, Glazier, Pietryka and Resnik2014). Despite the salience and visibility of presidential debates (Benoit, Hansen, and Verser Reference Benoit, Hansen and Verser2003; Jamieson and Birdsell 1990; Marcus and Mackuen Reference Marcus and Mackuen1993), few studies have collected real-time reactions that allow for the study of individual debate moments; those that have done so use very small samples (e.g., Fridkin et al. Reference Fridkin, Kenney, Allen Gershon, Shafer and Woodall2007; McKinney and Rill Reference McKinney and Rill2009; Pfau, Houston, and Semmler 2005).

Thus, we set out to measure debate reactions using a web application, or “app,” that we designed for use on smartphones. ^{Footnote 2} The app was also accessible from tablets and personal computers, allowing viewers to react to the debates in real time from anywhere with Internet connectivity. A screenshot of this app, React Labs: Educate, is displayed in figure 1. Respondents used the app while watching the debates live, indicating (at any time they wished) whether they “agreed” or “disagreed” with the candidates and whether they thought the candidates were “spinning” or “dodging” the question.

Figure 1

React Labs: Educate App Interface

(Color online)

We needed a larger, more diverse sample of app users than any of our campuses could provide in isolation or combined. Therefore we targeted our recruitment efforts at instructors across the country, knowing that they are uniquely able to encourage student participation (e.g., in exchange for extra credit). To encourage instructors to register their classes and promote participation, we designed an incentive package aimed at helping them to achieve some of their own teaching and learning goals.

The materials we provided to registered instructors are available on the project website (http://reactlabseducate.wordpress.com). Before the debates, registered instructors received the following materials:

• • PowerPoint slides and lecture notes covering the history of presidential debates—including YouTube links to memorable debate moments as well as research on debate rhetoric, debate strategies, and debate effects

• • discussion questions

• • a list of resources, websites, and research collections on presidential campaigns and debates

• • citations and abstracts of relevant debate research

• • alternative assignments for students unable to watch the debates live

After the debates, registered instructors also received the following:

• • Within 12 hours of each debate: presentation-ready PowerPoint slides with preliminary results from respondents who used the app

• • After the final debate: for each debate, a list of their students who participated

These resources linked political science teaching and research, helping instructors discuss the debates in a way that connected theory with contemporary politics.

We recruited instructors by sending more than 120 individual e-mails inviting colleagues to participate in the project and by sending invitations to key listservs and blogs. ^{Footnote 3} Instructors registered their classes to participate through the project website. Each registered course was assigned a unique course identification number, which enabled us to send instructors confirmation of their students’ participation but also required us to send a unique e-mail with instructions and the course identification number for each registered class. This challenge was made easier by Gmail’s Mail Merge, which allowed us to merge e-mail addresses, course identification numbers, instructor names, and course names from a database into individual e-mails, thereby automating the process of sending individualized messages. ^{Footnote 4}

We embedded a predebate survey in the app itself and used a paid (but relatively inexpensive) subscription to SurveyMonkey® to administer a postdebate survey. Survey Monkey® provided the capacity to handle a high volume of student participants, to ask a large number of follow-up questions, and to download the results in a spreadsheet.

Following through on our promise to provide next-day figures and preliminary results proved challenging. We offered our graduate students free food and good cheer to stay up all night after each debate, crunching numbers and compiling PowerPoint slides. Although the process was labor intensive, we felt that providing instructors with immediate results that they could use in class to facilitate discussions of the debates was a critical incentive for participation.

Our research design represents a major advance in external validity. In terms of representativeness, the app allows us to draw on a large and diverse enough sample to include the variation we need for analysis. In terms of artificiality, the app allows students to participate in the study from wherever they would normally watch a debate (e.g., home, a friend’s house, or a debate-watch party).

THE TEACHING AND LEARNING BENEFITS OF CROWDSOURCING

In addition to the methodological and logistical benefits of our crowdsourcing approach, our solution facilitated teaching and learning. Because of their salience and scale, presidential debates represent key opportunities to encourage student engagement with the political process, which can improve political knowledge and civic skills—especially among those with lower initial levels of political interest (Beaumont et al. Reference Beaumont, Colby, Ehrlich and Torney-Purta2006). When instructors highlight engagement and civic themes, their students’ future political engagement and voter turnout increase (Hillygus Reference Hillygus2005; McCartney, Bennion, and Simpson Reference McCartney, Bennion and Simpson2013). Furthermore, watching debates tends to boost political efficacy, trust, and information among youth while decreasing cynicism (Kaid, McKinney, and Tedesco 2007; McKinney and Rill Reference McKinney and Rill2009). Many of our student participants likely would not have watched the debates were it not for the app and the incentives that we encouraged instructors to offer. Even for those students who would have watched anyway, using our app turned watching TV—a generally passive activity—into an interactive experience. Extensive research has demonstrated that active learning techniques improve test scores (McCarthy and Anderson Reference McCarthy and Anderson2000), engagement with the material (Brown and King Reference Brown and King2000; Hess Reference Hess1999; Ruben Reference Ruben1999; Wolfe and Crooktall Reference Wolfe and Crooktall1998), learning (Pace et al. Reference Pace, Bishel, Beck, Holquist and Makowski1990; Perry Reference Perry1968; Sutro Reference Sutro1985; Washbush and Gosen Reference Washbush and Gosen2001), and interest (Hess Reference Hess1999; Smith and Boyer Reference Smith and Boyer1996). Although we do not directly measure these effects here, the literature leads us to expect that using the app aided student learning.

Our crowdsourcing method benefited instructors as well. During the month of October 2012, our publicly available webpage featuring overnight result summaries was accessed more than 5,000 times. In addition to the result summaries, participating instructors accessed our password-protected teaching-resources webpage 450 times. We view the teaching benefits of our study—providing instructors with easy-to-use classroom materials and a method by which to actively engage students in the political process—as a hopeful indication that the colleague-crowdsourcing approach can facilitate a symbiotic relationship between teaching and research.

THE FUTURE OF COLLEAGUE CROWDSOURCING

We believe colleague crowdsourcing holds considerable promise for future studies, particularly in light of ongoing technological innovations, which make national (or even international) crowdsourcing increasingly feasible. Our app facilitated crowdsourcing by enabling participation across the country, but there are many other potential uses of colleague crowdsourcing; we certainly do not expect all scholars to create an app.

For example, colleague crowdsourcing might be used to foster large-scale and geographically diverse participation in studies using survey platforms such as Qualtrics® and SurveyMonkey®—or participation by specific target groups, such as first-generation college students or Muslims. Colleague crowdsourcing could be used to collect simple cross-sectional survey data, panel data during the course of an academic term, or data derived from survey experiments. It also could be used to measure aspects of the political environment (e.g., counting yard signs or political bumper stickers). In addition, we can imagine the incentive portion of the crowdsourcing approach taking many forms, including access to the data, webcast guest lecturers, and research notes on the findings for use in class. With enough lead time to include information about a study in their syllabi and/or to incorporate time for discussion in their lecture plans, many instructors may be keen to encourage student participation in an interesting study. In short, the crowdsourcing approach as a recruitment technique is flexible and scalable. Overall, new research technologies coupled with colleague crowdsourcing create a rich opportunity to incorporate research methods, local and global findings, and temporally relevant data in the classroom in a way that can aid research efforts while stimulating a new level of active learning.