The Mechanics: How It Works

Real-time polling technology has been used in popular television shows such as “Who Wants to be a Millionaire?” In our course, students had no trouble interfacing with these hand-held devices.¹

At the start of each class period, the instructor distributed the transmitters (which are about the size of a small remote control) to the students.² Students pressed buttons on the numerical keypad of the hand-held transmitters to indicate their responses to questions asked by the instructor. Two small infrared receivers, installed in the classroom, immediately transferred the information to a connected laptop computer, which tabulated the responses within seconds. Subject to the instructor's preference, a histogram could either automatically appear on the projection screen in the front of the classroom, or be summoned with a mouse click. Students were thus able to respond in “real time” to questions posed by the instructor.

After its initial introduction during the seventh lecture, the polling technology was used in nine of the remaining 12 lectures. Thus, it was relevant for many, but not necessarily all, of the class meetings. During these two-hour lectures, the instructor posed between five and 13 questions to students. Some were logistical in nature (e.g., “What time would you prefer to hold the review session?”). Others were used to check comprehension of the course material (e.g., “How well do you understand Converse's main findings?”; “How comfortable are you with interpreting these coefficients?”). Still others were standard survey questions relevant to the substance of the class (e.g., “Where would you place George W. Bush on the issue of abortion?”), and others were questions designed to spark discussion (e.g., “Do you think the media has more or less impact today compared with Lippmann's time?”; “What do you think should be the most important factor in citizens' presidential vote choice?”). All of these questions required response alternatives to be displayed onscreen along with the question, and up to nine response alternatives could be offered (although the number of alternatives was typically between two and five).

We designed two studies to help us evaluate the impact of the technology in this course. The first focused on the extent to which the technology improved students' awareness of the opinions of their fellow classmates. The second was a brief pen-and-paper survey administered at the end of the quarter gauging students' assessments of the technology.

Study 1: Improving Perceptions of the Class's Distribution

As this was a course on public opinion, we were interested in exploring the degree to which the technology might improve students' awareness of the opinions of others.³

We anticipated that by seeing a clear graphical presentation of query results, students would become more aware of their fellow students' attitudinal predispositions. To test this possibility, we designed and administered a pre-test/intervention/post-test study.

We administered a pen-and-paper pre-test on the sixth day of class, prior to introducing the polling technology. The pre-test consisted of 10 public opinion questions. In addition to giving their own opinion, the students were asked to estimate the distribution of the class on these response options, by indicating their estimates of the percentage of the class that would hold each response option. The order of the questions was split-sampled, with a random half of the students receiving the questions in one order and the other half in the reverse-order. Additionally, the task order was split-sampled, with a random half of the students giving their own opinion first followed by estimating that of the class second, and vice versa for the other half of the students. Following this pre-test, the polling technology was introduced in Class #7 and was used for four consecutive class periods. The post-test was administered in Class #12; the time interval between surveys was about 2½ weeks.⁴

Three of the 10 opinion questions appeared in identical form on the pre-test and post-test: opinion on abortion, military involvement in Iraq, and a racial resentment item. It bears noting that none of these questions was specifically asked during the four class periods.⁵

Hence, any improvement in perceptions of the class distribution can be attributed not simply to rote memorizing of the information that appeared onscreen but rather to inference based on other questions asked during the class periods.

For each respondent, we calculated the raw difference (henceforth, deviation score) between the student's estimate of the class mean (calculated from the student's estimates of the percentage of the class that would fall within each category) and the actual class mean (compiled from each student's self-placement) for each of the three repeated statements. Table 1 shows the individual statements, the average deviation scores in the two groups, and the two-tailed p-values for two-sample difference of means tests.⁶

On the abortion item, we see that before the technology was introduced, students on average viewed the class as more liberal than it actually was (indicated by the value of −0.68 in Table 1). Following the introduction of the technology, this leftward bias fell significantly to −0.22. For the question on military involvement in Iraq, we see that the class initially held a conservative misperception; in the post-test, this rightward bias significantly fell. We see the same pattern in the third item. The difference between the students' estimates of the class means and the actual class means was significantly smaller after the introduction of the polling technology for all three items.

A final item, asking students to indicate their party identification and to estimate the partisan composition of the class, was also repeated on the pre-test and post-test. Before using the polling technology, the students tended to overestimate the percentage of Republicans in the class compared with the actual percentage. The mean estimate of the percentage of Republicans in the pre-test was 25.8% (s.d. 15.1), but the actual percentage was 17.3%; in the post-test, the mean estimate dropped to 15.6% (s.d. 10.5), and the actual percentage was 18.6%. In absolute value, the average estimate in the post-test was smaller than the estimate in the pre-test (one-tailed p < 0.05), suggesting a significant improvement in estimates in the percentage of Republicans in the class. Further, the dispersion of estimates of percentage of Republican identifiers declined significantly (one-tailed p < 0.011). There was no difference between the pre-test and the post-test in estimating the percentage of Independents in the class (for both the pre-test and the post-test, the class on average underestimated the percentage of Independents by about five percentage points). For the percentage of Democrats, the class underestimated the percentage of Democrats in the pre-test (with a mean estimate of 59.5% (s.d. = 16.5) when the actual percentage was 65.4%), but then overestimated the percentage of Democrats in the post-test (with a mean estimate of 72.4% (s.d.= 13.5), when the actual percentage was 65.1%). On party identification, at least, the technology did not consistently improve accuracy of perceptions.

An alternative hypothesis is that the passage of time improved students' accuracy. Based on this simple design, we are unable to rule out this possibility. However, it is also likely that some students become self-conscious when asked to express an opinion in front of their classmates, especially when a controversial topic is being discussed, and particularly when there exists some expectation of what the majority of students will think (be it a predominantly liberal or a predominantly conservative campus). When students feel that their views represent those of a minority, or when they fear that their minority views may be misunderstood (Berinsky 1999), they may simply demur from revealing their views. This pattern of non-response could compound over the course of the term, yielding a “spiral of silence” that accentuates students' hesitation to voice their views and that eventually results in silencing students who feel that they may hold minority views (Noelle-Neuman 1974; 1993). As a consequence, class discussions, without this technology, could become more, rather than less, homogeneous. This process could stifle the expression of diverse opinions and thus distort students' perceptions of the class's preferences. This technology provides one way of improving students' sense of the opinions of others, while offering several other benefits, detailed below.

Study 2: Student Evaluations of the Technology

At the end of the quarter, students were given a brief pen-and-paper questionnaire. The students' responses to the polling technology were overwhelmingly positive, and they support our contentions of increased willingness to participate and increased knowledge of the views of others. The raw frequencies appear in Table 2. Nearly all students (98%) liked using the system in class; nearly 90% believed the polling technology made it easier to express honest opinions; and over 90% believed it helped the class discuss controversial issues. The vast majority (95%) disagreed with the statement that the technology decreased their interest in the course; nearly 90% believed that the technology helped them pay more attention in class; and nearly all (93%) disagreed with the statement that too much class time was devoted to the technology. Finally, 95% heartily endorsed its use in other courses, with two-thirds of the class giving the strongest endorsement possible.

At the end of the questionnaire, students were given the opportunity to provide open-ended comments. These comments reflect their wide-ranging, yet nearly unanimous, enthusiasm for the technology.⁷

Several of the comments reflected our expectations regarding how the technology would influence student participation, engagement, and learning. One student remarked, “I enjoyed the PRS technology very much. It helped me participate in class which is usually a bit difficult for me, because I am usually more reserved and shy.” Another student was intrigued by the results: “Thought these were fun and helpful. Some of the results REALLY surprised me.” Another student remarked on the extent to which the technology allowed for unbiased expression of views: “USE IT! Had an immediate comforting effect that allowed me to express my views and see where my preferences were relative to other data such as the NES without being too partisan in voice.” Finally, another student noted how the interface encouraged greater attention and engagement: “I was raised on video games and have no trouble admitting that this ‘tricked’ me into paying more attention than I usually would. In addition, it generally proved to be a really good discussion starter.”

Experiencing Research

There are several extensions to the applicability of the polling technology. As one example, the technology can be used to field a public opinion poll from the students at the start of a term. Having students respond to standard survey questions is valuable in many ways: it sparks introspection, a sense of empathy with survey respondents, and even criticism of and frustration with standard question wordings and response alternatives. In short, by having students actively respond to questions, they reflect more on the methodology of survey research. Further, because the responses are saved electronically, the responses from the poll can be converted to a usable dataset available for students to analyze during the course.

Another application of the technology incorporates elements of experimental research into courses. For example, the technology allowed us to gather real-time responses to Tversky and Kahneman's (1981) classic framing example:

Half of the class received the survival frame, and half of the class received the mortality frame (the questions were passed out on pieces of paper, so that students responded to only one version of the question). Each half of the class registered their preferences, and students could immediately see the impact of the framing experiment on their views.⁸

As another example of having students experience experimental manipulations, we used the technology to examine the impact of source cue manipulations. Half of the students were told a position was endorsed by a conservative political elite and the other half were told that the position was endorsed by a liberal elite, and they were asked to register their preferences.

The technology enables students to respond to survey questions and interact with experimental manipulations, giving them first-hand experience with the research that we as political scientists conduct. By directly experiencing the experimental stimuli, students were more engaged in the substance of the course material and could not hold themselves invulnerable to these framing and persuasion effects. They are transformed from passive observers to active participants, and this increases their critical evaluation of research methodology and empirical findings.

Checking In and Monitoring Progress

The polling technology also provides opportunities to monitor students' progress. Each of the transmitters has a unique identification number that can be linked to each student's name (although, to preserve anonymity, we did not take advantage of this in the public opinion course). The transmitters can therefore be used for taking attendance or even for in-class multiple-choice examinations.

The polling technology can also be used for assessing students' levels of understanding. Some students may feel too intimidated to raise questions for clarification. In this course, students were given several occasions to indicate whether they wanted clarification on substantive points and on assignments. When students saw that others in the class had questions, this seemed to legitimize their desires for clarification, and the floor was open for questions. Another way the technology assessed levels of understanding was through the incorporation of multiple-choice questions on concepts discussed in the lectures using pre- and post-lecture questions to determine if students showed improvement in their understanding of concepts following the lecture (which they did).

Other Applications

Instructors might explore still other applications with the technology. These include (but are not limited to) campaign simulations, demonstrations of electoral laws, and methodological courses in formal theory and statistical research methods. We have used the technology as part of a teaching collaboration centered on a campaign simulation. One instructor's class had the task of crafting a presidential campaign between three political parties, while the other instructor's class served as the mock electorate. We began the collaboration by having each of the three political parties contribute questions for a public opinion poll. We then fielded the public opinion poll among the mock electorate. The political parties analyzed the results of the public opinion poll to craft their campaign strategies. At the end of the term, each political party “ran” their campaign (by way of a presentation to the mock electorate). Following these presentations, the mock electorate voted in the final election using the polling technology. The instructors then led a discussion relating the campaign strategies to the relevant readings and lectures in both courses.

The technology can also provide efficiency gains to instructors who already use some form of balloting or hand-raising in their courses. For example, in courses on electoral laws, the real-time polling technology can be used to demonstrate several concepts, such as preference cycling predicted by Arrow's theorem. Instructors can also use the technology to illustrate the impact of manipulations of the number of choice alternatives, the number of decision makers, and the electoral rules (e.g., majority rule, Borda count, Hare system, Single Transferable Vote). For instructors who typically rely on hand-raises to illustrate properties of electoral laws, the real-time polling technology promises vast increases in efficiency and enables more spontaneity in demonstrations.

Another application of the technology applies to courses in formal theory and statistical research methods. The technology would enable instructors to play multiple player games (e.g., N-player prisoners' dilemmas and coordination games, public goods games, and beauty contests) in real-time with students to demonstrate and reinforce concepts of rationality. Game theory instructors could also ask students to, for example, solve a Nash Equilibrium in a simple game, register their answers, and then hold a class discussion regarding the results. For instructors of statistical research methods, the technology would easily lend itself to monitoring student comprehension of concepts and formulae, especially by having students solve problems, register their answers, and discuss how they arrived at their solutions.

Real-time polling technology has multiple applications for use in political science courses, from courses on public opinion and voting behavior to those on game theory and statistical research methods. It is, by no means, a panacea for the challenges of teaching lecture courses. It does, however, provide an opportunity for increasing student engagement, spurring discussion, monitoring student progress, demonstrating core concepts, and encouraging critical thinking regarding political science research.