One of the challenges of teaching political science is the discrepancy in experiences between undergraduates and government officials. Games that cast students as decision-makers may overcome this obstacle. When surveyed, students report that games enhance their understanding of abstract concepts and increase their interest in politics (Dougherty Reference Dougherty2003; Shellman and Turan Reference Shellman and Turan2006). Other analyses find that students learn more when instructors use techniques that engage them (Dorn Reference Dorn1989; Endersby and Shaw Reference Endersby and Shaw2009; Loggins Reference Loggins2009). Students are more likely to retain knowledge through activities that combine doing and saying rather than through other types of learning activities such as hearing, reading, or even doing and saying separately (Boyer et al. Reference Boyer, Caprioli, Denemark, Hanson and Lamy2000). Games can demonstrate how theories “work,” illustrate how institutions function, and get students to understand the emotions of actors in situations which are alien to them (Asal Reference Asal2005; Stover Reference Stover2007). Finally, politics makes the most sense if experienced or “played” rather than just discussed (Asal and Blake Reference Asal and Blake2006). Unsurprisingly, recent years have seen the publication of many articles on the use, construction, and integration of games and simulations in international relations instruction (Arnold Reference Arnold2015; Asal Reference Asal2005; Asal and Blake Reference Asal and Blake2006; Boyer, Trumbore, and Fricke Reference Boyer, Trumbore and Fricke2006; Dougherty Reference Dougherty2003; Haynes Reference Haynes2015; Simpson and Kaussler Reference Simpson and Kaussler2009; Wheeler Reference Wheeler2006). Some works challenge the efficacy of simulations. Raymond (Reference Raymond2010) found no statistically significant difference in learning between students that participated in a Model UN simulation and those that did not. The simulation, however, was not run by the instructor, resulting in a mismatch with course objectives.
Also, much of the evidence that games increase student learning is impressionistic (Shellman and Turan Reference Shellman and Turan2006; Wheeler Reference Wheeler2006) or relies on indirect assessments such as student surveys (Krain and Lantis Reference Krain and Lantis2006). Footnote 1 Extant works making direct assessments often employ natural experiments. For example, Frederking (Reference Frederking2005) reported that when he included a simulation in his Introduction to American Politics classes, exam scores improved significantly compared to years without a simulation. However, as Frederking notes, scholars are often limited in their ability to implement true experiments because we incorporate simulations for curricular reasons, not experimental ones.
Krain and Lantis (Reference Krain and Lantis2006) also directly assess the effectiveness of the Global Problems Summit simulation. Comparing the performance of simulation participants and non-participants on a pre- and post-test, they found scores improved roughly equally in both the control and test populations, but varied across groups in the areas of greatest improvement. This suggests that simulations may impart certain types of knowledge better than traditional instruction methods and vice versa. Such direct assessments remain rare and little has been done to build upon Krain’s and Lantis’ (Reference Krain and Lantis2006) insights to determine how games help students learn.
First, we directly assess the impact of participating in a game on student learning, allowing us to determine the effect of our game on student learning
We add to these findings in three ways. First, we directly assess the impact of participating in a game on student learning, allowing us to determine the effect of our game on student learning. Because games are time-consuming, it is important to gauge whether the time investment yields significant improvements in student learning. Second, we avoid using a game that directly mirrors substantive course material. We want to determine whether games illustrating similar concepts and mechanisms to the material being covered can be effective pedagogical tools, despite not directly reflecting the substantive material. Unfortunately, though many games exist, there is not a game for every topic. Since instructors may lack the time or expertise to construct their own games, the ability to use ready-made games that imperfectly fit the material being covered—but which illustrate similar mechanisms and concepts—would be a significant boon. Furthermore, by modeling the impact of a less directed gaming session, we may gain some insight into what students learn when they play thematically relevant games outside of class. Finally, we examine whether games benefit some students more than others. Perhaps avid gamers learn more because games illustrate concepts in a manner that is familiar to them. Prior studies have shown gamers and non-gamers have somewhat different learning styles (Gibson, Halverson, and Riedel Reference Gibson, Halverson, Riedel, Gibson, Aldrich and Prensky2007) and that gamers report increased interest in concepts they have been exposed to through games (Beedle and Wright Reference Beedle, Wright, Gibson, Aldrich and Prensky2007). Alternatively, perhaps non-gamers benefit more as they are exposed to new ways of seeing concepts. Either result would have considerable implications for how instructors incorporate games into the classroom (e.g., using classroom time to play games, versus providing students with supplementary materials). Likewise, we are interested in discovering whether the learning impact of games varies across student ability. Perhaps stronger students would gain more due to greater motivation. Or maybe weaker students would benefit more because games present an alternative to traditional teaching techniques that they are less able to follow.
In order to answer these questions, we had 39 undergraduate students in a 200-level international political economy course play an updated version of the 1960s board game Consensus. In the game, players allocate scarce campaign resources to win over domestic interest groups in the United States. With these groups’ support, players ultimately try to become elected President of the United States by winning states worth a majority of votes in the Electoral College. The game was incorporated during a unit on the effect of domestic politics on international trade policy.
EXPERIMENTAL DESIGN
Our experiment consisted of three components: a pre-quiz, a gaming session, and a post-quiz. Students were recruited from two introductory international political economy classes, an honors section (10 students) and a non-honors section (35 students), allowing us to assess the utility of games in assisting student learning between more and less advanced students.
Consistent with human subjects research protocol, students could opt out of sharing their data for research purposes; however, participation in the simulation was mandatory. The pre-quiz, participation in the game, and post-quiz were each worth 5% of a student’s grade, for a total of 15%. Thus, the stakes of the assignment were non-trivial. A substantial proportion of students participated in all parts of the exercise (39/45); most of the instances of non-participation resulted from student absences. Thus, we are reasonably confident our sample is not systematically biased by our recruitment process.
The pre-quiz consisted of two sections. The first section included six multiple choice questions on the relationship between electoral systems, domestic politics, and trade policy. Each concept drew from influential works on the politics of trade (Busch and Reinhardt Reference Busch and Reinhardt2000; Hiscox Reference Hiscox2003; McGillivray Reference McGillivray2004; Olson Reference Olson1965; Rogowski Reference Rogowski1987; Tsebelis Reference Tsebelis2002) and had been discussed in previous lectures and in the course textbook (Oatley Reference Oatley2012, 68–89). Students in the honors and non-honors classes were given different questions in order to prevent cheating. The quizzes are available in an online appendix. The second section inquired about student demographic information including: how frequently students played strategy board games or video games; gender; interest in the course topic; interest in international political economy; student assessment of their understanding of course material; and student assessment of their knowledge of which factors policymakers consider when they run in elections and govern countries.
Students were divided into groups to play Consensus. Students were provided with the rules to the game in advance, as well as a video demonstrating sample turns. In Consensus, candidates allocate scarce campaign hours to different states in a United States presidential election. Each turn, candidates simultaneously reveal the states they are campaigning in. More populous states require more campaign hours than less populous ones. Whichever candidate spends the most cumulative time in a state, leads in that state. Each state has ties to particular interest groups. For instance, Alabama has ties to the military-industrial complex, evangelical groups, and manufacturing interests. When a candidate takes the lead in a state, they gain the state’s campaigning hours toward interest groups that are influential within that state. A candidate controlling a majority of the hours for an interest group gains that group’s endorsement, granting them additional campaign hours to spend in states where the group is influential. A candidate wins the game by locking down a majority of the Electoral College. Our updated Consensus map, game rules, and interest group tables are included in the online appendix.
Consensus contains many useful features for explicating course concepts. The game involves an electoral contest under majoritarian rules, namely, the Electoral College. Because campaign hours are scarce, candidates’ campaigns often focus on battleground states, expending less effort in safe states. The importance of interest groups in the game also pushes students to envision parties as coalitions of interests, rather than ideological positions. Furthermore, some interest groups exhibit greater complementarities. For instance, the gun lobby and evangelical groups are often influential in overlapping states. As a result, students soon realize only so many interest group coalitions are electorally viable. Thus, Consensus makes students experience the ways American electoral institutions inhibit the emergence of a multiparty system and constrain political possibilities.
...Consensus makes students experience the ways American electoral institutions inhibit the emergence of a multiparty system and constrain political possibilities.
In the class following the gaming session, students took a post-quiz. The questions addressed similar underlying concepts to those in the pre-quiz, although the wording and details of each question were different. Students were also asked if they enjoyed the game, if they studied for the pre-quiz, post-quiz, both—or neither—and if they believed they had a good idea of which factors policymakers consider when running in elections and governing countries.
RESULTS
On average, student performance improved after playing the game. Students averaged 58.5% on the pre-quiz (SD = 1.393), and 65.8% on the post-quiz (SD = 1.317). We employed a paired sample t-test to assess whether individual student improvements were statistically significant. Footnote 2 Paired samples difference of means reveal that for the entire sample (N = 39) the result [t(39) = 1.716] was significant at the 95% level (p-value = .0471). Confining our analysis to the non-honors class (N = 32), the results were stronger [t(32) = 1.869] and significant at the 95% level, with a p-value of .0356. Thus, playing Consensus had a positive impact on student learning.
Additionally, student response to the game was largely positive. Many students conveyed they enjoyed the game and participated enthusiastically. When surveyed, 64.1% said they preferred the game to a lecture, while 33.3% were indifferent between the two, and 2.6% (a single student) preferred a traditional lecture format.
WHO BENEFITED FROM THE GAME?
Understanding who gains from games is essential to unlocking the underlying causal mechanisms by which games enhance learning. If a narrow subset of students (e.g., gamers, honors students) tends to gain from games, perhaps alternatives to games involving an entire class might be preferable. To assess who gained from playing Consensus, we used student performance on the post-quiz as our dependent variable, while controlling for performance on the pre-quiz. We employ a Tobit regression to address the issue that our data is censored (see Sigelman and Zeng Reference Sigelman and Zeng1999). What we are really interested in, here, is whether student knowledge of the domestic politics of trade policy changed. However, our post-quiz is limited in its ability to capture this. For instance, a strong undergraduate and a political science professor might both earn a perfect score, although the latter is more knowledgeable. Similarly, on the low end, two students with different (but low) knowledge of trade politics might both hypothetically score zero. Our Tobit regression analysis had a lower limit of zero and an upper limit of six, with five students hitting the upper limit. Footnote 3
We included three variables reflecting our questions about whether regular gamers benefit more from in-class games, our interest in the impact of games on honors and non-honors students, and our question of whether student enthusiasm predicted better results. To gauge whether students were gamers, we used the results of our survey question: “How often do you play board games and/or strategy video games?” We employed a variable ranging from zero (a student that never played games) to three (a student that often played games) to capture whether or not students were gamers. We also included a binary variable indicating whether students were in the honors program or not. In order to gauge the effect of student enthusiasm, we included a variable indicating whether students preferred the game to a lecture (+1), a lecture to the game (-1), or were indifferent (0).
We also included a set of controls. We controlled for performance (ranging from zero to six) on the pre-quiz. We also included two binary variables pertaining to studying behavior: one indicating whether a student studied only for the post-quiz, and another indicating whether or not a student studied at all. It is possible that some students saw their results improve because they studied only for the post-quiz. Additionally, if the game had some scholastic impact, its impact might be higher among students that studied the least, and thus, were not at a point where additional effort would yield diminishing returns. Finally, we controlled for gender. Because gamer culture skews male, excluding gender might undermine the robustness of the gamer variable. As table 1 shows, our sample did better on the post-quiz than the pre-quiz: 18% of students were honors students; 66.7% studied for both quizzes while 28.2% did not study; the gender balance was 59%-41% female-male; and the average student played games somewhere between “rarely” and “somewhat often.”
Descriptive Statistics for Tobit Model of Post-Quiz Performance
Student performance on the pre-quiz exhibited a positive, weakly significant impact on performance on the post-quiz (see table 2). Gamers also experienced statistically significant gains in quiz performance. In contrast, students studying only for the post-quiz and those that enjoyed the simulation exhibited a statistically significant worse performance on the post-quiz. Gender, honors status, and not studying for either quiz were all statistically insignificant.
Coefficients of Tobit Model for Post-Quiz Performance
Note: Robust Standard errors in parentheses
* p < .1, **p < .05, ***p < .01.
Among frequent gamers, the impact of playing the game was large. Using model 1, holding all binary variables at zero and non-binary variables at their mean, our model would predict a score of 3.088/6 for a student answering “never” to the gaming question, and a 4.477/6 for a frequent gamer. This relationship is depicted in figure 1.
Predicted Post-Quiz Score by Frequency of Gaming, with 95% Confidence Intervals
Our Tobit model sheds light on the causal mechanisms behind student learning in games. Although many students enjoyed the game, enjoyment predicted worse post-quiz performance. Footnote 4 We note that only one student expressed a preference for traditional lectures over the game. The low end of our “enjoyment” variable was dominated by students that enjoyed the game and lectures. Perhaps these students performed better because they gained from both learning experiences. Second, we wondered if students experiencing a poor result on the pre-quiz simply studied more, resulting in better post-quiz results. In fact, only two students fell into this camp, and those studying only for the post-quiz did worse. Interestingly, our findings suggest games can be advantageous for both honors and non-honors classes. Ceteris paribus, students in both honors and non-honors classes gained, challenging either the notion that games are only appropriate for advanced students able to follow complex rules, or the notion that games will have a larger impact on non-honors students.
DISCUSSION
Our most important finding is that gamers gained the most from our in-class games. We believe this result can be leveraged by instructors. In addition to using games in the classroom, instructors can encourage students to play games that are complementary to course concepts. For instance, a class exploring balance of power theory might encourage students to play Diplomacy, providing students with materials that facilitate debriefing. Instructors who play games could share debriefing notes for familiar games with instructors elsewhere. If, through collaboration, we could assemble a single centralized list of games, it would be simple for instructors to incorporate the list into their syllabi. A centralized list could help instructors looking for viable simulations, while also aiding professors in making recommendations to students that are gamers.
We certainly urge caution in the interpretation of our results. Our sample size (39 students) is not large. At the same time, we suspect that it would be difficult to incorporate our particular game in a larger class. Our institution is a medium-sized, urban university that has both ethnic and gender diversity. It is possible that characteristics of our university are favorable to simulations. Class sizes are often small enough that students get to know one another, making it easier to encourage collaboration. Thus, we expect our findings to hold best in similar institutions. Finally, we note that our exercise was not a true experiment, as we lacked a control group. Nonetheless, we believe our findings can be of help to instructors, as well as scholars researching games and learning.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S1049096516003218.
