Program assessment has become increasingly common as higher education seeks greater accountability to external constituencies and internal stakeholders including accreditation agencies and legislatures, university administrators, students, and parents—all of whom want to know how well students have learned and what competencies they have gained through their college education. Although assessment is not new, it is a nascent phenomenon in many institutions of higher learning in the United States, and changes in higher-education practices have made assessment more complex. One change is the widespread infusion of technology, especially online teaching. Assessment plans must have means and methods that can be used in online instruction as virtual classrooms augment—or replace—face-to-face modalities. However, existing literature on program assessment in political science rarely addresses strategies geared toward virtual-learning environments (e.g., Hamann Reference Hamann, Ishiyama, Miller and Simon2015; Young, Cartwright, and Rudy Reference Young, Cartwright and Rudy2014). This article briefly discusses the prominence of online teaching and learning in American higher education, evaluates what we know about student learning in online classrooms, and reflects on the implications for program assessment.
TECHNOLOGY IN HIGHER EDUCATION
Technological innovation, particularly the advance of the Internet, has thoroughly transformed higher education. College students have grown accustomed to a variety of learning environments, including flipped classrooms, in which active learning dominates the classroom and lecture material is assigned as homework; mixed-mode classes, in which a portion of the instruction is delivered online; and fully online courses. Consider the phenomenal growth and sheer number of students affected by online instruction: between 2002 and 2012, the percentage of students enrolled in at least one online course more than tripled, from 9.6% to 33.5% (Allen and Seaman Reference Allen and Seaman2014, 33). By 2014, public and private nonprofit four-year institutions continued to record rapid growth in the number of students taking one or more online courses at a rate of 7.2% and 12.7%, respectively, whereas private for-profit institutions experienced a decline of 8.7% for the first time (Allen and Seaman Reference Allen and Seaman2015, 5). Complete online degree programs are offered by an increasing number of institutions: 34.5% in 2002 and 62.4% in 2012 (Allen and Seaman Reference Allen and Seaman2013, 20).
In general, university administrators support the online model and value its pedagogical outcomes. In 2014, almost three quarters of academic leaders rated “learning outcomes for online instruction as the same as or superior to face-to-face instruction” (Allen and Seaman Reference Allen and Seaman2015, 5). Similarly, a US Department of Education meta-study concluded that “classes with online learning (whether taught completely online or blended) on average produce stronger student learning outcomes than do classes with solely face-to-face instruction” (Means et al. Reference Means, Toyama, Murphy, Bakia and Jones2010, 18). Yet, not everyone embraces the spread of the virtual classroom. Skeptics include professors of political science, some of whom view online teaching as a potential threat to or an attack on higher education in the United States (e.g., King and Sen Reference King and Sen2013a, 81; 2013b, 85). According to survey data, chief academic officers are aware of faculty concerns and reservations regarding online education; only 28% report that “their faculty members accept the ‘value and legitimacy’” of this instructional modality—virtually the same percentage as in 2003 (Allen and Seaman Reference Allen and Seaman2015, 6).
Because technology—and, in particular, the virtual classroom—is gaining an ever-growing presence in college education, it is imperative that online instruction be included in the assessment of programs in which students have the opportunity to complete part (or all) of their degree online. The Scholarship of Teaching and Learning provides a valuable first step in identifying how online courses can be assessed. When we better understand how instructors evaluate student learning in the online environment—and as we gain insight into successful (and unsuccessful) pedagogical strategies in that environment—we can consider how to integrate online or partially online instruction into program assessment.
…the existing disciplinary body of research on the effects of online and blended-learning classrooms, although still relatively small, nonetheless concludes that, overall, students are not negatively affected by taking courses in these technology-infused class settings.
WHAT WE KNOW ABOUT TEACHING AND LEARNING POLITICAL SCIENCE ONLINE
To gain a better understanding of what we know about teaching and learning with technology—and, more specifically, with courses conducted partially or entirely online—in political science, we identified all teaching-oriented articles published between 2005 and 2014 in three major disciplinary outlets for teaching and learning: PS: Political Science and Politics, Journal of Political Science Education, and International Studies Perspectives (Hamann, Pollock, and Smith Reference Hamann, Pollock, Wilson and Smith2015). Of those 481 articles, 70 (14.6%) discussed issues related to teaching with technology, which includes online teaching, mixed-mode instruction, clickers (i.e., student-response systems), online simulations (including games), the use of Internet sources, teaching with multimedia, software, and podcasting. In this article, we are particularly interested in online teaching. The dataset includes 16 articles that discussed courses taught entirely online and seven articles that are concerned with mixed-mode teaching (also known as “blended” courses, in which part of the instruction is conducted online and part face-to-face). Of these 70 articles, 40 (58%) evaluated the effectiveness of using technology in the classroom; others focused on describing how a particular technology is used in the classroom, how assignments are created in an online course, or on course design, for example.
To determine whether online teaching in political science is broadly evaluated in journals outside of the discipline, we selected two additional journals that focus on higher education with technology and that are distinguished by a high impact factor: Computers and Education and the International Journal of Computer-Supported Collaborative Learning. We coded more than 400 articles from these two journals (Hamann, Pollock, Smith, and Wilson Reference Hamann, Pollock, Wilson and Smith2015). In brief, we found that these two journals address questions related to technology with a higher frequency (75.3%) than the disciplinary journals, although the latter have recently given more attention to technology-driven instruction. The non-disciplinary, technology-focused journals, in turn, give scant attention to political science instruction; therefore, they are excluded from further discussion.
The question, then, is: What do we actually know about how and what students learn in online modalities? That is, does the existing scholarship of teaching and learning online in political science help those charged with program assessment to devise assessment plans that can incorporate online teaching?
SUBSTANTIVE FINDINGS ON TEACHING AND LEARNING ONLINE
Computer-mediated learning has become a fact of life for most people involved in higher education. Therefore, it is important to understand how political science instructors assess the effectiveness of online tools for education. Articles that evaluate online teaching and learning can be grouped into several broad categories. Some research assesses student-learning behavior, especially active learning, and is centered on virtual discussions. A second body of literature focuses on objective-knowledge gains. Other studies assess critical-thinking skills that are honed in the virtual environment.
One area of research focuses on the active-learning aspect of the virtual classroom, typically examined in the context of online discussions or simulations. Zeiser, Fuchs, and Engelkamp (Reference Zeiser, Fuchs and Engelkamp2013) studied the effects of discussions among students at an American and a German university as a supplement to their regular face-to-face instruction. Evaluating student essays, they observed an improvement in students’ understanding of cross-cultural differences and similarities as well as course concepts. Based on the students’ self-assessment, supplemental online discussions also are credited with producing higher-quality, in-class, face-to-face discussions (Trudeau Reference Trudeau2005). Schnurr, De Santo, and Craig (Reference Schnurr, De Santo and Craig2013) analyzed a blended-learning approach that uses a simulation; their findings suggest that students credit this approach with developing their understanding of complex policy environments and policy analysis (see also Parmentier Reference Parmentier2013). Roscoe (Reference Roscoe2012) found few differences in students’ academic performance in blended and face-to-face courses (see also Dolan Reference Dolan2008) but also noticed that students tended to drop out of online activities after a face-to-face session. Also assessing the effects of blended learning compared to face-to-face instruction, Wilson, Pollock, and Hamann (Reference Wilson, Pollock and Hamann2006) found that, overall, students gained more objective knowledge in a reduced seat-time setting compared to a course fully conducted face-to-face and, more specifically, that female students in the blended-learning environment had the highest gains in knowledge. Further evaluating online discussions, Smith and Bressler (Reference Smith and Bressler2013) used content analysis to gauge the effect of asynchronous discussion on student incivility and impoliteness and to explore the potential of anonymous discussions for students’ political-discourse education.
Focusing on student behavior in online discussions, Pollock, Hamann, and Wilson (2005, 12) demonstrated that student-to-student interaction is heightened when discussion groups are gender-balanced. Wilson, Pollock, and Hamann (2007, 140–1) created an index of active learning and found that students who are more active in online discussions—not simply posting but also reading posts—tend to achieve better course grades (see also Hamann, Pollock, and Wilson Reference Hamann, Pollock and Wilson2009).
Other research focuses on the contribution that online discussions can make to critical thinking. Williams and Lahman (Reference Williams and Lahman2011) used content analysis to uncover evidence of critical thinking in online discussions and concluded that when students are required to engage with one another in critical ways, they demonstrate more effective critical-thinking skills. These findings reinforce the importance of online discussions as a learning tool in higher education (similarly, see Hamann, Pollock, and Wilson Reference Hamann, Pollock and Wilson2009). Others considered the effect of online simulations on the development of student empathy. For example, Stover (Reference Stover2005) assessed the development of these character traits by referring to students’ open-ended responses on evaluations. This is particularly useful when trying to measure abstract ideas such as empathy.
Other studies focused on the role that virtual instruction has on content knowledge and attitudes. Botsch and Botsch (Reference Botsch and Botsch2012) tested changes in knowledge of American government, political attitudes, and behaviors using pretests and post-tests in both online and face-to-face classes. This research reinforces the notion that face-to-face courses have no advantage over online methods of instruction. Also using pretest and post-test experimental methods, Glazier (Reference Glazier2014) tested the effect of using satire on the political efficacy felt by students in both online and face-to-face courses. She concluded that, all things considered, the use of satire had a positive effect on political efficacy for students in both online and face-to-face classes. However, face-to-face students experienced a greater increase in efficacy compared to the control group in both situations. This difference is likely the result of the differences between students who sign up for online courses and those who enroll in face-to-face courses. More precisely, students in online classes often are older and more likely to have a heavier workload outside of the classroom (Glazier Reference Glazier2014, 869–870).
In summary, the existing disciplinary body of research on the effects of online and blended-learning classrooms, although still relatively small, nonetheless concludes that, overall, students are not negatively affected by taking courses in these technology-infused class settings. Given these findings, we now consider how online teaching can have a meaningful role in program assessment.
ONLINE TEACHING AND ASSESSMENT
How can online courses be used for meaningful program-assessment purposes? Conversely, how can assessment be designed to include online components of the curriculum? These questions are consequential because they address the validity of online instruction. It is crucial for programs that offer online courses to include them in their assessment; the assessment exercise documents what students have learned during their college education and identifies weak points in the curriculum, including online offerings.
At the same time, however, it is important that student work produced in online courses is treated no differently than the work produced in more traditional face-to-face courses. That is, students should be held to the same standards and expectations of achieving program goals as those who complete their college education entirely in a “brick-and-mortar” environment. The relevant issue is to integrate online courses into program assessment and to extract data that are useful for program assessment—not to design a separate assessment plan for the virtual classroom.
Pollock, Hamann, and Wilson (Reference Pollock, Hamann, Wilson, Deardorff, Hamann and Ishiyama2009) pointed out that work produced in online courses is stored and can be readily evaluated for assessment purposes. The authors contend that online classes are particularly conducive for evaluating objective knowledge, communication skills, and active learning and student engagement—provided that they are designed in such a way to facilitate or require active participation in the class. Thus, if the program assessment targets writing skills, for example, then online courses can serve as a repertoire of different types of writing (e.g., informal, used in discussions; formal, used in term papers and essays; and uploaded Power Point presentations used to assess presentation skills). Objective knowledge also can be extracted from materials submitted online. Essays and research papers already are digitally stored and can be reviewed using rubrics constructed specifically for program assessment. In addition, most course-management software programs allow for autograded quiz questions, which also facilitate the testing for objective knowledge. Footnote 1 These programs generally provide analytics for quiz questions, making it easy to determine which questions and concepts students have mastered and which ones are problematic. Critical-thinking skills can be assessed by analyzing discussion postings to determine the extent to which students demonstrate higher-level thinking using rubrics established for this purpose. That is, if in-class written assignments form a core part of assessment practices, then online classes can be a source of useful data available after the class has ended. More recently, many course-management software programs also allow for uploading voice and narrated Power Point files, which facilitates the assessment of verbal presentation skills.
At the same time, online courses or programs present challenges for assessment that are perhaps different from those in face-to-face courses or programs. For example, student demographics may be different in online classes: it is reasonable to expect that a higher proportion of nontraditional students—those with full-time jobs, those who live farther from campus, and those with family responsibilities—may be more likely to enroll in online courses or programs compared to traditional, full-time, residential college students (see, e.g., RITE n.d.). It might be useful to consider these systematic differences (where they exist) when interpreting data based on classroom performance. Furthermore, research suggests that the retention rate in online courses tends to be lower than in face-to-face courses, whereas other scholars find that online courses can increase retention and completion rates in higher education (e.g., Glader Reference Glader2013; RITE n.d.). Logistical issues may be present when program assessment includes comprehensive tests, such as nationally normed tests (e.g., the Educational Testing Service field test) because these are costly to administer online. If a department has composed its own exit tests, it can be challenging to find secure testing environments for online students. Another issue for departments to consider is the equivalence of the learning experience and assignments in face-to-face sections versus online versions of the same class. Although students are expected to have mastered the same material at the end of their courses and their college career, the educational process may have been significantly different. For instance, if students in an online section have weekly discussions and write short essays to demonstrate their mastery of the material whereas their peers in a face-to-face section are tested through two midterm exams and a final exam, their learning experience may be substantially different. In turn, these differences may be reflected in the assessment processes used, as well as in the outcomes.
The relevant issue is to integrate online courses into program assessment and to extract data that are useful for program assessment—not to design a separate assessment plan for the virtual classroom.
In summary, online courses must be worked into departmental and program-assessment processes when they are offered. They generally provide a wealth of material that can be used to gauge student performance in the program and therefore can usefully contribute to assessment. At the same time, it is crucial that assessment coordinators are aware of potential issues and difficulties when online instruction is integrated into program assessment.
CONCLUSION
Despite the rapid increase of instructional technology—and online instruction in particular—in colleges and universities, we still know relatively little about how and how much students learn when they take virtual classes in political science. Existing scholarship of teaching and learning that analyzes online political science instruction has not identified any systematic negative effect on student learning; however, this research still leaves much to be discovered about online teaching and learning. Yet, program assessment also considers what students learned in online courses (where they are offered). Student work delivered in online courses can be usefully analyzed for program-assessment purposes. Regardless of the course-delivery method, it is crucial that what a program chooses to assess (i.e., general objectives as well as specific measures) is aligned with program goals and missions and that assessment results inform curriculum and program revisions with the intent of improving student learning.
