BACKGROUND

The extant scholarship on teaching and learning has investigated whether and how modality can affect student learning (Alpert, Couch, and Harmon Reference Alpert, Couch and Harmon2016; Botsch and Botsch Reference Botsch and Botsch2012; Clawson, Deen, and Oxley Reference Clawson, Deen and Oxley2002; Daigle and Stuvland Reference Daigle and Stuvland2020; Glazier et al. Reference Glazier, Hamann, Pollock and Wilson2020; Hamann, Pollock, and Wilson Reference Hamann, Pollock and Wilson2009; Pollock and Wilson Reference Pollock and Wilson2002; Wilson, Pollock, and Hamann Reference Wilson, Pollock and Hamann2007; Xu and Jaggers Reference Xu and Jaggers2014). These studies, however, treat modality as a fixed course structure. Studies outside of political science have leveraged the emergency shift online due to COVID-19, finding some evidence that students learned more after moving online (Seitz and Rediske Reference Seitz and Rediske2021) but also that students were less confident in their learning (Prokes and Housel Reference Prokes and Housel2021), less satisfied with their experience (Kumalasari and Akmal Reference Kumalasari and Akmal2021), and felt more stress and disengagement (Besser, Flett, and Ziegler-Hill Reference Besser, Flett and Ziegler-Hill2022).

Unconsidered in political science and beyond are repeated modality changes throughout the semester. Whereas within-semester shifts in modality are a relatively new phenomenon for scholars of teaching and learning to study, they are likely here to stay. Internet-based technology, the proliferation of course modalities, and students’ and instructors’ experience with different modalities have created a new equilibrium in which courses can continue on schedule despite occasionally being unable to be conducted in person. However, these temporary modality shifts place unique demands on students and instructors that could come at a cost to student learning. The degree to which shifts induce uncertainty and stress and foster disengagement can hamper learning (e.g., Vogel and Schwabe Reference Vogel and Schwabe2016). The potential negative impact of shifts on learning may require instructors to adapt how they communicate with students and present course material to mitigate learning loss.

RESEARCH DESIGN

We address our research question by tracking the number of shifts in modality across 10 sections of “Introduction to American Government”—a required general education course at a large university—taught by five different instructors in Fall 2021 (Jansa and Ringsmuth Reference Jansa and Ringsmuth2022). In the event of a positive COVID-19 case among students enrolled in the course, instructors were allowed to (1) remain face-to-face with masks required; (2) switch to a hybrid format with masks required for the face-to-face portion; or (3) switch to a fully online (synchronous or asynchronous) format. Changes in modality were required to persist for two weeks, but additional positive COVID-19 cases could lead to an extension for an additional two weeks from the time the positive case was reported. We asked instructors to report their experiences with temporary modality shifts and then coded the number of shifts for each section (e.g., transitioning from face-to-face to online due to a positive COVID-19 case counted as one shift; moving back to face-to-face learning counted as a second shift). As table 1 illustrates, we observed some sections with no shifts in modality, whereas other sections experienced up to five during the semester. This number of shifts variable is used as a key independent variable in the following analyses.

Table 1

Variation in Modality Shifts by Section (Fall 2021)

Students were asked to participate in a two-wave survey to gauge changes in their self-assessed knowledge about American government. Wave 2 also measured students’ perceptions of how well instructors handled temporary modality shifts if their section experienced at least one shift during the semester. Additionally, we gathered data on students’ demographics and their academic background.Footnote ² In all, 928 students completed wave 1, a response rate of 54.9%; 724 students completed wave 2, a response rate of 42.9%. A total of 587 students completed both waves, which allowed us to pair their responses and measure growth during the course of the semester.

STUDENTS THOUGHT INSTRUCTORS EFFECTIVELY HANDLED MODALITY SHIFTS

To understand student satisfaction with how instructors handled changing formats, we asked them to register their support on a five-point scale ranging from “strongly disagree” (0) to “strongly agree” (4) for the following statement: “Overall, my instructor handled the temporary change(s) to delivering the course well.” Figure 1 exhibits notable skew because students resoundingly were pleased with how their instructors navigated the temporary shifts from face-to-face classes to online sessions (mean = 3.2; median = 4.0). The data suggest that it is possible for instructors to guide students through significant and unpredictable disruptions to one of the most fundamental aspects of a college course—that is, the format of content delivery—in a positive way. By this time—the second full semester during the pandemic—instructors had found ways to navigate frequent, unplanned shifts from face-to-face meetings to remote learning to the satisfaction of students.

Figure 1

Distribution of Instructor Handling of Temporary Modality Changes

CLARITY, EFFECTIVE TECHNOLOGY USE, AND ENGAGING MATERIAL MAKE SHIFTS NAVIGABLE

To understand what made instructors successful at handling modality shifts, we examine three potential contributing factors: whether students thought instructors (1) provided clear instructions; (2) used technology effectively; and (3) made course material engaging (all using a five-point scale).Footnote ³ Figure 2 displays coefficients and 95% confidence intervals for a model with students’ overall assessment of the instructors’ handling of temporary modality changes (five-point scale) as the dependent variable, which included instructor fixed effects to control for instructor idiosyncrasies.Footnote ⁴

Figure 2

Instructor Handling of Temporary Modality Changes

Ordinary least squares regression with instructor fixed effects.

The results indicate that how instructors use technology to facilitate a temporary shift to online learning, communicate when executing modality changes, and present course material during shifts each relate significantly to students’ views on how well an instructor handled unplanned modality changes. Combined, these results suggest that under challenging circumstances—when available options and instructor autonomy may be restricted (e.g., by institutional policies or personal circumstances)—instructors retain tools to significantly mitigate the disruptiveness of changes in course modality.

A NEGATIVE IMPACT OF MODALITY SHIFTS ON STUDENT LEARNING

Although the general consensus among survey participants that instructors handled the modality shifts well is encouraging, we also are interested in whether these shifts influenced learning. We operationalize student learning as change in students’ self-assessment of political knowledge because it can be measured consistently across sections and instructors.Footnote ⁵ Moreover, it is positively associated with a key objective of American government and similar civics education courses: encouraging students to become active citizens by participating in politics (e.g., Lee and Matsuo Reference Lee and Matsuo2018).

Students were asked to self-assess their political knowledge in five areas: (1) what distinguishes the two parties from one another; (2) how elections work; (3) what the Constitution says; (4) how laws are made; and (5) how power is divided among the three branches of government. Participants rated their confidence in understanding on a five-point scale ranging from “not confident at all” to “extremely confident.”

We calculated each student’s score as an additive index in which the maximum score was 20 (i.e., “extremely confident” on all questions) and the minimum score was 0 (i.e., “not confident at all” on all questions). Students’ responses to these five questions created a strongly reliable scale with Cronbach’s alphas of 0.83 for wave 1 and 0.86 for wave 2. We took each student’s wave 2 score and subtracted the wave 1 score to capture change in knowledge confidence. Overall, students exhibited an average increase of 2.38 points, or about 12% growth during the semester.

As a first step, we examine the bivariate relationship between shifts and growth. We find that mean growth in knowledge confidence is significantly greater for students experiencing no modality shifts (3.33) compared to those who experienced at least one change (2.33, p = 0.1). This suggests a possible weak impact of modality shifts on student learning. Using chi-square tests, we also find that students who experienced more shifts were less likely to report feeling actively engaged and less likely to attend class (p<0.01 for both). This reinforces the idea that within-semester shifts can negatively impact students by reducing interactions, attendance, and student learning.

HANDLING TRANSITIONS WELL CAN MITIGATE LEARNING LOSS FOR A FEW SHIFTS

These initial results show a weak, negative impact of shifts on students’ learning and stronger negative impacts on students’ attendance and perceived opportunities for interaction. However, the manner in which professors provide instructions, use technology, and present course material can help students to navigate short-term modality shifts. Given these results and what we know from the extant literature, we expect that the impact of temporary modality shifts on student learning will be lessened when instructors handle these disruptions well.

To measure instructor handling of modality shifts, we create an additive index that combines whether an instructor provided clear instructions, effectively used technology, and made course material engaging with the overall instructor handling question. The index ranges from 0 (“strongly disagree” on all questions) to 16 (“strongly agree” on all questions) and has a mean of 12.2, which again reflects a generally positive assessment of how instructors handled temporary modality changes. These four questions create a strongly reliable scale (Cronbach’s alpha = 0.88).

We use this measure as a key independent variable in a series of hierarchical linear models that test the impact of the number of shifts and an instructor’s handling of them on learning. The models account for the nested nature of the data with instructor- and section-level random effects. Change in knowledge confidence serves as the dependent variable in each model. We use the number of shifts measure to capture modality changes. The results are displayed in table 2.

Table 2

Change in Confidence in Knowledge Hierarchical Linear Model

Note: Standard errors in parentheses; ⁺p<0.1, *p<0.05, ^**p<0.01.

Model 1 examines only the number of shifts. As expected, more changes in modality is negatively related to growth in students’ knowledge confidence. An additional modality shift in a section corresponds to a 0.19 decrease in students’ assessment of their knowledge, further suggesting that within-semester modality shifts can negatively impact student learning (p<0.1).

Model 2 adds the instructor handling index as an independent variable, which excludes those who did not experience a modality shift (i.e., online-only sections).Footnote ⁶ We find in Model 2 that a one-unit increase in students’ ratings of an instructor’s handling of modality shifts corresponds to an 0.08 increase in knowledge confidence during the semester. Furthermore, the negative effect of the number of shifts dissipates with the inclusion of the instructor handling index. The results suggest that skilled instructor handling of modality shifts can enhance student learning.

To examine whether instructor handling can mitigate the impact of modality shifts on student learning, we include an interaction between the number of shifts and the instructor handling index in model 3. We also control for students’ wave 1 knowledge confidence because students who start with higher levels of confidence have less room for growth (Meirick and Wackman Reference Meirick and Wackman2004) and they may overestimate their confidence before learning about politics (Rogers and Gooch Reference Rogers and Gooch2021). We also controlled for student-level factors (i.e., class standing, GPA, gender, racial/ethnic minority status, and parents’ education) that may have varied across types of sections and respondents and affected students’ self-assessment of learning.Footnote ⁷

The results in model 3 suggest that instructors can take actions—particularly when there are three or fewer modality shifts in a semester—that shield students from problems related to unplanned changes to course modality. Figure 3 plots the conditional marginal effect of modality shifts on student learning. We hold the instructor handling index at its mean value that, given its distribution, represents positive assessments of how instructors handled modality shifts. The x-axis displays the frequency of modality changes present in the data. Figure 3 demonstrates that the marginal effect of good instructor handling on students’ knowledge confidence is strongest when the number of modality shifts is lower (a 0.16 increase in knowledge confidence; p = 0.01 for two shifts) and decreases as the number of shifts increases (0.11, p = 0.01; 0.05, p = 0.09; and 0, p = n.s. at three, four, and five shifts, respectively). Although there are limits to instructors’ power, they can take steps to offset the potential negative consequences of a few unplanned modality shifts during a semester.

Figure 3

Margins Plot for Shifts X Instructor Handling Index

The model also indicates that minority students reported less growth in knowledge confidence, reinforcing the idea that there are ongoing challenges to equity in civic education. Additionally, students with a higher GPA and a higher level of knowledge confidence at the beginning of the semester reported less knowledge-confidence growth, which suggests that students from advantageous positions (i.e., a strong academic record and confidence in their political knowledge) may have less room to increase their understanding of politics.Footnote ⁸