In 2010, with the approval of the American Political Science Association (APSA) Council, the APSA Standing Committee on Teaching and Learning formed a Working Group on Assessment. Its purpose was to learn what political science programs were doing to meet assessment demands and whether APSA should have an expanded role to help departments respond. Creation of the Working Group Footnote 1 followed a multiyear discussion by members of the APSA Council, which concluded that the organization should facilitate assessment. Meeting at the 2011 Teaching and Learning Conference, the Working Group on Assessment realized that although assessment conference papers had offered evidence of excellent assessment practices at many institutions, it was less clear how most political science programs throughout the country were conducting and using assessment. To enhance its understanding, the Working Group on Assessment decided to conduct a national survey of political science programs to learn more about their assessment practices.
This article briefly introduces program assessment, presents results of the national survey, and suggests a potential role that the APSA could offer. Discussion of the survey provides increased understanding of issues related to assessment methods, motivation, use, and resources.
EVOLUTION OF DEMANDS FOR ASSESSMENT
Typically, faculty members begin their first professional position knowing little about the purpose or practice of outcomes assessment. Yet, assessment has been an increasing expectation for faculty and academic administrators since a series of reports in the early 1980s sounded the alarm that higher education was not keeping pace with the educational needs of the country (Association of American Colleges 1985; National Institute of Education 1984; US Department of Education 1983). Most of the reports also recommended that colleges and universities systematically assess student-learning outcomes for purposes of educational improvement and accountability. As discussed in McClellan’s article in this symposium, regional accrediting agencies expected institutions to develop an assessment plan by the late 1980s. Nevertheless, many in higher education believed assessment was a passing fad. This view, still held by some programs today, is captured by one of the APSA study’s respondents who wrote, “[A]ssessment is just another round in the 100-year conversation about distribution, sequencing, essential skills, and content.” However, unlike other higher-education fads, assessment has become institutionalized in the regulatory processes of higher education and, as we teach in our own courses, changes that have been institutionalized are more likely to endure. Regional accrediting agencies have incrementally increased their assessment demands so that today, every academic program is expected to have and use an effective student-learning-outcomes assessment process.
After 30 years, it is evident that those who believed the assessment fad would fade underestimated its staying power. Alternatively, assessment has not proven to be as transformative as advocates once proclaimed.
PROGRAM ASSESSMENT EXPECTATIONS
In the 1980s, early advocates for assessment—including Northeast Missouri State University (now Truman State University), Alverno College, King’s College, and the University of Tennessee—received significant national attention for their efforts (Ewell Reference Ewell1984; Hutchings Reference Hutchings1987). Their assessment systems tended to be university-wide processes led by campus presidents and provosts with supporting activity at the program level. The 1990s witnessed an increase in the enthusiasm for classroom assessment techniques (Angelo and Cross Reference Angelo and Cross1993). However, as powerful as classroom assessment can be for a faculty member, it does not meet the program assessment requirements of regional and specialized accreditation. Today, program-level assessment is the primary focus of regulators’ attention.
In the APSA assessment survey, 84% of political science departments responded that they had adopted or were in the process of developing student-learning outcomes.
Conceptually, program assessment is a process that begins with the establishment of expected student-learning outcomes. Faculty and staff are encouraged to identify “What will program participants know and be able to do on completion of the program?” From that base of agreement, it follows that faculty have the potential to structure curricula to elevate achievement of desired student-learning outcomes. As data are collected and evidence-based deliberation occurs, reformed practices can be initiated to increase student outcomes.
Typical of many cyclical processes, the flow from one stage to another often is less than linear. Faculty are expected to adopt program outcomes, develop and implement assessment measures, and use collaborative discussions about assessment results to seek continuous improvement. From the public’s point of view, this should be easy to accomplish; however, assessment experience makes it clear that rather than being a seamless process, endless loops, stalled efforts, information gestation, and sporadic attention should be expected.
SURVEY METHODOLOGY
The APSA Working Group on Assessment contemplated the idea of constructing its own survey items to learn about assessment practices used in the political science discipline. However, in its review of other assessment surveys, members discovered a national survey on departmental program assessment conducted by the National Institute for Learning Outcomes Assessment (NILOA) (Ewell, Paulson, and Kinzie Reference Ewell, Paulson and Kinzie2011). One of NILOA’s primary findings was that the social sciences were the least likely to assess student learning and the least likely to use assessment results. Working group members accepted that many in the discipline were unenthusiastic about assessment, but it was not evident that social science lagged behind other arts and sciences disciplines. However, the NILOA questionnaire was similar to the one that the APSA Working Group sought to create, and using NILOA’s questions offered the opportunity to compare APSA survey results to the NILOA “all departments” and “social sciences” metrics.
After securing permission from the NILOA to use its questions, finalizing the APSA questionnaire, gaining Institutional Review Board approval, and coordinating with APSA survey processes, the questionnaire was sent in April 2013 to all political science departments at institutions offering undergraduate baccalaureate degrees. Questionnaires were returned from 252 programs—that is, 21% of the distribution list. Compared to previous assessment surveys that used the APSA department list, the 2013 survey response rate exceeded that of 17% for Kelly and Klunk’s assessment survey (2003) but was less than the 24% response to a 2007 assessment survey (Ishiyama et al. Reference Ishiyama, Parker, Quinn and Young2007). Categorized by highest degree offered, 62% were baccalaureate, 22% were masters, and 16% were doctoral-degree–granting programs. According to APSA staff, the response rate was consistent with its departmental surveys.
The NILOA survey used a two-stage process that began with contacting chief academic officers at all 2,719 accredited undergraduate institutions in the United States (Ewell, Paulson, and Kinzie Reference Ewell, Paulson and Kinzie2011). They were asked to provide contact information for specified program areas that had been randomly assigned by NILOA researchers. In the second stage, the NILOA sent questionnaires to heads of programs; from that list, a 30% response rate totaling 982 responses was received. Because responses did not yield sufficient numbers to report by discipline, the NILOA created 11 disciplinary “clusters” for reporting. Political science was part of the social science cluster.
Both the APSA and the NILOA surveys had lower response rates than desired and the NILOA had an additional selection process in the first stage that might have affected which programs were chosen in the second stage. Therefore, it is reasonable to suspect that programs engaged in assessment may be overrepresented in the responses. It is also possible that those who were not in favor of assessment had an additional motive to respond.
HOW HAS POLITICAL SCIENCE RESPONDED TO ASSESSMENT PRESSURES?
Prior Studies
As regional accreditors increased demands for assessment, programs in political science have had to respond to campus decennial deadlines for regional accreditation reports and site visits. In some cases campuses created sustained assessment programs but on many campuses assessment has been an episodic effort. As accreditation agencies have begun changing their processes to include ongoing review of institutions between site visits, the pressure on political science programs to solidify their assessment systems continues. It seems important to examine how the discipline has responded.
A few previous studies offer insight to the discipline’s assessment practices. In their survey, Kelly and Klunk (Reference Kelly and Klunk2003) found that 51% of political science departments reported formal adoption of student-learning outcomes. Another 28% reported that student-learning outcomes were under discussion or development. Four years later, 60.5% of departments reported offering a capstone course (Ishiyama et al. Reference Ishiyama, Parker, Quinn and Young2007). Whereas the NILOA findings of Ewell, Paulson, and Kinzie (Reference Ewell, Paulson and Kinzie2011) inferred that the social science disciplines were least likely to assess student learning and to use assessment results, the APSA study provides a more thorough description of assessment practices within political science, including student-learning outcomes, assessment methods, use, program changes, and motivation to assess.
Student-Learning Outcomes
The first stage in the idealized model of program assessment is to agree about what students should know and be able to do—otherwise known as student-learning outcomes. In the APSA assessment survey, 84% of political science departments responded that they had adopted or were in the process of developing student-learning outcomes.
Assessment Methods Adopted
The political science discipline reported that it uses many methods of assessment. Programs indicated widespread use of rubrics (77%), capstones (76%), culminating projects (60%), performance assessments (60%), and locally developed exams (54%). Some programs also used external exams (22%), comprehensive exams (26%), and portfolios (32%). Under the “other” category, respondents listed exit interviews, graduate-school placement rates, employment rates, graduation rates, the National Survey of Student Engagement, and alumni surveys.
Use of Assessment Results
Almost 90% of political science programs reported very high assessment usage rates for self-studies and program reviews (88%). Approximately half used assessment for improving their curricula (54%), evaluating program performance (48%), and departmental planning (47%). Fewer programs reported using assessment to improve instruction (44%) and to support budget requests (39%).
Subsequent Program Changes
When asked whether assessment led to program changes, 70% of respondents reported curriculum changes. More than half (57%) reported changes to their assessment practices, and 40% identified modifications in teaching practices. It is interesting that 137 respondents offered an open-ended statement of changes made as a result of assessment. To characterize those responses, they are clustered according to several themes that appeared in the responses. Sequencing and intentionality comprised the theme of the changes in 44 programs. Changes included embedding specific skills into the curricular structure and connecting a sophomore methods course to subsequent upper-level research courses and the capstone. Many changes were cited in capstone assignments and some programs made the capstone a degree requirement; a total of 26 programs made capstone changes. Other changes included adding and deleting courses (20), changing prerequisites especially for methods courses (9), adding writing and research requirements (18), and adding faculty and staff discussions of assessment data (5).
Motivation and Resources to Assess
Political science programs cited several key influences that cause them to assess. The most frequently noted were institutional accreditation (80%), program-review requirements (64%), and faculty and staff interest in educational improvement (62%). Other pressures for assessment included governing-board mandates (47%), national calls for accountability (42%), state-board mandates (34%), specialized accreditation (e.g., the Network of Schools of Public Policy, Affairs, and Administration) (18%), and professional disciplinary associations (14%). When asked what would advance their efforts to assess, political science programs ranked several items highly: more faculty release time (63%), additional stipends (63%), information from other programs (62%), and better tests or measures (55%). In respondents’ open-ended comments, three requested help from the APSA in creating a low-cost, standardized instrument with centralized data collection and reports; another wanted a central site for sharing rubrics. Several respondents raised concerns about conducting assessment in small departments; others were critical of the costs of assessment in terms of time and money; and one suggested helping programs find a minimal compliance model because it would appeal to many departments.
Political science programs cited several key influences that cause them to assess. The most frequently noted were institutional accreditation (80%), program-review requirements (64%), and faculty and staff interest in educational improvement (62%).
HOW DOES POLITICAL SCIENCE COMPARE WITH OTHER ACADEMIC PROGRAMS?
Political science programs overwhelmingly reported developing student-learning outcomes, adopting multiple and diverse assessment measures, using assessment results especially for program review and self-studies, and making subsequent changes in curricular structure and assignments. The data also show that several programs have neither adopted outcomes nor engaged in assessment. Nevertheless, comparing the APSA data and the NILOA data for all departments and the social sciences cluster will enhance the quality of our understanding of political science’s responsiveness to assessment. Data for political science are from the APSA survey; data for “all departments” and the “social sciences” are from the NILOA data (Ewell, Paulson, and Kinzie Reference Ewell, Paulson and Kinzie2011).
Student-Learning Outcomes
Comparison of the NILOA study data for “all departments” and the “social sciences” cluster reveals that there is no significant difference between political science and the national average for all departments. Figure 1 shows that, on average, 86% of academic programs nationally reported that they have adopted student-learning outcomes or that they are under development (UD). The social science cluster outpaces both political science and the all-department adoption of student-learning outcomes: 95% reported adoption of outcomes or UD. The NILOA also studied whether university-wide learning outcomes are established by institutions. Figure 1 shows that respondents report a slightly higher level of adoption of program-level outcomes (86%) than university-wide outcomes (72%)—a pattern that reflects the trend of increasing assessment activity at the program level.
Learning Outcomes
Use of Assessment Results
The data in figure 2 show how assessment results are used by academic programs; political science reported less use than the average for all academic departments. This is the case for every category of assessment use except “self-studies/program review,” in which political science exceeds that reported by all departments and the social sciences cluster (i.e., 88%, 74%, and 64%, respectively). The political science pattern of use is similar to the social sciences pattern. However, political science trails other programs in its use of assessment for instruction improvement, program performance evaluation, and departmental planning.
Use of Assessment Results
Assessment Methods Adopted
As shown in figure 3, the most used assessment instrument according to the NILOA study is the capstone course: 68.5% of all departments nationally, 77.2% of social science departments, and 76% of political science departments reported using capstones for assessment. Political science is also a more enthusiastic adopter of rubrics (77%) than all academic departments (64%) and the social sciences cluster (57%). In other types of assessment, political science use is greater than the social sciences and similar to the national data for all programs. The exception is external national exams and comprehensive exams, in which political science and the social sciences are less inclined to use these assessment methods.
Instruments Used
Motivation and Resources to Assess
One question that this study answers is why departments perform assessment. In figure 4, the data show that the most reported motivation for assessment is institutional accreditation, which is conducted by one of six regional accrediting agencies. Of all departments, 83% cited institutional accreditation as a primary reason for assessing; 74% and 80%, respectively, of social science and political science departments agreed. Political science often cited faculty and staff interest in improving education as their reason for performing assessment (64%); however, this is well below the average for all programs (84%) and the social sciences (84%). Program review is a motivation noted by 75.2% of all departments compared with 65% for the social sciences and 64% for political science programs.
Reasons for Assessment
To encourage the use of more assessment, the NILOA survey asked departments what would help to advance their program’s assessment plan. Figure 5 shows that, with requests for more faculty release time and additional stipends, the preferences of political science are similar to those of all departments and the social science cluster. Political science deviates from the NILOA results only in its more frequent request for information from other departments; that is, 62% of political science programs requested information from other programs as a way to encourage assessment, compared with 46% and 44% for all departments and the social sciences cluster, respectively.
Changes to Encourage Assessment
Subsequent Program Changes
Last, the survey asked respondents to categorize any changes they had made as a result of assessment. Figure 6 presents results for changes made to curriculum, teaching practice, and assessment practice. Most political science departments reported that they made changes to their curriculum, but they did so less frequently than the social sciences cluster and somewhat less than all departments (i.e., 70%, 83%, and 74%, respectively). More than half of political science departments (54%) reported changes to assessment practice, which was more than all departments (49%) but less than the social sciences cluster (67%). Finally, the least reported changes made were to teaching practices; at 40%, political science was between the 47% for all departments and the 35.2% for the social sciences cluster.
The more concerning and less positive results for political science include questions related to the use of assessment for improvement of education. Political science cited this less often as a motive for assessment, and it uses assessment less often for instructional improvement, program evaluation, and departmental planning.
Changes Made
CONCLUSION
The good news is that most political science programs are in compliance with national requirements for establishing and assessing student-learning outcomes. The open-ended comments of the APSA survey demonstrate that assessment can be a useful nudge to consider and adopt more intentional student-learning outcomes and curricula. Two areas in which political science excels are adopting student-learning outcomes and using a broad array of assessment techniques. Unlike the NILOA conclusion of Ewell, Paulson, and Kinzie (Reference Ewell, Paulson and Kinzie2011) that the social sciences assess less frequently than other disciplines, this study demonstrates that the responsiveness of political science to assessment demands is higher in many areas than the social sciences and, in some areas, higher than the average for all departments. Even in the use of assessment data, the APSA survey reveals more frequent use of assessment for self-studies and program reviews.
The more concerning and less positive results for political science include questions related to the use of assessment for improvement of education. Political science cited this less often as a motive for assessment, and it uses assessment less often for instructional improvement, program evaluation, and departmental planning. Although open-ended comments suggest that this happens in some political science programs, the survey data show that compliance is a more frequent motive for assessment than educational improvement.
The use of assessment for improvement by the political science discipline is a question for further study. It is possible that political science responds strategically—that is, using the data when completing required self-studies and program reviews and when seeking additional funds but relying on it much less frequently to inform the core teaching and learning activities of the academy. This stance is consistent with the neo-institutional literature on responsiveness, which finds that organizations tend to decouple required data collection from their core mission activity (Young, Cartwright, and Rudy Reference Young, Cartwright and Rudy2014). Given this organizational tendency, future research may clarify the circumstances that lead to greater integration of outcomes assessment with curricular planning, teaching, and student-learning processes.
Expanding from department-level assessment practices to the discipline of political science as a whole, perhaps the APSA should consider how assessment can be used to promote its efforts to represent the discipline. It stands to reason that national discipline associations that can state expected student-learning outcomes, curricular best practices, and evidence of student achievement will be better representatives and advocates for their discipline. A national study could be undertaken that would enable the APSA to accomplish this effort. For example, if the APSA had evidence that political science programs excel in teaching civics, writing, and critical-thinking skills, every department in the country could build on this for its reporting and planning. Ultimately, positive findings in a national study would help the discipline use assessment offensively to preempt and respond to those who publicly question the value of the discipline. Using assessment to enhance communication about the discipline, why students should study it, and how it elevates individual skills and economic capacity could become the most strategic benefit of the assessment movement.
ACKNOWLEDGMENTS
I want to thank my student assistants Claire Reynolds, Amanda Shetler, Ryan Kohler, and Jacob Fallman for their help with administration and analysis of the survey. I also want to thank Kimberly Mealy, APSA Director, Education, Professional, and Diversity Program for her assistance in administration of the survey.
