Because assessment has moved beyond the immediate demands of regional accreditors, more colleges and university systems have begun to institutionalize the culture of assessment through program review. Through the APSA departmental survey on assessment, symposium contributor Candace Young demonstrates that 88% of responding political science departments used the findings for self-studies or program reviews. As one of the highest outcomes in the survey, this reveals a growing reliance on departmental program reviews within the discipline.
A quick search of “academic program review” on the Internet reveals hundreds of university guidebooks that clearly document various policies and expectations related to program review. These resources demonstrate a wide variety of approaches, but most share a few key elements: a self-study conducted by the department under review; a list of data and documents to be collected and evaluated; an analysis of strengths, weaknesses, opportunities, and challenges facing the program; an assessment by people external to the program and, often, the university; a summative meeting with the academic administration regarding findings; and a future plan for implementation. Conrad and Wilson (Reference Conrad and Wilson1986) determined that although the purpose of a program review is to enhance overall quality, this objective can be defined in multiple ways—improving reputation, increasing resources, advancing outcomes, and enhancing the value-added nature of the academic program. They found that most institutions pursued aspects of all four elements.
The relevance of such a systematic review is particularly heightened for unaccredited disciplines such as political science, which is disadvantaged compared to other disciplines that go through the regular formal accreditation–reaccreditation process, such as public administration and public policy. Disciplines with accrediting agencies experience external pressures on the institution to increase the number of faculty, grow the operating budget, obtain additional equipment, and decrease class sizes. For disciplines like political science, the institutionally mandated or self-initiated external review process is the only consistent leverage we have with administrators; it is often our only opportunity to improve our access to resources and to coherently inform them of the value and success of our programs. Many faculty and departmental leaders are legitimately concerned that this process will be wasted time—uncompensated and in addition to current work—resulting in frustration and disengagement. However, department chairs who lead the faculty in seriously engaging this process may derive long-term benefits for their department or program. An effective program review enables us to demonstrate the impact of the discipline on institutional strategic goals and objectives, to provide an external authoritative voice regarding the quality of our work, to explore the funding of annual curricular assessment from a more global perspective, to force the department to engage collectively in periodic self-evaluation, and to leverage the administration for additional funds and resources in subsequent years.
Collective engagement by a department will garner the greatest results. There are common strategies that departments may implement to enhance the potential impact of the process. As a chair, I have led departments at several different types of institutions in a self-study process and/or site visit, and I have worked subsequently to implement changes. In addition, I have been an external reviewer at many institutions with processes that vary greatly in design: a single evaluator, two external evaluators, an external team, an internal team, and online. Each approach has similarities that are described in this article and my recommendations draw on discussions and explicit suggestions from other department chairs and external reviewers. Footnote 1
An effective program review enables us to demonstrate the impact of the discipline on institutional strategic goals and objectives, to provide an external authoritative voice regarding the quality of our work, to explore the funding of annual curricular assessment from a more global perspective, to force the department to engage collectively in periodic self-evaluation, and to leverage the administration for additional funds and resources in subsequent years.
DEPARTMENTAL ENGAGEMENT
A key factor in the success of this process is whether a department engages as a team to identify the outcomes it wants from the program review. If professors and the chair perceive the process as yet another administrative mandate resulting in limited tangible outcomes, they will not invest. Instead, there must be buy-in from the faculty. One way to stimulate buy-in is to emphasize the opportunity to advance departmental goals. If the process can answer questions pertinent to the faculty, resolve conflicts within the department, or promote agendas in which the department already is invested, it does not matter what ultimately happens with the program review at the provost level. By engaging faculty, students, and alumni in the self-study, the process can lead deliberately to financial-development opportunities, providing greater resources for both students and professors. Through the documentation of successes, the program can increase its visibility in the institution and the local community.
To increase faculty investment in the process, a department should determine its ideal outcomes at the beginning of the self-study (e.g., decisions regarding future directions in curriculum, determinations related to the success of previous attempts, additional resources to grow or to improve quality, and preparation for future changes facing the department) rather than simply completing institutional forms and checking administrative boxes. All programs confront difficult questions and necessary decisions regarding program staffing, curriculum, and priorities that are easily tabled and ignored during the busy term. Program review is an opportunity to force ourselves to address these questions and receive feedback from a neutral party from within our discipline. If the department perceives the external reviewer as an ally in the quest to improve the quality of the program rather than a hostile critic seeking to undermine its work, the review process experiences greater success in rendering beneficial results.
SUGGESTIONS FOR DEPARTMENT CHAIRS AND HEADS
As department chairs begin to plan the semester- or year-long review process, several issues should be considered. A most important element for ensuring a successful review is awareness of the institution’s goals and timetables. It is easy to allow the program-review process to consume an inordinate amount of time. However, the remainder of your yearly work does not go away during the program-review process. It is important to decide how much time should be budgeted for the process and to adhere to it. However, be realistic in making this determination.
Another priority of the departmental leadership is to ensure that faculty, staff, and students are engaged in the program-review process to promote universal investment in it. This requires assignments of expected faculty contributions, careful discussions of previous program-review outcomes and responses, and a specific list of materials needed from all faculty distributed early in the process (e.g., updated curriculum vita [CVs], syllabi, placement information, alumni and internship data, and student research projects supervised). It is useful to divide the self-study into smaller elements that subcommittees or individuals can draft. These elements can be based on either the framework for the self-study report provided by the institution or one of the many models available online. The final responsibility of the departmental leadership is in leveraging outcomes of the self-study to benefit the program. This may require the department head to know which resources are required to implement the proposed plans and how to pursue them.
Another priority of the departmental leadership is to ensure that faculty, staff, and students are engaged in the program-review process to promote universal investment in it.
SELECTING REVIEWERS
One of the most significant decisions for a department to make is the choice of the external reviewer. The reviewer should be an ally seeking to help the department improve and willing to assist in constructing a compelling case for the academic administration. The reviewer also should be capable of giving honest feedback and have enough experience to generate creative solutions to apparently intractable problems. The circumstances of the department are critical in determining characteristics of an ideal reviewer. The selectee must be conversant with similarly situated departments and should have experience with an institution in a similar Carnegie classification; also, location within the same region will facilitate logistical difficulties. Depending on the purpose of the review, departments may want to find a reviewer from a peer school (helping to articulate the unique needs and concerns of the program) or from an aspirant peer (assisting the program in defining the next steps for improvement or growth). An institution may require a selected reviewer to be from a program with the same degrees offered or to have administrative experience. Other institutions may prohibit selecting former classmates, collaborators, and colleagues as external reviewers.
Many institutions set a budget for an external reviewer’s compensation; others allow it to be negotiated. Political scientists in the mid-South and Midwest typically are compensated between $1,000 and $1,500 plus expenses; this may vary in other regions of the country. This amount incorporates the time necessary to review the self-study, conduct the site visit, and write the external report. If the university’s financial contribution is inadequate for the stipend, it can be supplemented from a department’s budget to compensate the appropriate external reviewer who will contribute the most to the process.
A good source for a list of potential reviewers based on a department’s criteria is the APSA Departmental Services Program (DSP). Footnote 2 Programs that are departmental members should e-mail the program director noting the preferred relevant attributes to be reflected in the reviewer (e.g., type of institution, region, and degrees offered). Individuals seeking to participate as an external reviewer also can share their information with the DSP to be included in the database. It is important to obtain departmental consensus on the ranked list of reviewers. If the department will use a team of external reviewers, be sure to select a team that reflects a range of subfields and experiences that address the needs and concerns of your program (e.g., reviewers with experience in internships, undergraduate research, pre-law advising, Model UN teams, capstone programs, masters or doctoral degrees, and assessment needs). Quickly contact the people on the shortlist for their availability and interest and obtain copies of CVs to submit to the dean or provost for approval. After the individual or team is identified, schedule dates for the site visit as soon as possible—especially meetings with the dean, the provost’s office, and the closing meeting—they need to be on everyone’s calendar early in the process. Having established dates also will provide the timeframe for the self-study to be completed.
WRITING THE SELF-STUDY
When writing the self-study report, keep in mind the purpose and focus of the program review. Provide a clear picture of the department, its future challenges, accomplishments, reasons for choices, discoveries from annual assessment activities, and current issues that need to be resolved. Remember that the external reviewer does not know your context or your institution. Many self-study reports merely list institutional requirements, without the contextual material necessary for a reasonable person to interpret accurately. The self-study should be written in a way that an outsider can comprehend. In composing the report, do not be self-congratulatory but note what you do well, where your weaknesses are, and the steps taken for improvement. It may be useful to frame the study by responding to the outcomes of previous self-studies. By noting the concerns raised in earlier review cycles, the department can use the yearly assessment results and other data to determine how successfully it has addressed them.
It is easy to focus too much on faculty research; however, the overall program review in most circumstances is about students. Therefore, whereas faculty research and professional accomplishments are significant in that they demonstrate their competency, be sure to relate them to the curriculum. Unless an institution has a unique mission, most departmental outcomes should be related to students. Discuss various degree programs separately—undergraduate and graduate students are different, Masters and PhD programs are not replicates—their contexts are as different as their outcomes. These distinctions among degree programs should be clear.
The self-study should not be merely descriptive; it also should be analytical. The report should draw on all of the research skills that political scientists possess—our self-studies should be some of the best on campus. This means that evidence should be provided for all claims. As with all data, the context for accurate interpretation should be provided. For instance, if you provide data on retention and graduate rates for each degree offered, it helps to provide graduation rates of similar programs or the institution as a whole. Have graduation rates increased or decreased over time? Do these rates parallel changes at the institution? If not, what explains the change in percentages? If there is a problem, how has the department addressed it? What are the department’s future plans?
Program review is not a replacement for continuous assessment; rather, it provides an opportunity to routinely review the entire curricula, co-curricula, and departmental culture using the data collected through the annual assessment of student learning.
PROGRAM REVIEW AND ASSESSMENT
Program review is not a replacement for continuous assessment; rather, it provides an opportunity to routinely review the entire curricula, co-curricula, and departmental culture using the data collected through the annual assessment of student learning. Consider program review to be part of strategic planning and goal setting so that the department does not merely react semester by semester to institutional stimuli but instead is heading in a planned direction. The self-study provides a base point from which the program will develop; the external evaluator allows for peer review of both the assessment plan and its findings. This process is an opportunity for a department to evaluate program learning objectives—as well as strengths and weaknesses of the program as a whole—in a larger context. Assessment is a yearly evaluation of specific learning objectives that allows a department to engage in course correction. Whereas assessment tends to occur with an assumption of fixed resources, program review allows a department to consider what is possible with additional resources.
THE CAMPUS VISIT
For the campus visit to be successful, it must be carefully planned in advance. Consider all of the various members of the department and its partners that external reviewers should meet to obtain a clear and accurate understanding of the program (i.e., senior faculty, untenured faculty, adjunct faculty, online faculty, majors and minors, interdisciplinary programs in which faculty teach or direct, alumni, and community partners). These groups will vary among campuses, but a comprehensive visit and thorough self-study ensures that the faculty will not claim that reviewers did not understand their unique circumstances after the site visit. Therefore, it is important to plan this schedule early in the process. If it is difficult to get students to attend meetings, have the reviewers meet majors and minors in an established class. The department seeks to provide the reviewers with honest information, including observations that faculty might not obtain on their own. Allow reviewers to meet with various constituencies without a departmental presence or with untenured faculty without tenured colleagues in the room. To make this work, scheduling must be done well in advance so that meeting rooms are reserved and people are available to participate.
REVIEWERS
Treat the reviewers well and show off your university and community as you would for employment candidates (without the housing tour). They are more likely to complete the report in a timely manner if they have been treated hospitably. Meals are another opportunity for the external team to meet with various appropriate groups on and off campus. It is important to give reviewers an opportunity to ask for information before, during, and after the site visit. If documentation or data are requested, assume that it is essential for their deliberations and provide it quickly. Give external reviewers scheduled time to deliberate in advance of the cumulative meeting with the administration; this allows them to identify their key findings. The final meeting allows the chair to gauge institutional support for the department in upcoming budget cycles and determines implementation priorities.
SUGGESTIONS FOR REVIEWERS
Reviewers provide this professional service for many reasons: site visits allow them to observe other programs and their rationale, network with other political scientists, and observe new approaches that could be adapted to their home campus. It also is an opportunity to support the discipline and colleagues who, in other contexts, improve the quality of their political science program and curriculum. The reviewers’ purpose is to assist, not to point out all of the ways their “better” program differs from the program under review. It is helpful for reviewers to begin the relationship with department leaders by asking, “What do you hope to gain from this external review?” This keeps the reviewers and the department on “the same page” regarding outcomes—although it should not preclude conflicting interpretations.
USING PROGRAM REVIEW AS LEVERAGE
It is unfortunate but true that an outsider who makes identical statements as department leaders and senior faculty may be taken more seriously by academic administrators; however, most institutions will not take action based solely on an external review. It is important to have a clear sense of what the department desires and carefully advocate for it, using the external reviewer as a partner. It is obvious that different institutions use program review in various ways—as a baseline for future evaluations, the source of data for resource distribution, or to simply check off a box and place a filled binder on a shelf. However, if a department has considered in detail the outcomes that it desires from the self-study and external review, it will be in a stronger position to effectively leverage the results.
Departments must translate the external reviewer’s discussion of best practices and disciplinary norms to address local circumstances and opportunities. A review may provide the evidence a department needs to successfully advocate for a new tenure track line in a key subfield. It is helpful for the department to demonstrate how the external reviewer’s recommendations align with the strategic plan of the university, address an area of campus emphasis, or correct a problem area that is an administration priority. For instance, the need to add a mandatory methods course in the curriculum—with its concomitant expensive software subscription or required new faculty line—can be shown to enhance the university’s plan to pursue undergraduate research experiences. Program review is the opportunity every seven years to receive close direct attention from the provost—which individual departments generally do not receive—and to provide information about the department that will not be lost in the deluge of routine reporting. We can demonstrate to upper-level academic administrators how our discipline supports broader institutional goals. This instruction is helpful, especially for a department whose president, provost, or dean is not familiar with the social and behavioral sciences. With the resource challenges facing higher education today and the negative perception of political science, program review is our opportunity to rewrite the script relative to our departments on our own campuses.
